DeepAI AI Chat
Log In Sign Up

Pruning Neural Networks with Interpolative Decompositions

07/30/2021
by   Jerry Chee, et al.
cornell university
0

We introduce a principled approach to neural network pruning that casts the problem as a structured low-rank matrix approximation. Our method uses a novel application of a matrix factorization technique called the interpolative decomposition to approximate the activation output of a network layer. This technique selects neurons or channels in the layer and propagates a corrective interpolation matrix to the next layer, resulting in a dense, pruned network with minimal degradation before fine tuning. We demonstrate how to prune a neural network by first building a set of primitives to prune a single fully connected or convolution layer and then composing these primitives to prune deep multi-layer networks. Theoretical guarantees are provided for pruning a single hidden layer fully connected network. Pruning with interpolative decompositions achieves strong empirical results compared to the state-of-the-art on multiple applications from one and two hidden layer networks on Fashion MNIST to VGG and ResNets on CIFAR-10. Notably, we achieve an accuracy of 93.62 ± 0.36 reduction. This gains 0.02

READ FULL TEXT

page 1

page 2

page 3

page 4

10/25/2020

Neuron Merging: Compensating for Pruned Neurons

Network pruning is widely used to lighten and accelerate neural network ...
12/07/2021

i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery

We propose a novel, structured pruning algorithm for neural networks – t...
01/15/2021

Convolutional Neural Network with Pruning Method for Handwritten Digit Recognition

CNN model is a popular method for imagery analysis, so it could be utili...
09/11/2020

Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation

A common technique for compressing a neural network is to compute the k-...
11/10/2020

Dirichlet Pruning for Neural Network Compression

We introduce Dirichlet pruning, a novel post-processing technique to tra...
02/04/2021

Universal Approximation Theorems of Fully Connected Binarized Neural Networks

Neural networks (NNs) are known for their high predictive accuracy in co...
10/01/2021

Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)

A central goal in deep learning is to learn compact representations of f...