i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery

12/07/2021
by   Cameron R. Wolfe, et al.
0

We propose a novel, structured pruning algorithm for neural networks – the iterative, Sparse Structured Pruning algorithm, dubbed as i-SpaSP. Inspired by ideas from sparse signal recovery, i-SpaSP operates by iteratively identifying a larger set of important parameter groups (e.g., filters or neurons) within a network that contribute most to the residual between pruned and dense network output, then thresholding these groups based on a smaller, pre-defined pruning ratio. For both two-layer and multi-layer network architectures with ReLU activations, we show the error induced by pruning with i-SpaSP decays polynomially, where the degree of this polynomial becomes arbitrarily large based on the sparsity of the dense network's hidden representations. In our experiments, i-SpaSP is evaluated across a variety of datasets (i.e., MNIST and ImageNet) and architectures (i.e., feed forward networks, ResNet34, and MobileNetV2), where it is shown to discover high-performing sub-networks and improve upon the pruning efficiency of provable baseline methodologies by several orders of magnitude. Put simply, i-SpaSP is easy to implement with automatic differentiation, achieves strong empirical results, comes with theoretical convergence guarantees, and is efficient, thus distinguishing itself as one of the few computationally efficient, practical, and provable pruning algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/30/2023

Generalization Bounds for Magnitude-Based Pruning via Sparse Matrix Sketching

In this paper, we derive a novel bound on the generalization error of Ma...
research
08/28/2023

A Generalization of Continuous Relaxation in Structured Pruning

Deep learning harnesses massive parallel floating-point processing to tr...
research
07/31/2021

Provably Efficient Lottery Ticket Discovery

The lottery ticket hypothesis (LTH) claims that randomly-initialized, de...
research
07/30/2021

Pruning Neural Networks with Interpolative Decompositions

We introduce a principled approach to neural network pruning that casts ...
research
08/19/2020

Data-Independent Structured Pruning of Neural Networks via Coresets

Model compression is crucial for deployment of neural networks on device...
research
05/21/2019

Revisiting hard thresholding for DNN pruning

The most common method for DNN pruning is hard thresholding of network w...
research
11/18/2019

Provable Filter Pruning for Efficient Neural Networks

We present a provable, sampling-based approach for generating compact Co...

Please sign up or login with your details

Forgot password? Click here to reset