Basis Scaling and Double Pruning for Efficient Transfer Learning

08/06/2021
by   Ken C. L. Wong, et al.
0

Transfer learning allows the reuse of deep learning features on new datasets with limited data. However, the resulting models could be unnecessarily large and thus inefficient. Although network pruning can be applied to improve inference efficiency, existing algorithms usually require fine-tuning and may not be suitable for small datasets. In this paper, we propose an algorithm that transforms the convolutional weights into the subspaces of orthonormal bases where a model is pruned. Using singular value decomposition, we decompose a convolutional layer into two layers: a convolutional layer with the orthonormal basis vectors as the filters, and a layer that we name "BasisScalingConv", which is responsible for rescaling the features and transforming them back to the original space. As the filters in each transformed layer are linearly independent with known relative importance, pruning can be more effective and stable, and fine tuning individual weights is unnecessary. Furthermore, as the numbers of input and output channels of the original convolutional layer remain unchanged, basis pruning is applicable to virtually all network architectures. Basis pruning can also be combined with existing pruning algorithms for double pruning to further increase the pruning capability. With less than 1 in the classification accuracy, we can achieve pruning ratios up to 98.9 parameters and 98.6

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/25/2022

Fine-tuning Pruned Networks with Linear Over-parameterization

Structured pruning compresses neural networks by reducing channels (filt...
research
10/25/2020

Neuron Merging: Compensating for Pruned Neurons

Network pruning is widely used to lighten and accelerate neural network ...
research
10/11/2018

Rethinking the Value of Network Pruning

Network pruning is widely used for reducing the heavy computational cost...
research
02/14/2020

Layer-wise Pruning and Auto-tuning of Layer-wise Learning Rates in Fine-tuning of Deep Networks

Existing fine-tuning methods use a single learning rate over all layers....
research
11/19/2016

Pruning Convolutional Neural Networks for Resource Efficient Inference

We propose a new formulation for pruning convolutional kernels in neural...
research
10/25/2021

Network compression and faster inference using spatial basis filters

We present an efficient alternative to the convolutional layer through u...
research
05/26/2021

Dynamic Probabilistic Pruning: A general framework for hardware-constrained pruning at different granularities

Unstructured neural network pruning algorithms have achieved impressive ...

Please sign up or login with your details

Forgot password? Click here to reset