Pruning by Active Attention Manipulation

10/20/2022
by   Zahra Babaiee, et al.
2

Filter pruning of a CNN is typically achieved by applying discrete masks on the CNN's filter weights or activation maps, post-training. Here, we present a new filter-importance-scoring concept named pruning by active attention manipulation (PAAM), that sparsifies the CNN's set of filters through a particular attention mechanism, during-training. PAAM learns analog filter scores from the filter weights by optimizing a cost function regularized by an additive term in the scores. As the filters are not independent, we use attention to dynamically learn their correlations. Moreover, by training the pruning scores of all layers simultaneously, PAAM can account for layer inter-dependencies, which is essential to finding a performant sparse sub-network. PAAM can also train and generate a pruned network from scratch in a straightforward, one-stage training process without requiring a pre-trained network. Finally, PAAM does not need layer-specific hyperparameters and pre-defined layer budgets, since it can implicitly determine the appropriate number of filters in each layer. Our experimental results on different network architectures suggest that PAAM outperforms state-of-the-art structured-pruning methods (SOTA). On CIFAR-10 dataset, without requiring a pre-trained baseline network, we obtain 1.02 reduction, on ResNet56 and ResNet110, respectively. Similarly, on the ImageNet dataset, PAAM achieves 1.06 parameters on ResNet50. For Cifar-10, this is better than the SOTA with a margin of 9.5

READ FULL TEXT
research
04/15/2022

End-to-End Sensitivity-Based Filter Pruning

In this paper, we present a novel sensitivity-based filter pruning algor...
research
10/28/2020

Data Agnostic Filter Gating for Efficient Deep Networks

To deploy a well-trained CNN model on low-end computation edge devices, ...
research
06/11/2019

BasisConv: A method for compressed representation and learning in CNNs

It is well known that Convolutional Neural Networks (CNNs) have signific...
research
09/08/2022

CAP: instance complexity-aware network pruning

Existing differentiable channel pruning methods often attach scaling fac...
research
10/15/2021

Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction

Dynamic model pruning is a recent direction that allows for the inferenc...
research
09/21/2021

Learning Interpretable Concept Groups in CNNs

We propose a novel training methodology – Concept Group Learning (CGL) –...
research
09/27/2022

Sauron U-Net: Simple automated redundancy elimination in medical image segmentation via filter pruning

We present Sauron, a filter pruning method that eliminates redundant fea...

Please sign up or login with your details

Forgot password? Click here to reset