Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN

07/30/2021
by   Xiaohan Ding, et al.
0

The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops. However, the training objective of CNNs usually tends to minimize an accuracy-related loss function without any attention paid to the redundancy, making the redundancy distribute randomly on all the filters, such that removing any of them may trigger information loss and accuracy drop, necessitating a following finetuning step for recovery. In this paper, we propose to manipulate the redundancy during training to facilitate network pruning. To this end, we propose a novel Centripetal SGD (C-SGD) to make some filters identical, resulting in ideal redundancy patterns, as such filters become purely redundant due to their duplicates; hence removing them does not harm the network. As shown on CIFAR and ImageNet, C-SGD delivers better performance because the redundancy is better organized, compared to the existing methods. The efficiency also characterizes C-SGD because it is as fast as regular SGD, requires no finetuning, and can be conducted simultaneously on all the layers even in very deep CNNs. Besides, C-SGD can improve the accuracy of CNNs by first training a model with the same architecture but wider layers then squeezing it into the original width.

READ FULL TEXT
research
04/08/2019

Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure

The redundancy is widely recognized in Convolutional Neural Networks (CN...
research
05/12/2019

Approximated Oracle Filter Pruning for Destructive CNN Width Optimization

It is not easy to design and run Convolutional Neural Networks (CNNs) du...
research
06/16/2022

Asymptotic Soft Cluster Pruning for Deep Neural Networks

Filter pruning method introduces structural sparsity by removing selecte...
research
01/22/2017

Optimization on Product Submanifolds of Convolution Kernels

Recent advances in optimization methods used for training convolutional ...
research
08/14/2020

AntiDote: Attention-based Dynamic Optimization for Neural Network Runtime Efficiency

Convolutional Neural Networks (CNNs) achieved great cognitive performanc...
research
07/07/2020

Lossless CNN Channel Pruning via Gradient Resetting and Convolutional Re-parameterization

Channel pruning (a.k.a. filter pruning) aims to slim down a convolutiona...
research
10/21/2019

Separable Convolutional Eigen-Filters (SCEF): Building Efficient CNNs Using Redundancy Analysis

The high model complexity of deep learning algorithms enables remarkable...

Please sign up or login with your details

Forgot password? Click here to reset