Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining

06/10/2019
by   Yasutoshi Ida, et al.
0

Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this problem, we propose Network Implosion, it erases multiple layers from Residual Networks without degrading accuracy. Our key idea is to introduce a priority term that identifies the importance of a layer; we can select unimportant layers according to the priority and erase them after the training. In addition, we retrain the networks to avoid critical drops in accuracy after layer erasure. A theoretical assessment reveals that our erasure and retraining scheme can erase layers without accuracy drop, and achieve higher accuracy than is possible with training from scratch. Our experiments show that Network Implosion can, for classification on Cifar-10/100 and ImageNet, reduce the number of layers by 24.00 to 42.86 percent without any drop in accuracy.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/28/2017

ShaResNet: reducing residual network parameter number by sharing weights

Deep Residual Networks have reached the state of the art in many image p...
research
05/28/2016

Weighted Residuals for Very Deep Networks

Deep residual networks have recently shown appealing performance on many...
research
05/26/2018

The Singular Values of Convolutional Layers

We characterize the singular values of the linear transformation associa...
research
01/27/2023

Deep Residual Compensation Convolutional Network without Backpropagation

PCANet and its variants provided good accuracy results for classificatio...
research
09/10/2019

VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks

Improving weight sparsity is a common strategy for producing light-weigh...
research
11/30/2017

Convolutional Networks with Adaptive Computation Graphs

Do convolutional networks really need a fixed feed-forward structure? Of...
research
07/15/2017

Ternary Residual Networks

Sub-8-bit representation of DNNs incur some discernible loss of accuracy...

Please sign up or login with your details

Forgot password? Click here to reset