DeepAI AI Chat
Log In Sign Up

Layer-wise Model Pruning based on Mutual Information

by   Chun Fan, et al.

The proposed pruning strategy offers merits over weight-based pruning techniques: (1) it avoids irregular memory access since representations and matrices can be squeezed into their smaller but dense counterparts, leading to greater speedup; (2) in a manner of top-down pruning, the proposed method operates from a more global perspective based on training signals in the top layer, and prunes each layer by propagating the effect of global signals through layers, leading to better performances at the same sparsity level. Extensive experiments show that at the same sparsity level, the proposed strategy offers both greater speedup and higher performances than weight-based pruning methods (e.g., magnitude pruning, movement pruning).


page 1

page 2

page 3

page 4


Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning

Magnitude-based pruning is one of the simplest methods for pruning neura...

A Deeper Look at the Layerwise Sparsity of Magnitude-based Pruning

Recent discoveries on neural network pruning reveal that, with a careful...

Advancing Model Pruning via Bi-level Optimization

The deployment constraints in practical applications necessitate the pru...

Max-Affine Spline Insights Into Deep Network Pruning

In this paper, we study the importance of pruning in Deep Networks (DNs)...

Fast ConvNets Using Group-wise Brain Damage

We revisit the idea of brain damage, i.e. the pruning of the coefficient...

Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity

Network pruning can reduce the high computation cost of deep neural netw...

Performance optimizations on deep noise suppression models

We study the role of magnitude structured pruning as an architecture sea...