Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization

06/14/2018
by   Yibo Yang, et al.
0

We propose a simple and easy to implement neural network compression algorithm that achieves results competitive with more complicated state-of-the-art methods. The key idea is to modify the original optimization problem by adding K independent Gaussian priors (corresponding to the k-means objective) over the network parameters to achieve parameter quantization, as well as an L1 penalty to achieve pruning. Unlike many existing quantization-based methods, our method uses hard clustering assignments of network parameters, which adds minimal change or overhead to standard network training. We also demonstrate experimentally that tying neural network parameters provides less gain in generalization performance than changing network architecture and connectivity patterns entirely.

READ FULL TEXT

page 13

page 14

research
02/05/2019

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Quantization of neural networks has become common practice, driven by th...
research
02/13/2017

Soft Weight-Sharing for Neural Network Compression

The success of deep learning in numerous application domains created the...
research
12/15/2022

Towards Hardware-Specific Automatic Compression of Neural Networks

Compressing neural network architectures is important to allow the deplo...
research
07/06/2023

Pruning vs Quantization: Which is Better?

Neural network pruning and quantization techniques are almost as old as ...
research
05/15/2020

A flexible, extensible software framework for model compression based on the LC algorithm

We propose a software framework based on the ideas of the Learning-Compr...
research
09/01/2022

On Quantizing Implicit Neural Representations

The role of quantization within implicit/coordinate neural networks is s...
research
10/24/2019

A Comparative Study of Neural Network Compression

There has recently been an increasing desire to evaluate neural networks...

Please sign up or login with your details

Forgot password? Click here to reset