Smallify: Learning Network Size while Training

06/10/2018
by   Guillaume Leclerc, et al.
0

As neural networks become widely deployed in different applications and on different hardware, it has become increasingly important to optimize inference time and model size along with model accuracy. Most current techniques optimize model size, model accuracy and inference time in different stages, resulting in suboptimal results and computational inefficiency. In this work, we propose a new technique called Smallify that optimizes all three of these metrics at the same time. Specifically we present a new method to simultaneously optimize network size and model performance by neuron-level pruning during training. Neuron-level pruning not only produces much smaller networks but also produces dense weight matrices that are amenable to efficient inference. By applying our technique to convolutional as well as fully connected models, we show that Smallify can reduce network size by 35X with a 6X improvement in inference time with similar accuracy as models found by traditional training techniques.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/12/2018

ClosNets: a Priori Sparse Topologies for Faster DNN Training

Fully-connected layers in deep neural networks (DNN) are often the throu...
research
05/28/2019

Inference with Hybrid Bio-hardware Neural Networks

To understand the learning process in brains, biologically plausible alg...
research
10/16/2021

Neural Network Pruning Through Constrained Reinforcement Learning

Network pruning reduces the size of neural networks by removing (pruning...
research
06/19/2019

Joint Pruning on Activations and Weights for Efficient Neural Networks

With rapidly scaling up of deep neural networks (DNNs), extensive resear...
research
11/21/2018

Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks

In this work, we propose a graph-adaptive pruning (GAP) method for effic...
research
03/28/2022

Pruning In Time (PIT): A Lightweight Network Architecture Optimizer for Temporal Convolutional Networks

Temporal Convolutional Networks (TCNs) are promising Deep Learning model...
research
07/13/2023

Efficient SGD Neural Network Training via Sublinear Activated Neuron Identification

Deep learning has been widely used in many fields, but the model trainin...

Please sign up or login with your details

Forgot password? Click here to reset