Sparseout: Controlling Sparsity in Deep Networks

04/17/2019
by   Najeeb Khan, et al.
0

Dropout is commonly used to help reduce overfitting in deep neural networks. Sparsity is a potentially important property of neural networks, but is not explicitly controlled by Dropout-based regularization. In this work, we propose Sparseout a simple and efficient variant of Dropout that can be used to control the sparsity of the activations in a neural network. We theoretically prove that Sparseout is equivalent to an L_q penalty on the features of a generalized linear model and that Dropout is a special case of Sparseout for neural networks. We empirically demonstrate that Sparseout is computationally inexpensive and is able to control the desired level of sparsity in the activations. We evaluated Sparseout on image classification and language modelling tasks to see the effect of sparsity on these tasks. We found that sparsity of the activations is favorable for language modelling performance while image classification benefits from denser activations. Sparseout provides a way to investigate sparsity in state-of-the-art deep learning models. Source code for Sparseout could be found at <https://github.com/najeebkhan/sparseout>.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/21/2020

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

Overfitting is one of the most common problems when training deep neural...
research
11/21/2016

Generalized Dropout

Deep Neural Networks often require good regularizers to generalize well....
research
08/27/2018

Sparsity in Deep Neural Networks - An Empirical Investigation with TensorQuant

Deep learning is finding its way into the embedded world with applicatio...
research
09/13/2019

DASNet: Dynamic Activation Sparsity for Neural Network Efficiency Improvement

To improve the execution speed and efficiency of neural networks in embe...
research
06/28/2020

Layer Sparsity in Neural Networks

Sparsity has become popular in machine learning, because it can save com...
research
11/28/2020

Optical Phase Dropout in Diffractive Deep Neural Network

Unitary learning is a backpropagation that serves to unitary weights upd...
research
11/04/2016

Information Dropout: Learning Optimal Representations Through Noisy Computation

The cross-entropy loss commonly used in deep learning is closely related...

Please sign up or login with your details

Forgot password? Click here to reset