AL2: Progressive Activation Loss for Learning General Representations in Classification Neural Networks

03/07/2020
by   Majed El Helou, et al.
0

The large capacity of neural networks enables them to learn complex functions. To avoid overfitting, networks however require a lot of training data that can be expensive and time-consuming to collect. A common practical approach to attenuate overfitting is the use of network regularization techniques. We propose a novel regularization method that progressively penalizes the magnitude of activations during training. The combined activation signals produced by all neurons in a given layer form the representation of the input image in that feature space. We propose to regularize this representation in the last feature layer before classification layers. Our method's effect on generalization is analyzed with label randomization tests and cumulative ablations. Experimental results show the advantages of our approach in comparison with commonly-used regularizers on standard benchmark datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/21/2020

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

Overfitting is one of the most common problems when training deep neural...
research
04/13/2021

The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks

Overfitting is one of the fundamental challenges when training convoluti...
research
11/14/2018

Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization

Overfitting frequently occurs in deep learning. In this paper, we propos...
research
06/22/2022

Consistency of Neural Networks with Regularization

Neural networks have attracted a lot of attention due to its success in ...
research
10/09/2021

Pairwise Margin Maximization for Deep Neural Networks

The weight decay regularization term is widely used during training to c...
research
08/19/2023

PDL: Regularizing Multiple Instance Learning with Progressive Dropout Layers

Multiple instance learning (MIL) was a weakly supervised learning approa...
research
02/27/2023

Complex Clipping for Improved Generalization in Machine Learning

For many machine learning applications, a common input representation is...

Please sign up or login with your details

Forgot password? Click here to reset