Improving Generalization by Controlling Label-Noise Information in Neural Network Weights

02/19/2020
by   Hrayr Harutyunyan, et al.
23

In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation sometimes help, but do not prevent this behavior. If one considers neural network weights as random variables that depend on the data and stochasticity of training, the amount of memorized information can be quantified with the Shannon mutual information between weights and the vector of all training labels given inputs, I(w : y|x). We show that for any training algorithm, low values of this term correspond to reduction in memorization of label-noise and better generalization bounds. To obtain these low values, we propose training algorithms that employ an auxiliary network that predicts gradients in the final layers of a classifier without accessing labels. We illustrate the effectiveness of our approach on versions of MNIST, CIFAR-10, and CIFAR-100 corrupted with various noise models, and on a large-scale dataset Clothing1M that has noisy labels.

READ FULL TEXT

page 7

page 17

page 18

research
08/23/2022

A Study on the Impact of Data Augmentation for Training Convolutional Neural Networks in the Presence of Noisy Labels

Label noise is common in large real-world datasets, and its presence har...
research
05/09/2017

Learning Deep Networks from Noisy Labels with Dropout Regularization

Large datasets often have unreliable labels-such as those obtained from ...
research
12/21/2020

Regularization in neural network optimization via trimmed stochastic gradient descent with noisy label

Regularization is essential for avoiding over-fitting to training data i...
research
09/08/2019

L_DMI: An Information-theoretic Noise-robust Loss Function

Accurately annotating large scale dataset is notoriously expensive both ...
research
03/22/2021

On the Robustness of Monte Carlo Dropout Trained with Noisy Labels

The memorization effect of deep learning hinders its performance to effe...
research
03/29/2022

Agreement or Disagreement in Noise-tolerant Mutual Learning?

Deep learning has made many remarkable achievements in many fields but s...
research
05/29/2019

Where is the Information in a Deep Neural Network?

Whatever information a Deep Neural Network has gleaned from past data is...

Please sign up or login with your details

Forgot password? Click here to reset