Understanding Generalization of Deep Neural Networks Trained with Noisy Labels

05/27/2019
by   Wei Hu, et al.
0

Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. When the training dataset contains a fraction of noisy labels, can neural networks be resistant to over-fitting and still generalize on the true distribution? Inspired by recent theoretical work that established connections between over-parameterized neural networks and neural tangent kernel (NTK), we propose two simple regularization methods for this purpose: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, both methods are related to kernel ridge regression with respect to the NTK, and we prove their generalization guarantee on the true data distribution despite being trained using noisy labels. The generalization bound is independent of the network size, and only depends on the training inputs and true labels (instead of noisy labels) as well as the noise level in the labels. Empirical results verify the effectiveness of these methods on noisily labeled datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/06/2020

Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network

Overparametrized neural networks trained by gradient descent (GD) can pr...
research
12/16/2021

Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimation

Over-parameterized deep neural networks are able to achieve excellent tr...
research
02/10/2021

Input Similarity from the Neural Network Perspective

We first exhibit a multimodal image registration task, for which a neura...
research
12/19/2018

A Note on Lazy Training in Supervised Differentiable Programming

In a series of recent theoretical works, it has been shown that strongly...
research
09/28/2018

Pumpout: A Meta Approach for Robustly Training Deep Neural Networks with Noisy Labels

It is challenging to train deep neural networks robustly on the industri...
research
07/02/2020

A Revision of Neural Tangent Kernel-based Approaches for Neural Networks

Recent theoretical works based on the neural tangent kernel (NTK) have s...
research
03/19/2018

On the importance of single directions for generalization

Despite their ability to memorize large datasets, deep neural networks o...

Please sign up or login with your details

Forgot password? Click here to reset