DeepAI
Log In Sign Up

Understanding and Improving Early Stopping for Learning with Noisy Labels

06/30/2021
by   Yingbin Bai, et al.
0

The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods. To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering a DNN as a whole. However, a DNN can be considered as a composition of a series of layers, and we find that the latter layers in a DNN are much more sensitive to label noise, while their former counterparts are quite robust. Therefore, selecting a stopping point for the whole network may make different DNN layers antagonistically affected each other, thus degrading the final performance. In this paper, we propose to separate a DNN into different parts and progressively train them to address this problem. Instead of the early stopping, which trains a whole DNN all at once, we initially train former DNN layers by optimizing the DNN with a relatively large number of epochs. During training, we progressively train the latter DNN layers by using a smaller number of epochs with the preceding layers fixed to counteract the impact of noisy labels. We term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with the early stopping, PES can help to obtain more promising and stable results. Furthermore, by combining PES with existing approaches on noisy label training, we achieve state-of-the-art performance on image classification benchmarks.

READ FULL TEXT
11/19/2019

Prestopping: How Does Early Stopping Help Generalization against Label Noise?

Noisy labels are very common in real-world training data, which lead to ...
12/07/2022

PADDLES: Phase-Amplitude Spectrum Disentangled Early Stopping for Learning with Noisy Labels

Convolutional Neural Networks (CNNs) have demonstrated superiority in le...
10/20/2019

Leveraging inductive bias of neural networks for learning without explicit human annotations

Classification problems today are typically solved by first collecting e...
12/23/2020

Noisy Labels Can Induce Good Representations

The current success of deep learning depends on large-scale labeled data...
09/30/2022

Effective Early Stopping of Point Cloud Neural Networks

Early stopping techniques can be utilized to decrease the time cost, how...
09/28/2018

Pumpout: A Meta Approach for Robustly Training Deep Neural Networks with Noisy Labels

It is challenging to train deep neural networks robustly on the industri...
08/19/2022

Intersection of Parallels as an Early Stopping Criterion

A common way to avoid overfitting in supervised learning is early stoppi...