DeepAI AI Chat
Log In Sign Up

Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels

by   Pengfei Chen, et al.

For multi-class classification under class-conditional label noise, we prove that the accuracy metric itself can be robust. We concretize this finding's inspiration in two essential aspects: training and validation, with which we address critical issues in learning with noisy labels. For training, we show that maximizing training accuracy on sufficiently many noisy samples yields an approximately optimal classifier. For validation, we prove that a noisy validation set is reliable, addressing the critical demand of model selection in scenarios like hyperparameter-tuning and early stopping. Previously, model selection using noisy validation samples has not been theoretically justified. We verify our theoretical results and additional claims with extensive experiments. We show characterizations of models trained with noisy labels, motivated by our theoretical results, and verify the utility of a noisy validation set by showing the impressive performance of a framework termed noisy best teacher and student (NTS). Our code is released.


Prestopping: How Does Early Stopping Help Generalization against Label Noise?

Noisy labels are very common in real-world training data, which lead to ...

Improving group robustness under noisy labels using predictive uncertainty

The standard empirical risk minimization (ERM) can underperform on certa...

Train on Validation: Squeezing the Data Lemon

Model selection on validation data is an essential step in machine learn...

Private Semi-supervised Knowledge Transfer for Deep Learning from Noisy Labels

Deep learning models trained on large-scale data have achieved encouragi...

Counterfactual Cross-Validation: Effective Causal Model Selection from Observational Data

What is the most effective way to select the best causal model among pot...

Combining Distance to Class Centroids and Outlier Discounting for Improved Learning with Noisy Labels

In this paper, we propose a new approach for addressing the challenge of...

Dependency Leakage: Analysis and Scalable Estimators

In this paper, we prove the first theoretical results on dependency leak...