Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

07/09/2021
by   John Miller, et al.
0

For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 ImageNet, a synthetic pose estimation task derived from YCB objects, satellite imagery classification in FMoW-WILDS, and wildlife classification in iWildCam-WILDS. The strong correlations hold across model architectures, hyperparameters, training set size, and training duration, and are more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/07/2021

Predicting with Confidence on Unseen Distributions

Recent work has shown that the performance of machine learning models ca...
research
05/04/2023

On the nonlinear correlation of ML performance between data subpopulations

Understanding the performance of machine learning (ML) models across div...
research
02/20/2022

Deconstructing Distributions: A Pointwise Framework of Learning

In machine learning, we traditionally evaluate the performance of a sing...
research
05/09/2023

Even Small Correlation and Diversity Shifts Pose Dataset-Bias Issues

Distribution shifts are common in real-world datasets and can affect the...
research
03/03/2023

Diagnosing Model Performance Under Distribution Shift

Prediction models can perform poorly when deployed to target distributio...
research
07/06/2023

Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification

Test-time adaptation (TTA) is a technique aimed at enhancing the general...
research
10/20/2022

Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization

Machine learning systems are often applied to data that is drawn from a ...

Please sign up or login with your details

Forgot password? Click here to reset