Why do classifier accuracies show linear trends under distribution shift?

12/31/2020
by   Horia Mania, et al.
0

Several recent studies observed that when classification models are evaluated on two different data distributions, the models' accuracies on one distribution are approximately a linear function of their accuracies on another distribution. We offer an explanation for these observations based on two assumptions that can be assessed empirically: (1) certain events have similar probabilities under the two distributions; (2) the probability that a lower accuracy model correctly classifies a data point sampled from one distribution when a higher accuracy model classifies it incorrectly is small.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/27/2017

Extrapolating Expected Accuracies for Large Multi-Class Problems

The difficulty of multi-class classification generally increases with th...
research
06/30/2021

The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

Although machine learning models typically experience a drop in performa...
research
10/28/2019

Generative Well-intentioned Networks

We propose Generative Well-intentioned Networks (GWINs), a novel framewo...
research
06/05/2018

On Latent Distributions Without Finite Mean in Generative Models

We investigate the properties of multidimensional probability distributi...
research
11/21/2022

First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains

Real-world machine learning applications often involve deploying neural ...
research
06/10/2022

Lightweight Conditional Model Extrapolation for Streaming Data under Class-Prior Shift

We introduce LIMES, a new method for learning with non-stationary stream...
research
03/11/2021

Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models

Recent studies indicate that NLU models are prone to rely on shortcut fe...

Please sign up or login with your details

Forgot password? Click here to reset