Why do classifier accuracies show linear trends under distribution shift?

12/31/2020
by   Horia Mania, et al.
0

Several recent studies observed that when classification models are evaluated on two different data distributions, the models' accuracies on one distribution are approximately a linear function of their accuracies on another distribution. We offer an explanation for these observations based on two assumptions that can be assessed empirically: (1) certain events have similar probabilities under the two distributions; (2) the probability that a lower accuracy model correctly classifies a data point sampled from one distribution when a higher accuracy model classifies it incorrectly is small.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

12/27/2017

Extrapolating Expected Accuracies for Large Multi-Class Problems

The difficulty of multi-class classification generally increases with th...
06/30/2021

The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

Although machine learning models typically experience a drop in performa...
10/28/2019

Generative Well-intentioned Networks

We propose Generative Well-intentioned Networks (GWINs), a novel framewo...
06/05/2018

On Latent Distributions Without Finite Mean in Generative Models

We investigate the properties of multidimensional probability distributi...
10/27/2020

Selective Classification Can Magnify Disparities Across Groups

Selective classification, in which models are allowed to abstain on unce...
12/19/2019

Causal statistical modeling and calculation of distribution functions of classification features

Statistical system models provide the basis for the examination of vario...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.