A Unified View of Label Shift Estimation

by   Saurabh Garg, et al.

Label shift describes the setting where although the label distribution might change between the source and target domains, the class-conditional probabilities (of data given a label) do not. There are two dominant approaches for estimating the label marginal. BBSE, a moment-matching approach based on confusion matrices, is provably consistent and provides interpretable error bounds. However, a maximum likelihood estimation approach, which we call MLLS, dominates empirically. In this paper, we present a unified view of the two methods and the first theoretical characterization of the likelihood-based estimator. Our contributions include (i) conditions for consistency of MLLS, which include calibration of the classifier and a confusion matrix invertibility condition that BBSE also requires; (ii) a unified view of the methods, casting the confusion matrix as roughly equivalent to MLLS for a particular choice of calibration method; and (iii) a decomposition of MLLS's finite-sample error into terms reflecting the impacts of miscalibration and estimation error. Our analysis attributes BBSE's statistical inefficiency to a loss of information due to coarse calibration. We support our findings with experiments on both synthetic data and the MNIST and CIFAR10 image recognition datasets.



There are no comments yet.


page 1

page 2

page 3

page 4


Finite-sample risk bounds for maximum likelihood estimation with arbitrary penalties

The MDL two-part coding index of resolvability provides a finite-sampl...

Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift

Adversarial learning has demonstrated good performance in the unsupervis...

Fast Rates for a kNN Classifier Robust to Unknown Asymmetric Label Noise

We consider classification in the presence of class-dependent asymmetric...

Convergence Rates of Active Learning for Maximum Likelihood Estimation

An active learner is given a class of models, a large set of unlabeled e...

On maximum-likelihood estimation in the all-or-nothing regime

We study the problem of estimating a rank-1 additive deformation of a Ga...

Calibration for computer experiments with binary responses

Calibration refers to the estimation of unknown parameters which are pre...

Self-training Avoids Using Spurious Features Under Domain Shift

In unsupervised domain adaptation, existing theory focuses on situations...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.