Distribution-free uncertainty quantification for classification under label shift

03/04/2021
by   Aleksandr Podkopaev, et al.
0

Trustworthy deployment of ML models requires a proper measure of uncertainty, especially in safety-critical applications. We focus on uncertainty quantification (UQ) for classification problems via two avenues – prediction sets using conformal prediction and calibration of probabilistic predictors by post-hoc binning – since these possess distribution-free guarantees for i.i.d. data. Two common ways of generalizing beyond the i.i.d. setting include handling covariate and label shift. Within the context of distribution-free UQ, the former has already received attention, but not the latter. It is known that label shift hurts prediction, and we first argue that it also hurts UQ, by showing degradation in coverage and calibration. Piggybacking on recent progress in addressing label shift (for better prediction), we examine the right way to achieve UQ by reweighting the aforementioned conformal and calibration procedures whenever some unlabeled data from the target distribution is available. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.

READ FULL TEXT

page 5

page 9

page 16

page 17

page 24

research
10/06/2020

Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures

Uncertainty quantification for complex deep learning models is increasin...
research
07/21/2022

JAWS: Predictive Inference Under Covariate Shift

We propose JAWS, a series of wrapper methods for distribution-free uncer...
research
06/29/2020

Unsupervised Calibration under Covariate Shift

A probabilistic model is said to be calibrated if its predicted probabil...
research
11/07/2022

A Semiparametric Efficient Approach To Label Shift Estimation and Quantification

Transfer Learning is an area of statistics and machine learning research...
research
12/14/2022

Post-hoc Uncertainty Learning using a Dirichlet Meta-Model

It is known that neural networks have the problem of being over-confiden...
research
01/07/2021

Distribution-Free, Risk-Controlling Prediction Sets

While improving prediction accuracy has been the focus of machine learni...
research
06/07/2023

Label Shift Quantification with Robustness Guarantees via Distribution Feature Matching

Quantification learning deals with the task of estimating the target lab...

Please sign up or login with your details

Forgot password? Click here to reset