Conformal prediction beyond exchangeability

02/27/2022
by   Rina Foygel Barber, et al.
0

Conformal prediction is a popular, modern technique for providing valid predictive inference for arbitrary machine learning models. Its validity relies on the assumptions of exchangeability of the data, and symmetry of the given model fitting algorithm as a function of the data. However, exchangeability is often violated when predictive models are deployed in practice. For example, if the data distribution drifts over time, then the data points are no longer exchangeable; moreover, in such settings, we might want to use an algorithm that treats recent observations as more relevant, which would violate the assumption that data points are treated symmetrically. This paper proposes new methodology to deal with both aspects: we use weighted quantiles to introduce robustness against distribution drift, and design a new technique to allow for algorithms that do not treat data points symmetrically. Our algorithms are provably robust, with substantially less loss of coverage when exchangeability is violated due to distribution drift or other challenging features of real data, while also achieving the same coverage guarantees as existing conformal prediction methods if the data points are in fact exchangeable. Finally, we demonstrate the practical utility of these new tools with simulations and real-data experiments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/02/2023

Design-based conformal prediction

Conformal prediction is an assumption-lean approach to generating distri...
research
03/12/2019

The limits of distribution-free conditional predictive inference

We consider the problem of distribution-free predictive inference, with ...
research
07/17/2018

Analyzing Hypersensitive AI: Instability in Corporate-Scale Machine Learning

Predictive geometric models deliver excellent results for many Machine L...
research
03/30/2018

On inference validity of weighted U-statistics under data heterogeneity

Motivated by challenges on studying a new correlation measurement being ...
research
11/21/2017

Generating Analytic Insights on Human Behaviour using Image Processing

This paper proposes a method to track human figures in physical spaces a...
research
01/20/2022

Predictive Inference with Weak Supervision

The expense of acquiring labels in large-scale statistical machine learn...
research
08/17/2022

Conformal Inference for Online Prediction with Arbitrary Distribution Shifts

Conformal inference is a flexible methodology for transforming the predi...

Please sign up or login with your details

Forgot password? Click here to reset