Online Multivalid Learning: Means, Moments, and Prediction Intervals

by   Varun Gupta, et al.

We present a general, efficient technique for providing contextual predictions that are "multivalid" in various senses, against an online sequence of adversarially chosen examples (x,y). This means that the resulting estimates correctly predict various statistics of the labels y not just marginally – as averaged over the sequence of examples – but also conditionally on x ∈ G for any G belonging to an arbitrary intersecting collection of groups 𝒢. We provide three instantiations of this framework. The first is mean prediction, which corresponds to an online algorithm satisfying the notion of multicalibration from Hebert-Johnson et al. The second is variance and higher moment prediction, which corresponds to an online algorithm satisfying the notion of mean-conditioned moment multicalibration from Jung et al. Finally, we define a new notion of prediction interval multivalidity, and give an algorithm for finding prediction intervals which satisfy it. Because our algorithms handle adversarially chosen examples, they can equally well be used to predict statistics of the residuals of arbitrary point prediction methods, giving rise to very general techniques for quantifying the uncertainty of predictions of black box algorithms, even in an online adversarial setting. When instantiated for prediction intervals, this solves a similar problem as conformal prediction, but in an adversarial environment and with multivalidity guarantees stronger than simple marginal coverage guarantees.


page 1

page 2

page 3

page 4


Moment Multicalibration for Uncertainty Estimation

We show how to achieve the notion of "multicalibration" from Hébert-John...

Any-Order Online Interval Selection

We consider the problem of online interval scheduling on a single machin...

Conformal Inference for Online Prediction with Arbitrary Distribution Shifts

Conformal inference is a flexible methodology for transforming the predi...

Interpretable Machines: Constructing Valid Prediction Intervals with Random Forests

An important issue when using Machine Learning algorithms in recent rese...

Model-free Bootstrap and Conformal Prediction in Regression: Conditionality, Conjecture Testing, and Pertinent Prediction Intervals

Predictive inference under a general regression setting is gaining more ...

Game-Theoretic Algorithms for Conditional Moment Matching

A variety of problems in econometrics and machine learning, including in...

A Theory of Selective Prediction

We consider a model of selective prediction, where the prediction algori...

Please sign up or login with your details

Forgot password? Click here to reset