
Moment Multicalibration for Uncertainty Estimation
We show how to achieve the notion of "multicalibration" from HébertJohn...
read it

Interpretable Machines: Constructing Valid Prediction Intervals with Random Forests
An important issue when using Machine Learning algorithms in recent rese...
read it

Attenuate Locally, Win Globally: An Attenuationbased Framework for Online Stochastic Matching with Timeouts
Online matching problems have garnered significant attention in recent y...
read it

Toward Finding The Global Optimal of Adversarial Examples
Current machine learning models are vulnerable to adversarial examples (...
read it

A comparison of some conformal quantile regression methods
We compare two recently proposed methods that combine ideas from conform...
read it

Beyond Perturbations: Learning Guarantees with Arbitrary Adversarial Test Examples
We present a transductive learning algorithm that takes as input trainin...
read it

Network Moments: Extensions and SparseSmooth Attacks
The impressive performance of deep neural networks (DNNs) has immensely ...
read it
Online Multivalid Learning: Means, Moments, and Prediction Intervals
We present a general, efficient technique for providing contextual predictions that are "multivalid" in various senses, against an online sequence of adversarially chosen examples (x,y). This means that the resulting estimates correctly predict various statistics of the labels y not just marginally – as averaged over the sequence of examples – but also conditionally on x ∈ G for any G belonging to an arbitrary intersecting collection of groups 𝒢. We provide three instantiations of this framework. The first is mean prediction, which corresponds to an online algorithm satisfying the notion of multicalibration from HebertJohnson et al. The second is variance and higher moment prediction, which corresponds to an online algorithm satisfying the notion of meanconditioned moment multicalibration from Jung et al. Finally, we define a new notion of prediction interval multivalidity, and give an algorithm for finding prediction intervals which satisfy it. Because our algorithms handle adversarially chosen examples, they can equally well be used to predict statistics of the residuals of arbitrary point prediction methods, giving rise to very general techniques for quantifying the uncertainty of predictions of black box algorithms, even in an online adversarial setting. When instantiated for prediction intervals, this solves a similar problem as conformal prediction, but in an adversarial environment and with multivalidity guarantees stronger than simple marginal coverage guarantees.
READ FULL TEXT
Comments
There are no comments yet.