Robust Validation: Confident Predictions Even When Distributions Shift

08/10/2020
by   Maxime Cauchois, et al.
0

While the traditional viewpoint in machine learning and statistics assumes training and testing samples come from the same population, practice belies this fiction. One strategy—coming from robust statistics and optimization—is thus to build a model robust to distributional perturbations. In this paper, we take a different approach to describe procedures for robust predictive inference, where a model provides uncertainty estimates on its predictions rather than point predictions. We present a method that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an f-divergence ball around the training population. The method, based on conformal inference, achieves (nearly) valid coverage in finite samples, under only the condition that the training data be exchangeable. An essential component of our methodology is to estimate the amount of expected future data shift and build robustness to it; we develop estimators and prove their consistency for protection and validity of uncertainty estimates under shifts. By experimenting on several large-scale benchmark datasets, including Recht et al.'s CIFAR-v4 and ImageNet-V2 datasets, we provide complementary empirical results that highlight the importance of robust predictive validity.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/19/2021

Robustness via Cross-Domain Ensembles

We present a method for making neural network predictions robust to shif...
research
11/23/2021

Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach

We propose a model-free framework for sensitivity analysis of individual...
research
07/07/2021

Predicting with Confidence on Unseen Distributions

Recent work has shown that the performance of machine learning models ca...
research
06/01/2021

Adaptive Conformal Inference Under Distribution Shift

We develop methods for forming prediction sets in an online setting wher...
research
05/22/2022

Robust Flow-based Conformal Inference (FCI) with Statistical Guarantee

Conformal prediction aims to determine precise levels of confidence in p...
research
01/20/2022

Predictive Inference with Weak Supervision

The expense of acquiring labels in large-scale statistical machine learn...
research
08/17/2022

Conformal Inference for Online Prediction with Arbitrary Distribution Shifts

Conformal inference is a flexible methodology for transforming the predi...

Please sign up or login with your details

Forgot password? Click here to reset