Approximating Full Conformal Prediction at Scale via Influence Functions

02/02/2022
by   Javier Abad, et al.
0

Conformal prediction (CP) is a wrapper around traditional machine learning models, giving coverage guarantees under the sole assumption of exchangeability; in classification problems, for a chosen significance level ε, CP guarantees that the number of errors is at most ε, irrespective of whether the underlying model is misspecified. However, the prohibitive computational costs of full CP led researchers to design scalable alternatives, which alas do not attain the same guarantees or statistical power of full CP. In this paper, we use influence functions to efficiently approximate full CP. We prove that our method is a consistent approximation of full CP, and empirically show that the approximation error becomes smaller as the training set increases; e.g., for 10^3 training points the two methods output p-values that are <10^-3 apart: a negligible error for any practical application. Our methods enable scaling full CP to large real-world datasets. We compare our full CP approximation ACP to mainstream CP alternatives, and observe that our method is computationally competitive whilst enjoying the statistical predictive power of full CP.

READ FULL TEXT

page 17

page 18

research
07/25/2022

A novel Deep Learning approach for one-step Conformal Prediction approximation

Deep Learning predictions with measurable confidence are increasingly de...
research
03/29/2022

Split Conformal Prediction for Dependent Data

Split conformal prediction is a popular tool to obtain predictive interv...
research
02/05/2021

Exact Optimization of Conformal Predictors via Incremental and Decremental Learning

Conformal Predictors (CP) are wrappers around ML methods, providing erro...
research
05/11/2020

Nearest Neighbor Classifiers over Incomplete Information: From Certain Answers to Certain Predictions

Machine learning (ML) applications have been thriving recently, largely ...
research
06/13/2016

The Crossover Process: Learnability and Data Protection from Inference Attacks

It is usual to consider data protection and learnability as conflicting ...
research
10/18/2021

Learning Optimal Conformal Classifiers

Modern deep learning based classifiers show very high accuracy on test d...
research
08/09/2019

Concepts and Applications of Conformal Prediction in Computational Drug Discovery

Estimating the reliability of individual predictions is key to increase ...

Please sign up or login with your details

Forgot password? Click here to reset