Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

by   Anastasios N. Angelopoulos, et al.

We introduce Learn then Test, a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees regardless of the underlying model and (unknown) data-generating distribution. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. To accomplish this, we solve a key technical challenge: the control of arbitrary risks that are not necessarily monotonic. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use our framework to provide new calibration methods for several core machine learning tasks with detailed worked examples in computer vision.


page 2

page 10

page 15

page 17


Testing for Outliers with Conformal p-values

This paper studies the construction of p-values for nonparametric outlie...

Distribution-Free, Risk-Controlling Prediction Sets

While improving prediction accuracy has been the focus of machine learni...

Instance-Based Classification through Hypothesis Testing

Classification is a fundamental problem in machine learning and data min...

Interpreting Black Box Models with Statistical Guarantees

While many methods for interpreting machine learning models have been pr...

DiscreteFDR: An R package for controlling the false discovery rate for discrete test statistics

The simultaneous analysis of many statistical tests is ubiquitous in app...

False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking

With the rapid growth of crowdsourcing platforms it has become easy and ...

Approval policies for modifications to Machine Learning-Based Software as a Medical Device: A study of bio-creep

Successful deployment of machine learning algorithms in healthcare requi...