Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

10/03/2021
by   Anastasios N. Angelopoulos, et al.
44

We introduce Learn then Test, a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees regardless of the underlying model and (unknown) data-generating distribution. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. To accomplish this, we solve a key technical challenge: the control of arbitrary risks that are not necessarily monotonic. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use our framework to provide new calibration methods for several core machine learning tasks with detailed worked examples in computer vision.

READ FULL TEXT

page 2

page 10

page 15

page 17

04/16/2021

Testing for Outliers with Conformal p-values

This paper studies the construction of p-values for nonparametric outlie...
01/07/2021

Distribution-Free, Risk-Controlling Prediction Sets

While improving prediction accuracy has been the focus of machine learni...
01/03/2019

Instance-Based Classification through Hypothesis Testing

Classification is a fundamental problem in machine learning and data min...
03/29/2019

Interpreting Black Box Models with Statistical Guarantees

While many methods for interpreting machine learning models have been pr...
04/03/2019

DiscreteFDR: An R package for controlling the false discovery rate for discrete test statistics

The simultaneous analysis of many statistical tests is ubiquitous in app...
05/19/2016

False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking

With the rapid growth of crowdsourcing platforms it has become easy and ...
12/28/2019

Approval policies for modifications to Machine Learning-Based Software as a Medical Device: A study of bio-creep

Successful deployment of machine learning algorithms in healthcare requi...