Honest calibration assessment for binary outcome predictions

03/08/2022
by   Timo Dimitriadis, et al.
0

Probability predictions from binary regressions or machine learning methods ought to be calibrated: If an event is predicted to occur with probability x, it should materialize with approximately that frequency, which means that the so-called calibration curve p(x) should equal the bisector for all x in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid only subject to the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well specified model. We show that our bands have a finite sample coverage guarantee, are narrower than existing approaches, and adapt to the local smoothness and variance of the calibration curve p. In an application to model predictions of an infant having a low birth weight, the bounds give informative insights on model calibration.

READ FULL TEXT
research
05/05/2023

Calibration Assessment and Boldness-Recalibration for Binary Events

Probability predictions are essential to inform decision making in medic...
research
01/13/2014

Binary Classifier Calibration: Bayesian Non-Parametric Approach

A set of probabilistic predictions is well calibrated if the events that...
research
10/24/2019

Calibration tests in multi-class classification: A unifying framework

In safety-critical applications a probabilistic model is usually require...
research
08/25/2023

Calibration plots for multistate risk predictions models: an overview and simulation comparing novel approaches

Introduction. There is currently no guidance on how to assess the calibr...
research
09/15/2021

Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets

For interpreting the behavior of a probabilistic model, it is useful to ...
research
06/05/2023

Significance Bands for Local Projections

An impulse response function describes the dynamic evolution of an outco...
research
02/07/2020

Temporal Probability Calibration

In many applications, accurate class probability estimates are required,...

Please sign up or login with your details

Forgot password? Click here to reset