Simulation-Based Inference with WALDO: Perfectly Calibrated Confidence Regions Using Any Prediction or Posterior Estimation Algorithm

05/31/2022
by   Luca Masserano, et al.
0

The vast majority of modern machine learning targets prediction problems, with algorithms such as Deep Neural Networks revolutionizing the accuracy of point predictions for high-dimensional complex data. Predictive approaches are now used in many domain sciences to directly estimate internal parameters of interest in theoretical simulator-based models. In parallel, common alternatives focus on estimating the full posterior using modern neural density estimators such as normalizing flows. However, an open problem in simulation-based inference (SBI) is how to construct properly calibrated confidence regions for internal parameters with nominal conditional coverage and high power. Many SBI methods are indeed known to produce overly confident posterior approximations, yielding misleading uncertainty estimates. Similarly, existing approaches for uncertainty quantification in deep learning provide no guarantees on conditional coverage. In this work, we present WALDO, a novel method for constructing correctly calibrated confidence regions in SBI. WALDO reframes the well-known Wald test and uses Neyman inversion to convert point predictions and posteriors from any prediction or posterior estimation algorithm to confidence sets with correct conditional coverage, even for finite sample sizes. As a concrete example, we demonstrate how a recently proposed deep learning prediction approach for particle energies in high-energy physics can be recalibrated using WALDO to produce confidence intervals with correct coverage and high power.

READ FULL TEXT
research
06/09/2023

Conformalizing Machine Translation Evaluation

Several uncertainty estimation methods have been recently proposed for m...
research
05/29/2022

Calibrated Predictive Distributions via Diagnostics for Conditional Coverage

Uncertainty quantification is crucial for assessing the predictive abili...
research
06/11/2021

Conformal Bayesian Computation

We develop scalable methods for producing conformal Bayesian predictive ...
research
06/13/2019

Selective prediction-set models with coverage guarantees

Though black-box predictors are state-of-the-art for many complex tasks,...
research
09/09/2019

Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors

With rapid adoption of deep learning in high-regret applications, the qu...
research
11/01/2021

Uncertainty quantification for wide-bin unfolding: one-at-a-time strict bounds and prior-optimized confidence intervals

Unfolding is an ill-posed inverse problem in particle physics aiming to ...
research
03/11/2022

Distribution-free Prediction Sets Adaptive to Unknown Covariate Shift

Predicting sets of outcomes – instead of unique outcomes – is a promisin...

Please sign up or login with your details

Forgot password? Click here to reset