DeepAI AI Chat
Log In Sign Up

Improving Regression Uncertainty Estimates with an Empirical Prior

by   Eric Zelikman, et al.

While machine learning models capable of producing uncertainty estimates are becoming widespread, uncalibrated uncertainty estimates are often overconfident, and often assume predetermined probability distributions over the error which do not match the empirical calibration error. Most work on calibrating uncertainty estimates focuses on classification rather than regression, which introduces novel challenges. We present a calibration method referred to as Calibrating Regression Uncertainty Distributions Empirically (CRUDE) that does not assume a fixed uncertainty distribution, instead making the weaker assumption that error distributions have a constant shape across the output space, shifted by predicted mean and scaled by predicted standard deviation. CRUDE requires no training of the calibration estimator aside from a parameter to account for consistent bias in the predicted mean. Our method is distribution-agnostic and provides sharper and more accurate uncertainty estimates than state of the art techniques, as demonstrated by calibration and sharpness measures across many datasets.


page 4

page 5


Calibration tests beyond classification

Most supervised machine learning tasks are subject to irreducible predic...

A Consistent and Differentiable Lp Canonical Calibration Error Estimator

Calibrated probabilistic classifiers are models whose predicted probabil...

Probabilistic 3d regression with projected huber distribution

Estimating probability distributions which describe where an object is l...

Uncertainty Calibration in Bayesian Neural Networks via Distance-Aware Priors

As we move away from the data, the predictive uncertainty should increas...

Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions

Uncertainty awareness is crucial to develop reliable machine learning mo...

Uncertainty estimation under model misspecification in neural network regression

Although neural networks are powerful function approximators, the underl...

Distribution Calibration for Regression

We are concerned with obtaining well-calibrated output distributions fro...