DeepAI AI Chat
Log In Sign Up

Improving Regression Uncertainty Estimates with an Empirical Prior

05/26/2020
by   Eric Zelikman, et al.
14

While machine learning models capable of producing uncertainty estimates are becoming widespread, uncalibrated uncertainty estimates are often overconfident, and often assume predetermined probability distributions over the error which do not match the empirical calibration error. Most work on calibrating uncertainty estimates focuses on classification rather than regression, which introduces novel challenges. We present a calibration method referred to as Calibrating Regression Uncertainty Distributions Empirically (CRUDE) that does not assume a fixed uncertainty distribution, instead making the weaker assumption that error distributions have a constant shape across the output space, shifted by predicted mean and scaled by predicted standard deviation. CRUDE requires no training of the calibration estimator aside from a parameter to account for consistent bias in the predicted mean. Our method is distribution-agnostic and provides sharper and more accurate uncertainty estimates than state of the art techniques, as demonstrated by calibration and sharpness measures across many datasets.

READ FULL TEXT

page 4

page 5

10/21/2022

Calibration tests beyond classification

Most supervised machine learning tasks are subject to irreducible predic...
10/13/2022

A Consistent and Differentiable Lp Canonical Calibration Error Estimator

Calibrated probabilistic classifiers are models whose predicted probabil...
03/09/2023

Probabilistic 3d regression with projected huber distribution

Estimating probability distributions which describe where an object is l...
07/17/2022

Uncertainty Calibration in Bayesian Neural Networks via Distance-Aware Priors

As we move away from the data, the predictive uncertainty should increas...
05/10/2021

Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions

Uncertainty awareness is crucial to develop reliable machine learning mo...
11/23/2021

Uncertainty estimation under model misspecification in neural network regression

Although neural networks are powerful function approximators, the underl...
05/15/2019

Distribution Calibration for Regression

We are concerned with obtaining well-calibrated output distributions fro...