DeepAI AI Chat
Log In Sign Up

Evaluating and Calibrating Uncertainty Prediction in Regression Tasks

by   Dan Levi, et al.
General Motors

Predicting not only the target but also an accurate measure of uncertainty is important for many applications and in particular safety-critical ones. In this work we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing approach for evaluating the calibration of a regression uncertainty [Kuleshov et al. 2018] has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new evaluation method that escapes this caveat using a simple histogram-based approach inspired by reliability diagrams used in classification tasks. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple scaling-based calibration that preforms well in our experimental tests. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO and KITTI datasets.


page 1

page 2

page 3

page 4


Evaluating model calibration in classification

Probabilistic classifiers output a probability distribution on target cl...

Unsupervised Calibration under Covariate Shift

A probabilistic model is said to be calibrated if its predicted probabil...

Improved Predictive Uncertainty using Corruption-based Calibration

We propose a simple post hoc calibration method to estimate the confiden...

Towards Better Performance and More Explainable Uncertainty for 3D Object Detection of Autonomous Vehicles

In this paper, we propose a novel form of the loss function to increase ...

Confidence Calibration for Object Detection and Segmentation

Calibrated confidence estimates obtained from neural networks are crucia...

Exploring Instance-Level Uncertainty for Medical Detection

The ability of deep learning to predict with uncertainty is recognized a...

Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees

We propose Instance-Based Uncertainty estimation for Gradient-boosted re...