DeepAI AI Chat
Log In Sign Up

Evaluating and Calibrating Uncertainty Prediction in Regression Tasks

05/28/2019
by   Dan Levi, et al.
UNIVERSITY OF TORONTO
General Motors
0

Predicting not only the target but also an accurate measure of uncertainty is important for many applications and in particular safety-critical ones. In this work we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing approach for evaluating the calibration of a regression uncertainty [Kuleshov et al. 2018] has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new evaluation method that escapes this caveat using a simple histogram-based approach inspired by reliability diagrams used in classification tasks. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple scaling-based calibration that preforms well in our experimental tests. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO and KITTI datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/19/2019

Evaluating model calibration in classification

Probabilistic classifiers output a probability distribution on target cl...
06/29/2020

Unsupervised Calibration under Covariate Shift

A probabilistic model is said to be calibrated if its predicted probabil...
06/07/2021

Improved Predictive Uncertainty using Corruption-based Calibration

We propose a simple post hoc calibration method to estimate the confiden...
06/22/2020

Towards Better Performance and More Explainable Uncertainty for 3D Object Detection of Autonomous Vehicles

In this paper, we propose a novel form of the loss function to increase ...
02/25/2022

Confidence Calibration for Object Detection and Segmentation

Calibrated confidence estimates obtained from neural networks are crucia...
12/23/2020

Exploring Instance-Level Uncertainty for Medical Detection

The ability of deep learning to predict with uncertainty is recognized a...
05/23/2022

Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees

We propose Instance-Based Uncertainty estimation for Gradient-boosted re...