Improving model calibration with accuracy versus uncertainty optimization

12/14/2020
by   Ranganath Krishnan, et al.
12

Obtaining reliable and accurate quantification of uncertainty estimates from deep neural networks is important in safety-critical applications. A well-calibrated model should be accurate when it is certain about its prediction and indicate high uncertainty when it is likely to be inaccurate. Uncertainty calibration is a challenging problem as there is no ground truth available for uncertainty estimates. We propose an optimization method that leverages the relationship between accuracy and uncertainty as an anchor for uncertainty calibration. We introduce a differentiable accuracy versus uncertainty calibration (AvUC) loss function that allows a model to learn to provide well-calibrated uncertainties, in addition to improved accuracy. We also demonstrate the same methodology can be extended to post-hoc uncertainty calibration on pretrained models. We illustrate our approach with mean-field stochastic variational inference and compare with state-of-the-art methods. Extensive experiments demonstrate our approach yields better model calibration than existing methods on large-scale image classification tasks under distributional shift.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 7

page 13

page 18

page 19

09/21/2021

Bayesian Confidence Calibration for Epistemic Uncertainty Modelling

Modern neural networks have found to be miscalibrated in terms of confid...
10/30/2019

Heteroscedastic Calibration of Uncertainty Estimators in Deep Learning

The role of uncertainty quantification (UQ) in deep learning has become ...
07/03/2020

Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain

Label disagreement between human experts is a common issue in the medica...
01/08/2021

Approaching Neural Network Uncertainty Realism

Statistical models are inherently uncertain. Quantifying or at least upp...
08/23/2019

Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks

Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy perf...
03/16/2020

Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning

This paper studies the problem of post-hoc calibration of machine learni...
05/28/2019

Evaluating and Calibrating Uncertainty Prediction in Regression Tasks

Predicting not only the target but also an accurate measure of uncertain...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.