Towards calibrated and scalable uncertainty representations for neural networks

10/28/2019
by   Nabeel Seedat, et al.
12

For many applications it is critical to know the uncertainty of a neural network's predictions. While a variety of neural network parameter estimation methods have been proposed for uncertainty estimation, they have not been rigorously compared across uncertainty measures. We assess four of these parameter estimation methods to calibrate uncertainty estimation using four different uncertainty measures: entropy, mutual information, aleatoric uncertainty and epistemic uncertainty. We also evaluate their calibration using expected calibration error. We additionally propose a novel method of neural network parameter estimation called RECAST, which combines cosine annealing with warm restarts with Stochastic Gradient Langevin Dynamics, capturing more diverse parameter distributions. When benchmarked against mutilated data from MNIST, we show that RECAST is well-calibrated and when combined with predictive entropy and epistemic uncertainty it offers the best calibrated measure of uncertainty when compared to recent methods.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

04/26/2021

Recalibration of Aleatoric and Epistemic Regression Uncertainty in Medical Imaging

The consideration of predictive uncertainty in medical imaging with deep...
12/05/2020

Quantifying Aleatoric and Epistemic Uncertainty Using Density Estimation in Latent Space

The distribution of a neural network's latent representations has been s...
12/02/2021

Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning

Within the last few years, there has been a move towards using statistic...
09/22/2021

A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation

Uncertainty assessment has gained rapid interest in medical image analys...
09/20/2019

Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Bias

Supporting model interpretability for complex phenomena where annotators...
03/05/2019

Evaluation of Neural Network Uncertainty Estimation with Application to Resource-Constrained Platforms

The ability to accurately estimate uncertainties in neural network predi...
02/22/2021

Improving Uncertainty Calibration via Prior Augmented Data

Neural networks have proven successful at learning from complex data dis...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.