Towards calibrated and scalable uncertainty representations for neural networks

10/28/2019
by   Nabeel Seedat, et al.
12

For many applications it is critical to know the uncertainty of a neural network's predictions. While a variety of neural network parameter estimation methods have been proposed for uncertainty estimation, they have not been rigorously compared across uncertainty measures. We assess four of these parameter estimation methods to calibrate uncertainty estimation using four different uncertainty measures: entropy, mutual information, aleatoric uncertainty and epistemic uncertainty. We also evaluate their calibration using expected calibration error. We additionally propose a novel method of neural network parameter estimation called RECAST, which combines cosine annealing with warm restarts with Stochastic Gradient Langevin Dynamics, capturing more diverse parameter distributions. When benchmarked against mutilated data from MNIST, we show that RECAST is well-calibrated and when combined with predictive entropy and epistemic uncertainty it offers the best calibrated measure of uncertainty when compared to recent methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/24/2022

Analytic Mutual Information in Bayesian Neural Networks

Bayesian neural networks have successfully designed and optimized a robu...
research
12/02/2021

Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning

Within the last few years, there has been a move towards using statistic...
research
09/12/2023

Epistemic Modeling Uncertainty of Rapid Neural Network Ensembles for Adaptive Learning

Emulator embedded neural networks, which are a type of physics informed ...
research
02/18/2022

Model Calibration of the Liquid Mercury Spallation Target using Evolutionary Neural Networks and Sparse Polynomial Expansions

The mercury constitutive model predicting the strain and stress in the t...
research
09/22/2021

A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation

Uncertainty assessment has gained rapid interest in medical image analys...
research
12/07/2021

Accurate parameter estimation using scan-specific unsupervised deep learning for relaxometry and MR fingerprinting

We propose an unsupervised convolutional neural network (CNN) for relaxa...

Please sign up or login with your details

Forgot password? Click here to reset