Revisiting the Calibration of Modern Neural Networks

06/15/2021
by   Matthias Minderer, et al.
18

Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.

READ FULL TEXT

page 25

page 26

06/18/2020

Calibrated Reliable Regression using Maximum Mean Discrepancy

Accurate quantification of uncertainty is crucial for real-world applica...
12/14/2020

Improving model calibration with accuracy versus uncertainty optimization

Obtaining reliable and accurate quantification of uncertainty estimates ...
02/04/2015

Artificial neural networks in calibration of nonlinear mechanical models

Rapid development in numerical modelling of materials and the complexity...
08/23/2019

Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks

Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy perf...
05/27/2019

On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

Mixup zhang2017mixup is a recently proposed method for training deep neu...
07/23/2019

Uncertainty in the MAN Data Calibration & Trend Estimates

We investigate trend identification in the LML and MAN atmospheric ammon...
03/21/2017

Overcoming model simplifications when quantifying predictive uncertainty

It is generally accepted that all models are wrong -- the difficulty is ...