Revisiting the Calibration of Modern Neural Networks

by   Matthias Minderer, et al.

Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.


page 25

page 26


Calibrated Reliable Regression using Maximum Mean Discrepancy

Accurate quantification of uncertainty is crucial for real-world applica...

Improving model calibration with accuracy versus uncertainty optimization

Obtaining reliable and accurate quantification of uncertainty estimates ...

Artificial neural networks in calibration of nonlinear mechanical models

Rapid development in numerical modelling of materials and the complexity...

Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks

Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy perf...

On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

Mixup zhang2017mixup is a recently proposed method for training deep neu...

Uncertainty in the MAN Data Calibration & Trend Estimates

We investigate trend identification in the LML and MAN atmospheric ammon...

Overcoming model simplifications when quantifying predictive uncertainty

It is generally accepted that all models are wrong -- the difficulty is ...