Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

10/28/2019
by   Meelis Kull, et al.
27

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

READ FULL TEXT

page 3

page 5

page 16

page 17

page 25

research
05/01/2019

Unsupervised Temperature Scaling: Post-Processing Unsupervised Calibration of Deep Models Decisions

Great performances of deep learning are undeniable, with impressive resu...
research
03/15/2020

Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks

Predicting calibrated confidence scores for multi-class deep networks is...
research
01/21/2019

Calibration with Bias-Corrected Temperature Scaling Improves Domain Adaptation Under Label Shift in Modern Neural Networks

Label shift refers to the phenomenon where the marginal probability p(y)...
research
09/08/2018

On the Calibration of Nested Dichotomies for Large Multiclass Tasks

Nested dichotomies are used as a method of transforming a multiclass cla...
research
06/28/2022

SLOVA: Uncertainty Estimation Using Single Label One-Vs-All Classifier

Deep neural networks present impressive performance, yet they cannot rel...
research
10/07/2022

Class-wise and reduced calibration methods

For many applications of probabilistic classifiers it is important that ...
research
05/15/2021

Calibrating sufficiently

When probabilistic classifiers are trained and calibrated, the so-called...

Please sign up or login with your details

Forgot password? Click here to reset