-
Measuring Calibration in Deep Learning
The reliability of a machine learning model's confidence in its predicti...
read it
-
Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
Post-hoc calibration is a common approach for providing high-quality con...
read it
-
Evaluating model calibration in classification
Probabilistic classifiers output a probability distribution on target cl...
read it
-
Calibration of Neural Networks using Splines
Calibrating neural networks is of utmost importance when employing them ...
read it
-
Overcoming model simplifications when quantifying predictive uncertainty
It is generally accepted that all models are wrong -- the difficulty is ...
read it
-
Better Multi-class Probability Estimates for Small Data Sets
Many classification applications require accurate probability estimates ...
read it
-
Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks
Predicting calibrated confidence scores for multi-class deep networks is...
read it
Calibration tests in multi-class classification: A unifying framework
In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class classification that generalize existing measures such as the expected calibration error, the maximum calibration error, and the maximum mean calibration error. We propose and evaluate empirically different consistent and unbiased estimators for a specific class of measures based on matrix-valued kernels. Importantly, these estimators can be interpreted as test statistics associated with well-defined bounds and approximations of the p-value under the null hypothesis that the model is calibrated, significantly improving the interpretability of calibration measures, which otherwise lack any meaningful unit or scale.
READ FULL TEXT
Comments
There are no comments yet.