Calibrating for Class Weights by Modeling Machine Learning

05/10/2022
by   Andrew Caplin, et al.
0

A much studied issue is the extent to which the confidence scores provided by machine learning algorithms are calibrated to ground truth probabilities. Our starting point is that calibration is seemingly incompatible with class weighting, a technique often employed when one class is less common (class imbalance) or with the hope of achieving some external objective (cost-sensitive learning). We provide a model-based explanation for this incompatibility and use our anthropomorphic model to generate a simple method of recovering likelihoods from an algorithm that is miscalibrated due to class weighting. We validate this approach in the binary pneumonia detection task of Rajpurkar, Irvin, Zhu, et al. (2017).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/09/2020

MetaInfoNet: Learning Task-Guided Information for Sample Reweighting

Deep neural networks have been shown to easily overfit to biased trainin...
research
05/04/2021

A Finer Calibration Analysis for Adversarial Robustness

We present a more general analysis of H-calibration for adversarially ro...
research
12/24/2017

Optimal Weighting for Exam Composition

A problem faced by many instructors is that of designing exams that accu...
research
02/11/2021

When and How Mixup Improves Calibration

In many machine learning applications, it is important for the model to ...
research
05/30/2023

When Does Optimizing a Proper Loss Yield Calibration?

Optimizing proper loss functions is popularly believed to yield predicto...
research
12/03/2019

Distance-Based Learning from Errors for Confidence Calibration

Deep neural networks (DNNs) are poorly-calibrated when trained in conven...
research
08/26/2021

Machine Learning for Discovering Effective Interaction Kernels between Celestial Bodies from Ephemerides

Building accurate and predictive models of the underlying mechanisms of ...

Please sign up or login with your details

Forgot password? Click here to reset