Adaptive calibration for binary classification

07/04/2021
by   Vladimir Vovk, et al.
0

This note proposes a way of making probability forecasting rules less sensitive to changes in data distribution, concentrating on the simple case of binary classification. This is important in applications of machine learning, where the quality of a trained predictor may drop significantly in the process of its exploitation. Our techniques are based on recent work on conformal test martingales and older work on prediction with expert advice, namely tracking the best expert.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/10/2023

Machine learning for sports betting: should forecasting models be optimised for accuracy or calibration?

Sports betting's recent federal legalisation in the USA coincides with t...
research
02/15/2021

Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification

Modern machine learning models with high accuracy are often miscalibrate...
research
05/31/2023

Bias Mitigation Methods for Binary Classification Decision-Making Systems: Survey and Recommendations

Bias mitigation methods for binary classification decision-making system...
research
11/11/2022

Rethinking Log Odds: Linear Probability Modelling and Expert Advice in Interpretable Machine Learning

We introduce a family of interpretable machine learning models, with two...
research
07/19/2022

Selecting applicants based on multiple ratings: Using binary classification framework as an alternative to inter-rater reliability

Inter-rater reliability (IRR) has been the prevalent quality and precisi...
research
03/16/2021

Learning to increase matching efficiency in identifying additional b-jets in the tt̅bb̅ process

The tt̅H(bb̅) process is an essential channel to reveal the Higgs proper...
research
03/04/2019

A Fundamental Performance Limitation for Adversarial Classification

Despite the widespread use of machine learning algorithms to solve probl...

Please sign up or login with your details

Forgot password? Click here to reset