Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration

11/15/2020
by   Shengjia Zhao, et al.
0

Decision makers often need to rely on imperfect probabilistic forecasts. While average performance metrics are typically available, it is difficult to assess the quality of individual forecasts and the corresponding utilities. To convey confidence about individual predictions to decision-makers, we propose a compensation mechanism ensuring that the forecasted utility matches the actually accrued utility. While a naive scheme to compensate decision-makers for prediction errors can be exploited and might not be sustainable in the long run, we propose a mechanism based on fair bets and online learning that provably cannot be exploited. We demonstrate an application showing how passengers could confidently optimize individual travel plans based on flight delay probabilities estimated by an airline.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/04/2022

Reconciling Individual Probability Forecasts

Individual probabilities refer to the probabilities of outcomes that are...
research
05/31/2023

Human-Aligned Calibration for AI-Assisted Decision Making

Whenever a binary classifier is used to provide decision support, it typ...
research
02/23/2023

Online Calibrated Regression for Adversarially Robust Forecasting

Accurately estimating uncertainty is a crucial component of decision-mak...
research
06/18/2020

Individual Calibration with Randomized Forecasting

Machine learning applications often require calibrated predictions, e.g....
research
08/08/2018

Reconciliation of probabilistic forecasts with an application to wind power

New methods are proposed for adjusting probabilistic forecasts to ensure...
research
09/29/2021

Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration

An increasingly common use case for machine learning models is augmentin...
research
10/01/2021

Personalized Rehabilitation Robotics based on Online Learning Control

The use of rehabilitation robotics in clinical applications gains increa...

Please sign up or login with your details

Forgot password? Click here to reset