Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions

12/27/2022
by   Jake C. Snell, et al.
13

Rigorous guarantees about the performance of predictive algorithms are necessary in order to ensure their responsible use. Previous work has largely focused on bounding the expected loss of a predictor, but this is not sufficient in many risk-sensitive applications where the distribution of errors is important. In this work, we propose a flexible framework to produce a family of bounds on quantiles of the loss distribution incurred by a predictor. Our method takes advantage of the order statistics of the observed loss values rather than relying on the sample mean alone. We show that a quantile is an informative way of quantifying predictive performance, and that our framework applies to a variety of quantile-based metrics, each targeting important subsets of the data distribution. We analyze the theoretical properties of our proposed method and demonstrate its ability to rigorously control loss quantiles on several real-world datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/27/2012

Predicting Conditional Quantiles via Reduction to Classification

We show how to reduce the process of predicting general order statistics...
research
05/26/2023

Distributional Reinforcement Learning with Dual Expectile-Quantile Regression

Successful applications of distributional reinforcement learning with qu...
research
01/17/2019

A Multi-Level Simulation Optimization Approach for Quantile Functions

Quantile is a popular performance measure for a stochastic system to eva...
research
02/09/2021

Regularization Strategies for Quantile Regression

We investigate different methods for regularizing quantile regression wh...
research
10/11/2021

Quantile-based hydrological modelling

Predictive uncertainty in hydrological modelling is quantified by using ...
research
06/01/2017

Supervised Quantile Normalisation

Quantile normalisation is a popular normalisation method for data subjec...
research
01/14/2022

Expectile-based hydrological modelling for uncertainty estimation: Life after mean

Predictions of hydrological models should be probabilistic in nature. Ou...

Please sign up or login with your details

Forgot password? Click here to reset