Understanding the Under-Coverage Bias in Uncertainty Estimation

by   Yu Bai, et al.

Estimating the data uncertainty in regression tasks is often done by learning a quantile function or a prediction interval of the true label conditioned on the input. It is frequently observed that quantile regression – a vanilla algorithm for learning quantiles with asymptotic guarantees – tends to under-cover than the desired coverage level in reality. While various fixes have been proposed, a more fundamental understanding of why this under-coverage bias happens in the first place remains elusive. In this paper, we present a rigorous theoretical study on the coverage of uncertainty estimation algorithms in learning quantiles. We prove that quantile regression suffers from an inherent under-coverage bias, in a vanilla setting where we learn a realizable linear quantile function and there is more data than parameters. More quantitatively, for α>0.5 and small d/n, the α-quantile learned by quantile regression roughly achieves coverage α - (α-1/2)· d/n regardless of the noise distribution, where d is the input dimension and n is the number of training data. Our theory reveals that this under-coverage bias stems from a certain high-dimensional parameter estimation error that is not implied by existing theories on quantile regression. Experiments on simulated and real data verify our theory and further illustrate the effect of various factors such as sample size and model capacity on the under-coverage bias in more practical setups.



There are no comments yet.


page 1

page 2

page 3

page 4


Uniform Inference for High-dimensional Quantile Regression: Linear Functionals and Regression Rank Scores

Hypothesis tests in models whose dimension far exceeds the sample size c...

Conformalized Quantile Regression

Conformal prediction is a technique for constructing prediction interval...

Improving Conditional Coverage via Orthogonal Quantile Regression

We develop a method to generate prediction intervals that have a user-sp...

Bayesian Quantile Matching Estimation

Due to increased awareness of data protection and corresponding laws man...

Can a single neuron learn quantiles?

A novel non-parametric quantile estimation method for continuous random ...

Multiscale quantile regression

We introduce a new methodology for analyzing serial data by quantile reg...

Theory meets Practice at the Median: a worst case comparison of relative error quantile algorithms

Estimating the distribution and quantiles of data is a foundational task...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.