Moment Multicalibration for Uncertainty Estimation

by   Christopher Jung, et al.

We show how to achieve the notion of "multicalibration" from Hébert-Johnson et al. [2018] not just for means, but also for variances and other higher moments. Informally, it means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well-and those predictions match the true distribution quantities when averaged not just over the population as a whole, but also when averaged over an enormous number of finely defined subgroups. It yields a principled way to estimate the uncertainty of predictions on many different subgroups-and to diagnose potential sources of unfairness in the predictive power of features across subgroups. As an application, we show that our moment estimates can be used to derive marginal prediction intervals that are simultaneously valid as averaged over all of the (sufficiently large) subgroups for which moment multicalibration has been obtained.


page 1

page 2

page 3

page 4


Online Multivalid Learning: Means, Moments, and Prediction Intervals

We present a general, efficient technique for providing contextual predi...

When Fourth Moments Are Enough

This note concerns a somewhat innocent question motivated by an observat...

Tail Bound Analysis for Probabilistic Programs via Central Moments

For probabilistic programs, it is usually not possible to automatically ...

Culling the herd of moments with penalized empirical likelihood

Models defined by moment conditions are at the center of structural econ...

Improved Calibration of Numerical Integration Error in Sigma-Point Filters

The sigma-point filters, such as the UKF, which exploit numerical quadra...

Using the discrete radon transformation for grayscale image moments

Image moments are weighted sums over pixel values in a given image and a...

Identification in Bayesian Estimation of the Skewness Matrix in a Multivariate Skew-Elliptical Distribution

Harvey et al. (2010) extended the Bayesian estimation method by Sahu et ...