Calibrated inference: statistical inference that accounts for both sampling uncertainty and distributional uncertainty

02/24/2022
by   Yujin Jeong, et al.
0

During data analysis, analysts often have to make seemingly arbitrary decisions. For example during data pre-processing there are a variety of options for dealing with outliers or inferring missing data. Similarly, many specifications and methods can be reasonable to address a domain question. This may be seen as a hindrance for reliable inference, since conclusions can change depending on the analyst's choices. In this paper, we argue that this situation is an opportunity to construct confidence intervals that account not only for sampling uncertainty but also some type of distributional uncertainty. The distributional uncertainty model is related to a variety of potential issues with the data, ranging from dependence between observations to selection bias and confounding. The final procedure can be seen as an extension of current statistical practice. However, the underlying assumptions are quite different. Standard statistical practice often relies on the i.i.d. assumption. We rely on a strictly weaker symmetry assumption stating that the empirical distribution and the target distribution differ by an isotropic distributional perturbation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/07/2021

The r-value: evaluating stability with respect to distributional shifts

Common statistical measures of uncertainty like p-values and confidence ...
research
09/09/2011

Ignorability in Statistical and Probabilistic Inference

When dealing with incomplete data in statistical learning, or incomplete...
research
09/19/2022

Distributionally robust and generalizable inference

We discuss recently developed methods that quantify the stability and ge...
research
09/04/2021

Confidence Distribution and Distribution Estimation for Modern Statistical Inference

This paper introduces to readers the new concept and methodology of conf...
research
10/11/2016

Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach

We study statistical inference and robust solution methods for stochasti...
research
12/23/2020

Testing whether a Learning Procedure is Calibrated

A learning procedure takes as input a dataset and performs inference for...
research
08/05/2017

Construction and Visualization of Optimal Confidence Sets for Frequentist Distributional Forecasts

The focus of this paper is on the quantification of sampling variation i...

Please sign up or login with your details

Forgot password? Click here to reset