Calibrated Model Criticism Using Split Predictive Checks

03/29/2022
by   Jiawei Li, et al.
0

Checking how well a fitted model explains the data is one of the most fundamental parts of a Bayesian data analysis. However, existing model checking methods suffer from trade-offs between being well-calibrated, automated, and computationally efficient. To overcome these limitations, we propose split predictive checks (SPCs), which combine the ease-of-use and speed of posterior predictive checks with the good calibration properties of predictive checks that rely on model-specific derivations or inference schemes. We develop an asymptotic theory for two types of SPCs: single SPCs and the divided SPC. Our results demonstrate that they offer complementary strengths: single SPCs provide superior power in the small-data regime or when the misspecification is significant and divided SPCs provide superior power as the dataset size increases or when the form of misspecification is more subtle. We validate the finite-sample utility of SPCs through extensive simulation experiments in exponential family and hierarchical models, and provide four real-data examples where SPCs offer novel insights and additional flexibility beyond what is available when using posterior predictive checks.

READ FULL TEXT
research
06/11/2021

Conformal Bayesian Computation

We develop scalable methods for producing conformal Bayesian predictive ...
research
02/18/2019

Conformal calibrators

Most existing examples of full conformal predictive systems, split-confo...
research
12/20/2017

Bayesian model checking: A comparison of tests

Two procedures for checking Bayesian models are compared using a simple ...
research
05/09/2021

Statistical Assessment of Replicability via Bayesian Model Criticism

Assessment of replicability is critical to ensure the quality and rigor ...
research
08/02/2019

Population Predictive Checks

Bayesian modeling has become a staple for researchers analyzing data. Th...
research
06/08/2023

Computational methods for fast Bayesian model assessment via calibrated posterior p-values

Posterior predictive p-values (ppps) have become popular tools for Bayes...
research
05/24/2016

Posterior Dispersion Indices

Probabilistic modeling is cyclical: we specify a model, infer its poster...

Please sign up or login with your details

Forgot password? Click here to reset