Population Predictive Checks

08/02/2019
by   Rajesh Ranganath, et al.
0

Bayesian modeling has become a staple for researchers analyzing data. Thanks to recent developments in approximate posterior inference, modern researchers can easily build, use, and revise complicated Bayesian models for large and rich data. These new abilities, however, bring into focus the problem of model assessment. Researchers need tools to diagnose the fitness of their models, to understand where a model falls short, and to guide its revision. In this paper we develop a new method for Bayesian model checking, the population predictive check (Pop-PC). Pop-PCs are built on posterior predictive checks (PPC), a seminal method that checks a model by assessing the posterior predictive distribution on the observed data. Though powerful, PPCs use the data twice---both to calculate the posterior predictive and to evaluate it---which can lead to overconfident assessments. Pop-PCs, in contrast, compare the posterior predictive distribution to the population distribution of the data. This strategy blends Bayesian modeling with frequentist assessment, leading to a robust check that validates the model on its generalization. Of course the population distribution is not usually available; thus we use tools like the bootstrap and cross validation to estimate the Pop-PC. Further, we extend Pop-PCs to hierarchical models. We study Pop-PCs on classical regression and a hierarchical model of text. We show that Pop-PCs are robust to overfitting and can be easily deployed on a broad family of models.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/06/2021

Posterior Predictive Null Checks

Bayesian model criticism is an important part of the practice of Bayesia...
12/20/2017

Bayesian model checking: A comparison of tests

Two procedures for checking Bayesian models are compared using a simple ...
05/24/2016

Posterior Dispersion Indices

Probabilistic modeling is cyclical: we specify a model, infer its poster...
05/09/2021

Statistical Assessment of Replicability via Bayesian Model Criticism

Assessment of replicability is critical to ensure the quality and rigor ...
01/28/2019

Hierarchical network models for structured exchangeable interaction processes

Network data often arises via a series of structured interactions among ...
03/29/2022

Calibrated Model Criticism Using Split Predictive Checks

Checking how well a fitted model explains the data is one of the most fu...
04/04/2021

Generalised Bayesian Structural Equation Modelling

We propose a generalised framework for Bayesian Structural Equation Mode...