Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison

08/24/2020
by   Tuomas Sivula, et al.
0

Leave-one-out cross-validation (LOO-CV) is a popular method for comparing Bayesian models based on their estimated predictive performance on new, unseen, data. Estimating the uncertainty of the resulting LOO-CV estimate is a complex task and it is known that the commonly used standard error estimate is often too small. We analyse the frequency properties of the LOO-CV estimator and study the uncertainty related to it. We provide new results of the properties of the uncertainty both theoretically and empirically and discuss the challenges of estimating it. We show that problematic cases include: comparing models with similar predictions, misspecified models, and small data. In these cases, there is a weak connection in the skewness of the sampling distribution and the distribution of the error of the LOO-CV estimator. We show that it is possible that the problematic skewness of the error distribution, which occurs when the models make similar predictions, does not fade away when the data size grows to infinity in certain situations.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

08/25/2020

Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance

When evaluating and comparing models using leave-one-out cross-validatio...
01/03/2020

Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data

Recently, new methods for model assessment, based on subsampling and pos...
12/23/2014

Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models

The future predictive performance of a Bayesian model can be estimated u...
01/27/2021

An Early Stopping Bayesian Data Assimilation Approach for Mixed-Logit Estimation

The mixed-logit model is a flexible tool in transportation choice analys...
04/05/2020

Graphical outputs and Spatial Cross-validation for the R-INLA package using INLAutils

Statistical analyses proceed by an iterative process of model fitting an...
07/31/2019

A Leisurely Look at Versions and Variants of the Cross Validation Estimator

Many versions of cross-validation (CV) exist in the literature; and each...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.