Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance

08/25/2020 ∙ by Tuomas Sivula, et al. ∙ 0

When evaluating and comparing models using leave-one-out cross-validation (LOO-CV), the uncertainty of the estimate is typically assessed using the variance of the sampling distribution. It is known, however, that no unbiased estimator for the variance can be constructed in a general case. While it has not been discussed before, it could be possible to construct such an estimator by considering specific models. In this paper, we show that an unbiased sampling distribution variance estimator is obtainable for the Bayesian normal model with fixed model variance using expected log pointwise predictive density (elpd) utility score. Instead of the obtained pointwise LOO-CV estimates, we estimate the variance directly from the observations. Motivated by the presented unbiased variance estimator, it could be possible to obtain other improved problem-specific estimators, not only unbiased ones, for assessing the uncertainty of LOO-CV estimation.



There are no comments yet.


page 1

page 2

page 3

page 4

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.