loocv_uncertainty
None
view repo
When evaluating and comparing models using leave-one-out cross-validation (LOO-CV), the uncertainty of the estimate is typically assessed using the variance of the sampling distribution. It is known, however, that no unbiased estimator for the variance can be constructed in a general case. While it has not been discussed before, it could be possible to construct such an estimator by considering specific models. In this paper, we show that an unbiased sampling distribution variance estimator is obtainable for the Bayesian normal model with fixed model variance using expected log pointwise predictive density (elpd) utility score. Instead of the obtained pointwise LOO-CV estimates, we estimate the variance directly from the observations. Motivated by the presented unbiased variance estimator, it could be possible to obtain other improved problem-specific estimators, not only unbiased ones, for assessing the uncertainty of LOO-CV estimation.
READ FULL TEXT
Leave-one-out cross-validation (LOO-CV) is a popular method for comparin...
read it
A basic result is that the sample variance for i.i.d. observations is an...
read it
Density Surface Models (DSMs) are two-stage models for estimating animal...
read it
Posterior distributions often feature intractable normalizing constants,...
read it
Representational similarity analysis (RSA) tests models of brain computa...
read it
We present an in-depth analysis of the sources of variance in
state-of-t...
read it
Systematic sampling is often used to select plot locations for forest
in...
read it
None
Comments
There are no comments yet.