Cross-Validation, Risk Estimation, and Model Selection

09/25/2019
by   Stefan Wager, et al.
0

Cross-validation is a popular non-parametric method for evaluating the accuracy of a predictive rule. The usefulness of cross-validation depends on the task we want to employ it for. In this note, I discuss a simple non-parametric setting, and find that cross-validation is asymptotically uninformative about the expected test error of any given predictive rule, but allows for asymptotically consistent model selection. The reason for this phenomenon is that the leading-order error term of cross-validation doesn't depend on the model being evaluated, and so cancels out when we compare two models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/05/2016

A note on adjusting R^2 for using with cross-validation

We show how to adjust the coefficient of determination (R^2) when used f...
research
10/20/2019

hv-Block Cross Validation is not a BIBD: a Note on the Paper by Jeff Racine (2000)

This note corrects a mistake in the paper "consistent cross-validatory m...
research
03/15/2023

Distribution-free Deviation Bounds of Learning via Model Selection with Cross-validation Risk Estimation

Cross-validation techniques for risk estimation and model selection are ...
research
06/18/2021

Local asymptotics of cross-validation in least-squares density estimation

In model selection, several types of cross-validation are commonly used ...
research
09/27/2019

Bootstrap Cross-validation Improves Model Selection in Pharmacometrics

Cross-validation assesses the predictive ability of a model, allowing on...
research
08/29/2020

Model selection for estimation of causal parameters

A popular technique for selecting and tuning machine learning estimators...
research
11/27/2021

Fast and Informative Model Selection using Learning Curve Cross-Validation

Common cross-validation (CV) methods like k-fold cross-validation or Mon...

Please sign up or login with your details

Forgot password? Click here to reset