Cross-validation failure: small sample sizes lead to large error bars

06/23/2017
by   Gaël Varoquaux, et al.
0

Predictive models ground many state-of-the-art developments in statistical brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers. The principled approach to establish their validity and usefulness is cross-validation, testing prediction on unseen data. Here, I would like to raise awareness on error bars of cross-validation, which are often underestimated. Simple experiments show that sample sizes of many neuroimaging studies inherently lead to large error bars, eg ±10 standard error across folds strongly underestimates them. These large error bars compromise the reliability of conclusions drawn with predictive models, such as biomarkers or methods developments where, unlike with cognitive neuroimaging MVPA approaches, more samples cannot be acquired by repeating the experiment across many subjects. Solutions to increase sample size must be investigated, tackling possible increases in heterogeneity of the data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/03/2012

Cross-conformal predictors

This note introduces the method of cross-conformal prediction, which is ...
research
03/23/2017

Cross-Validation with Confidence

Cross-validation is one of the most popular model selection methods in s...
research
11/26/2019

The Early Roots of Statistical Learning in the Psychometric Literature: A review and two new results

Machine and Statistical learning techniques become more and more importa...
research
12/27/2019

Statistical Agnostic Mapping: a Framework in Neuroimaging based on Concentration Inequalities

In the 70s a novel branch of statistics emerged focusing its effort in s...
research
06/16/2016

Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

Decoding, ie prediction from brain images or signals, calls for empirica...
research
01/06/2021

Cross-Validation and Uncertainty Determination for Randomized Neural Networks with Applications to Mobile Sensors

Randomized artificial neural networks such as extreme learning machines ...

Please sign up or login with your details

Forgot password? Click here to reset