Systematic Overestimation of Machine Learning Performance in Neuroimaging Studies of Depression

by   Claas Flint, et al.

We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from Major Depressive Disorder (MDD) and healthy controls. Drawing upon a balanced sample of N = 1,868 MDD patients and healthy controls from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of 61 of various sizes (N=4 to N=150) from the population and showed a strong risk of overestimation. Specifically, for small sample sizes (N=20), we observe accuracies of up to 95 up to 75 sufficiently large test sets effectively protect against performance overestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets.



There are no comments yet.


page 1

page 2

page 3

page 4


Machine learning of neuroimaging to diagnose cognitive impairment and dementia: a systematic review and comparative analysis

INTRODUCTION: Advanced machine learning methods might help to identify d...

Evaluation of serum vitamin D levels in patients with hepatitis B virus (HBV)

Introduction Recently, several studies have shown that vitamin D defi...

Accelerated functional brain aging in major depressive disorder: evidence from a large scale fMRI analysis of Chinese participants

Major depressive disorder (MDD) is one of the most common mental health ...

When are Deep Networks really better than Random Forests at small sample sizes?

Random forests (RF) and deep networks (DN) are two of the most popular m...

What to do if N is two?

The field of in-vivo neurophysiology currently uses statistical standard...

Almost instant brain atlas segmentation for large-scale studies

Large scale studies of group differences in healthy controls and patient...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.