Meaningless comparisons lead to false optimism in medical machine learning

by   Orianna DeMasi, et al.

A new trend in medicine is the use of algorithms to analyze big datasets, e.g. using everything your phone measures about you for diagnostics or monitoring. However, these algorithms are commonly compared against weak baselines, which may contribute to excessive optimism. To assess how well an algorithm works, scientists typically ask how well its output correlates with medically assigned scores. Here we perform a meta-analysis to quantify how the literature evaluates their algorithms for monitoring mental wellbeing. We find that the bulk of the literature (∼77 ignore patient baseline state. For example, having an algorithm that uses phone data to diagnose mood disorders would be useful. However, it is possible to over 80 guessing that each patient has their own average mood - the patient-specific baseline. Thus, an algorithm that just predicts that our mood is like it usually is can explain the majority of variance, but is, obviously, entirely useless. Comparing to the wrong (population) baseline has a massive effect on the perceived quality of algorithms and produces baseless optimism in the field. To solve this problem we propose "user lift" that reduces these systematic errors in the evaluation of personalized medical monitoring.


page 1

page 2

page 3

page 4


Medical Deep Learning – A systematic Meta-Review

Deep learning had a remarkable impact in different scientific discipline...

Determination of Personalized Asthma Triggers from Evidence based on Multimodal Sensing and Mobile Application

Objective: Asthma is a chronic pulmonary disease with multiple triggers ...

Simultaneous imputation and disease classification in incomplete medical datasets using Multigraph Geometric Matrix Completion (MGMC)

Large-scale population-based studies in medicine are a key resource towa...

Machine Learning Evaluation of the Echo-Chamber Effect in Medical Forums

We propose the Echo-Chamber Effect assessment of an online forum. Sentim...

SUBIC: A Supervised Bi-Clustering Approach for Precision Medicine

Traditional medicine typically applies one-size-fits-all treatment for t...

An End-to-End Authentication Mechanism for Wireless Body Area Networks

Wireless Body Area Network (WBAN) ensures high-quality healthcare servic...

Top-N Recommendation Algorithms: A Quest for the State-of-the-Art

Research on recommender systems algorithms, like other areas of applied ...

Please sign up or login with your details

Forgot password? Click here to reset