Log In Sign Up

Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions

by   Lena Maier-Hein, et al.

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.


page 1

page 5

page 6

page 10

page 12

page 17


BIAS: Transparent reporting of biomedical image analysis challenges

The number of biomedical image analysis challenges organized per year is...

Common Limitations of Image Processing Metrics: A Picture Story

While the importance of automatic image analysis is increasing at an eno...

Sources of performance variability in deep learning-based polyp detection

Validation metrics are a key prerequisite for the reliable tracking of s...

Methods and open-source toolkit for analyzing and visualizing challenge results

Biomedical challenges have become the de facto standard for benchmarking...

BioFors: A Large Biomedical Image Forensics Dataset

Research in media forensics has gained traction to combat the spread of ...

Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems

In the medical image analysis field, organizing challenges with associat...