DeepAI
Log In Sign Up

Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions

06/06/2018
by   Lena Maier-Hein, et al.
10

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

READ FULL TEXT

page 1

page 5

page 6

page 10

page 12

page 17

10/09/2019

BIAS: Transparent reporting of biomedical image analysis challenges

The number of biomedical image analysis challenges organized per year is...
04/12/2021

Common Limitations of Image Processing Metrics: A Picture Story

While the importance of automatic image analysis is increasing at an eno...
11/17/2022

Sources of performance variability in deep learning-based polyp detection

Validation metrics are a key prerequisite for the reliable tracking of s...
10/11/2019

Methods and open-source toolkit for analyzing and visualizing challenge results

Biomedical challenges have become the de facto standard for benchmarking...
08/30/2021

BioFors: A Large Biomedical Image Forensics Dataset

Research in media forensics has gained traction to combat the spread of ...
10/26/2018

Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems

In the medical image analysis field, organizing challenges with associat...