Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation

by   Lars Schmarje, et al.

High-quality data is necessary for modern machine learning. However, the acquisition of such data is difficult due to noisy and ambiguous annotations of humans. The aggregation of such annotations to determine the label of an image leads to a lower data quality. We propose a data-centric image classification benchmark with nine real-world datasets and multiple annotations per image to investigate and quantify the impact of such data quality issues. We focus on a data-centric perspective by asking how we could improve the data quality. Across thousands of experiments, we show that multiple annotations allow a better approximation of the real underlying class distribution. We identify that hard labels can not capture the ambiguity of the data and this might lead to the common issue of overconfident models. Based on the presented datasets, benchmark baselines, and analysis, we create multiple research opportunities for the future.


Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality

High-quality data is crucial for the success of machine learning, but la...

Improve learning combining crowdsourced labels by weighting Areas Under the Margin

In supervised learning – for instance in image classification – modern m...

A Vision-Centric Approach for Static Map Element Annotation

The recent development of online static map element (a.k.a. HD Map) cons...

Classification Under Ambiguity: When Is Average-K Better Than Top-K?

When many labels are possible, choosing a single one can lead to low pre...

Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels

When human annotators are given a choice about what to label in an image...

Please sign up or login with your details

Forgot password? Click here to reset