From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

05/22/2020
by   Dimitris Tsipras, et al.
24

Building rich machine learning datasets in a scalable manner often necessitates a crowd-sourced data collection pipeline. In this work, we use human studies to investigate the consequences of employing such a pipeline, focusing on the popular ImageNet dataset. We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset—including the introduction of biases that state-of-the-art models exploit. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for. Finally, our findings emphasize the need to augment our current model training and evaluation toolkit to take such misalignments into account. To facilitate further research, we release our refined ImageNet annotations at https://github.com/MadryLab/ImageNetMultiLabel.

READ FULL TEXT

page 21

page 22

page 27

page 28

page 30

page 31

page 32

page 33

research
01/05/2023

Beyond web-scraping: Crowd-sourcing a geographically diverse image dataset

Current dataset collection methods typically scrape large amounts of dat...
research
08/16/2021

Towards Efficient and Data Agnostic Image Classification Training Pipeline for Embedded Systems

Nowadays deep learning-based methods have achieved a remarkable progress...
research
04/11/2023

Zoom is what you need: An empirical study of the power of zoom and spatial biases in image classification

Image classifiers are information-discarding machines, by design. Yet, h...
research
03/16/2022

Towards Formalizing HRI Data Collection Processes

Within the human-robot interaction (HRI) community, many researchers hav...
research
07/02/2021

Vox Populi, Vox DIY: Benchmark Dataset for Crowdsourced Audio Transcription

Domain-specific data is the crux of the successful transfer of machine l...
research
01/29/2019

Semantic Redundancies in Image-Classification Datasets: The 10 Don't Need

Large datasets have been crucial to the success of deep learning models ...
research
06/30/2023

TTSWING: a Dataset for Table Tennis Swing Analysis

We introduce TTSWING, a novel dataset designed for table tennis swing an...

Please sign up or login with your details

Forgot password? Click here to reset