FOCUS: Familiar Objects in Common and Uncommon Settings

10/07/2021
by   Priyatham Kattakinda, et al.
0

Standard training datasets for deep learning often contain objects in common settings (e.g., "a horse on grass" or "a ship in water") since they are usually collected by randomly scraping the web. Uncommon and rare settings (e.g., "a plane on water", "a car in snowy weather") are thus severely under-represented in the training data. This can lead to an undesirable bias in model predictions towards common settings and create a false sense of accuracy. In this paper, we introduce FOCUS (Familiar Objects in Common and Uncommon Settings), a dataset for stress-testing the generalization power of deep image classifiers. By leveraging the power of modern search engines, we deliberately gather data containing objects in common and uncommon settings in a wide range of locations, weather conditions, and time of day. We present a detailed analysis of the performance of various popular image classifiers on our dataset and demonstrate a clear drop in performance when classifying images in uncommon settings. By analyzing deep features of these models, we show that such errors can be due to the use of spurious features in model predictions. We believe that our dataset will aid researchers in understanding the inability of deep models to generalize well to uncommon settings and drive future work on improving their distributional robustness.

READ FULL TEXT

page 1

page 9

page 12

page 17

page 18

page 19

page 20

page 21

research
04/04/2020

ObjectNet Dataset: Reanalysis and Correction

Recently, Barbu et al introduced a dataset called ObjectNet which includ...
research
06/17/2019

The Cells Out of Sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers

Understanding if classifiers generalize to out-of-sample datasets is a c...
research
10/22/2019

WeatherNet: Recognising weather and visual conditions from street-level images using deep residual learning

Extracting information related to weather and visual conditions at a giv...
research
01/16/2017

3D tracking of water hazards with polarized stereo cameras

Current self-driving car systems operate well in sunny weather but strug...
research
12/09/2022

Spurious Features Everywhere – Large-Scale Detection of Harmful Spurious Features in ImageNet

Benchmark performance of deep learning classifiers alone is not a reliab...
research
06/30/2023

Dataset balancing can hurt model performance

Machine learning from training data with a skewed distribution of exampl...

Please sign up or login with your details

Forgot password? Click here to reset