BREEDS: Benchmarks for Subpopulation Shift

08/11/2020
by   Shibani Santurkar, et al.
5

We develop a methodology for assessing the robustness of models to subpopulation shift—specifically, their ability to generalize to novel data subpopulations that were not observed during training. Our approach leverages the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. This enables us to synthesize realistic distribution shifts whose sources can be precisely controlled and characterized, within existing large-scale datasets. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity. We then validate that the corresponding shifts are tractable by obtaining human baselines for them. Finally, we utilize these benchmarks to measure the sensitivity of standard model architectures as well as the effectiveness of off-the-shelf train-time robustness interventions. Code and data available at https://github.com/MadryLab/BREEDS-Benchmarks .

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 6

page 23

page 25

page 27

page 28

page 29

page 31

08/30/2021

SHIFT15M: Multiobjective Large-Scale Fashion Dataset with Distributional Shifts

Many machine learning algorithms assume that the training data and the t...
07/26/2021

Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts

Synthetic corruptions gathered into a benchmark are frequently used to m...
07/01/2020

Measuring Robustness to Natural Distribution Shifts in Image Classification

We study how robust current ImageNet models are to distribution shifts a...
08/10/2021

Retiring Adult: New Datasets for Fair Machine Learning

Although the fairness community has recognized the importance of data, r...
07/20/2021

Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning

Deep Metric Learning (DML) aims to find representations suitable for zer...
04/13/2022

Distributionally Robust Models with Parametric Likelihood Ratios

As machine learning models are deployed ever more broadly, it becomes in...
01/02/2018

DeepMind Control Suite

The DeepMind Control Suite is a set of continuous control tasks with a s...

Code Repositories

robustness

A library for experimenting with, training and evaluating neural networks, with a focus on adversarial robustness.


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.