No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems

11/25/2020
by   Nimit S. Sohoni, et al.
6

In real-world classification tasks, each class often comprises multiple finer-grained "subclasses." As the subclass labels are frequently unavailable, models trained using only the coarser-grained class labels often exhibit highly variable performance across different subclasses. This phenomenon, known as hidden stratification, has important consequences for models deployed in safety-critical applications such as medicine. We propose GEORGE, a method to both measure and mitigate hidden stratification even when subclass labels are unknown. We first observe that unlabeled subclasses are often separable in the feature space of deep models, and exploit this fact to estimate subclass labels for the training data via clustering techniques. We then use these approximate subclass labels as a form of noisy supervision in a distributionally robust optimization objective. We theoretically characterize the performance of GEORGE in terms of the worst-case generalization error across any subclass. We empirically validate GEORGE on a mix of real-world and benchmark image classification datasets, and show that our approach boosts worst-case subclass accuracy by up to 22 percentage points compared to standard training techniques, without requiring any information about the subclasses.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 20

page 26

12/31/2021

BARACK: Partially Supervised Group Robustness With Guarantees

While neural networks have shown remarkable success on classification ta...
11/23/2021

Semi-Supervised Learning with Taxonomic Labels

We propose techniques to incorporate coarse taxonomic labels to train im...
10/05/2018

Training Complex Models with Multi-Task Weak Supervision

As machine learning models continue to increase in complexity, collectin...
10/14/2021

Towards Understanding the Data Dependency of Mixup-style Training

In the Mixup training paradigm, a model is trained using convex combinat...
06/08/2021

Learning from Multiple Noisy Partial Labelers

Programmatic weak supervision creates models without hand-labeled traini...
10/15/2019

Learning Sample-Specific Models with Low-Rank Personalized Regression

Modern applications of machine learning (ML) deal with increasingly hete...
09/08/2019

Fine Grained Dataflow Tracking with Proximal Gradients

Dataflow tracking with Dynamic Taint Analysis (DTA) is an important meth...

Code Repositories

hidden-stratification

Combating hidden stratification with GEORGE


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.