An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization

02/25/2021
by   Elan Rosenfeld, et al.
0

A popular assumption for out-of-distribution generalization is that the training data comprises sub-datasets, each drawn from a distinct distribution; the goal is then to "interpolate" these distributions and "extrapolate" beyond them – this objective is broadly known as domain generalization. A common belief is that ERM can interpolate but not extrapolate and that the latter is considerably more difficult, but these claims are vague and lack formal justification. In this work, we recast generalization over sub-groups as an online game between a player minimizing risk and an adversary presenting new test distributions. Under an existing notion of inter- and extrapolation based on reweighting of sub-group likelihoods, we rigorously demonstrate that extrapolation is computationally much harder than interpolation, though their statistical complexity is not significantly different. Furthermore, we show that ERM – or a noisy variant – is provably minimax-optimal for both tasks. Our framework presents a new avenue for the formal analysis of domain generalization algorithms which may be of independent interest.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/23/2023

Improving Generalization with Domain Convex Game

Domain generalization (DG) tends to alleviate the poor generalization ca...
research
02/13/2020

Learn to Expect the Unexpected: Probably Approximately Correct Domain Generalization

Domain generalization is the problem of machine learning when the traini...
research
05/31/2023

Online-to-PAC Conversions: Generalization Bounds via Regret Analysis

We present a new framework for deriving bounds on the generalization bou...
research
06/07/2021

OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms

Deep learning has achieved tremendous success with independent and ident...
research
10/21/2022

Just Mix Once: Worst-group Generalization by Group Interpolation

Advances in deep learning theory have revealed how average generalizatio...
research
09/17/2020

Distributional Generalization: A New Kind of Generalization

We introduce a new notion of generalization – Distributional Generalizat...
research
09/18/2023

Multi-dimensional domain generalization with low-rank structures

In conventional statistical and machine learning methods, it is typicall...

Please sign up or login with your details

Forgot password? Click here to reset