Empirical or Invariant Risk Minimization? A Sample Complexity Perspective

10/30/2020
by   Kartik Ahuja, et al.
5

Recently, invariant risk minimization (IRM) was proposed as a promising solution to address out-of-distribution (OOD) generalization. However, it is unclear when IRM should be preferred over the widely-employed empirical risk minimization (ERM) framework. In this work, we analyze both these frameworks from the perspective of sample complexity, thus taking a firm step towards answering this important question. We find that depending on the type of data generation mechanism, the two approaches might have very different finite sample and asymptotic behavior. For example, in the covariate shift setting we see that the two approaches not only arrive at the same asymptotic solution, but also have similar finite sample behavior with no clear winner. For other distribution shifts such as those involving confounders or anti-causal variables, however, the two approaches arrive at different asymptotic solutions where IRM is guaranteed to be close to the desired OOD solutions in the finite sample regime, while ERM is biased even asymptotically. We further investigate how different factors – the number of environments, complexity of the model, and IRM penalty weight – impact the sample complexity of IRM in relation to its distance from the OOD solutions

READ FULL TEXT

page 1

page 2

page 3

page 4

10/23/2017

On Asymptotic Standard Normality of the Two Sample Pivot

The asymptotic solution to the problem of comparing the means of two het...
07/22/2020

Multi-reference alignment in high dimensions: sample complexity and phase transition

Multi-reference alignment entails estimating a signal in ℝ^L from its ci...
10/09/2021

On the asymptotic behavior of bubble date estimators

In this study, we extend the three-regime bubble model of Pang et al. (2...
06/08/2021

Intrinsic Dimension Estimation

It has long been thought that high-dimensional data encountered in many ...
02/09/2022

An Exploration of Multicalibration Uniform Convergence Bounds

Recent works have investigated the sample complexity necessary for fair ...
04/02/2021

Linear Systems can be Hard to Learn

In this paper, we investigate when system identification is statisticall...
10/21/2021

DAIR: Data Augmented Invariant Regularization

While deep learning through empirical risk minimization (ERM) has succee...

Code Repositories

OoD

Repository for theory and methods for Out-of-Distribution (OoD) generalization


view repo