Meta-Learned Invariant Risk Minimization

03/24/2021
by   Jun-Hyun Bae, et al.
0

Empirical Risk Minimization (ERM) based machine learning algorithms have suffered from weak generalization performance on data obtained from out-of-distribution (OOD). To address this problem, Invariant Risk Minimization (IRM) objective was suggested to find invariant optimal predictor which is less affected by the changes in data distribution. However, even with such progress, IRMv1, the practical formulation of IRM, still shows performance degradation when there are not enough training data, and even fails to generalize to OOD, if the number of spurious correlations is larger than the number of environments. In this paper, to address such problems, we propose a novel meta-learning based approach for IRM. In this method, we do not assume the linearity of classifier for the ease of optimization, and solve ideal bi-level IRM objective with Model-Agnostic Meta-Learning (MAML) framework. Our method is more robust to the data with spurious correlations and can provide an invariant optimal classifier even when data from each distribution are scarce. In experiments, we demonstrate that our algorithm not only has better OOD generalization performance than IRMv1 and all IRM variants, but also addresses the weakness of IRMv1 with improved stability.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/05/2019

Invariant Risk Minimization

We introduce Invariant Risk Minimization (IRM), a learning paradigm to e...
10/12/2020

The Risks of Invariant Risk Minimization

Invariant Causal Prediction (Peters et al., 2016) is a technique for out...
10/12/2021

Gated Information Bottleneck for Generalization in Sequential Environments

Deep neural networks suffer from poor generalization to unseen environme...
06/30/2021

Which Echo Chamber? Regions of Attraction in Learning with Decision-Dependent Distributions

As data-driven methods are deployed in real-world settings, the processe...
10/16/2021

Invariant Language Modeling

Modern pretrained language models are critical components of NLP pipelin...
01/01/2018

Towards Practical Conditional Risk Minimization

We study conditional risk minimization (CRM), i.e. the problem of learni...
10/24/2021

Kernelized Heterogeneous Risk Minimization

The ability to generalize under distributional shifts is essential to re...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.