DeepAI
Log In Sign Up

Gradient Matching for Domain Generalization

04/20/2021
by   Yuge Shi, et al.
0

Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive – requires computation of second-order derivatives – we derive a simpler first-order algorithm named Fish that approximates its optimization. We demonstrate the efficacy of Fish on 6 datasets from the Wilds benchmark, which captures distribution shift across a diverse range of modalities. Our method produces competitive results on these datasets and surpasses all baselines on 4 of them. We perform experiments on both the Wilds benchmark, which captures distribution shift in the real world, as well as datasets in DomainBed benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks.

READ FULL TEXT
06/04/2021

SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization

A major bottleneck in the real-world applications of machine learning mo...
08/03/2021

Domain Generalization via Gradient Surgery

In real-life applications, machine learning models often face scenarios ...
11/15/2022

HMOE: Hypernetwork-based Mixture of Experts for Domain Generalization

Due to the domain shift, machine learning systems typically fail to gene...
06/16/2022

Domain Generalization via Selective Consistency Regularization for Time Series Classification

Domain generalization methods aim to learn models robust to domain shift...
03/15/2022

Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization

Arbitrary style transfer (AST) and domain generalization (DG) are import...
05/16/2022

Generalizing to Evolving Domains with Latent Structure-Aware Sequential Autoencoder

Domain generalization aims to improve the generalization capability of m...
08/07/2022

Learning Modular Structures That Generalize Out-of-Distribution

Out-of-distribution (O.O.D.) generalization remains to be a key challeng...