Log In Sign Up

Gradient Matching for Domain Generalization

by   Yuge Shi, et al.

Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive – requires computation of second-order derivatives – we derive a simpler first-order algorithm named Fish that approximates its optimization. We demonstrate the efficacy of Fish on 6 datasets from the Wilds benchmark, which captures distribution shift across a diverse range of modalities. Our method produces competitive results on these datasets and surpasses all baselines on 4 of them. We perform experiments on both the Wilds benchmark, which captures distribution shift in the real world, as well as datasets in DomainBed benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks.


SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization

A major bottleneck in the real-world applications of machine learning mo...

Domain Generalization via Gradient Surgery

In real-life applications, machine learning models often face scenarios ...

HMOE: Hypernetwork-based Mixture of Experts for Domain Generalization

Due to the domain shift, machine learning systems typically fail to gene...

Domain Generalization via Selective Consistency Regularization for Time Series Classification

Domain generalization methods aim to learn models robust to domain shift...

Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization

Arbitrary style transfer (AST) and domain generalization (DG) are import...

Generalizing to Evolving Domains with Latent Structure-Aware Sequential Autoencoder

Domain generalization aims to improve the generalization capability of m...

Learning Modular Structures That Generalize Out-of-Distribution

Out-of-distribution (O.O.D.) generalization remains to be a key challeng...