Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?

06/05/2021
by   Dinghuai Zhang, et al.
0

Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and demonstrate the functional lottery ticket hypothesis: full network contains a subnetwork that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the subnetwork structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.

READ FULL TEXT
08/07/2022

Learning Modular Structures That Generalize Out-of-Distribution

Out-of-distribution (O.O.D.) generalization remains to be a key challeng...
12/20/2021

General Greedy De-bias Learning

Neural networks often make predictions relying on the spurious correlati...
08/16/2022

Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization

Learning invariant (causal) features for out-of-distribution (OOD) gener...
07/12/2018

Fast Modular Subset Sum using Linear Sketching

Given n positive integers, the Modular Subset Sum problem asks if a subs...
05/25/2022

Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlation

Many recent works indicate that the deep neural networks tend to take da...
11/15/2021

Axiomatic characterization of the χ^2 dissimilarity measure

We axiomatically characterize the χ^2 dissimilarity measure. To this end...
11/08/2019

Newmark algorithm for dynamic analysis with Maxwell chain model

This paper investigates a time-stepping procedure of the Newmark type fo...