Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?

by   Dinghuai Zhang, et al.

Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and demonstrate the functional lottery ticket hypothesis: full network contains a subnetwork that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the subnetwork structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.


Exploring Optimal Substructure for Out-of-distribution Generalization via Feature-targeted Model Pruning

Recent studies show that even highly biased dense networks contain an un...

Learning Modular Structures That Generalize Out-of-Distribution

Out-of-distribution (O.O.D.) generalization remains to be a key challeng...

General Greedy De-bias Learning

Neural networks often make predictions relying on the spurious correlati...

Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization

Learning invariant (causal) features for out-of-distribution (OOD) gener...

SFP: Spurious Feature-targeted Pruning for Out-of-Distribution Generalization

Model substructure learning aims to find an invariant network substructu...

Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification

Robustness is of central importance in machine learning and has given ri...

Generalized Hadamard differentiability of the copula mapping and its applications

We consider the copula mapping, which maps a joint cumulative distributi...

Please sign up or login with your details

Forgot password? Click here to reset