Estimating Generalization under Distribution Shifts via Domain-Invariant Representations

by   Ching-Yao Chuang, et al.

When machine learning models are deployed on a test distribution different from the training distribution, they can perform poorly, but overestimate their performance. In this work, we aim to better estimate a model's performance under distribution shift, without supervision. To do so, we use a set of domain-invariant predictors as a proxy for the unknown, true target labels. Since the error of the resulting risk estimate depends on the target risk of the proxy model, we study generalization of domain-invariant representations and show that the complexity of the latent representation has a significant influence on the target risk. Empirically, our approach (1) enables self-tuning of domain adaptation models, and (2) accurately estimates the target error of given models under distribution shift. Other applications include model selection, deciding early stopping and error detection.


page 1

page 2

page 3

page 4


Improving Out-of-Distribution Robustness via Selective Augmentation

Machine learning algorithms typically assume that training and test exam...

Mandoline: Model Evaluation under Distribution Shift

Machine learning models are often deployed in different settings than th...

Towards Optimization and Model Selection for Domain Generalization: A Mixup-guided Solution

The distribution shifts between training and test data typically undermi...

Learning to Ignore: Fair and Task Independent Representations

Training fair machine learning models, aiming for their interpretability...

Causally-motivated Shortcut Removal Using Auxiliary Labels

Robustness to certain distribution shifts is a key requirement in many M...

Fair Predictors under Distribution Shift

Recent work on fair machine learning adds to a growing set of algorithmi...

Learning Representations that Support Robust Transfer of Predictors

Ensuring generalization to unseen environments remains a challenge. Doma...