On Calibration and Out-of-domain Generalization

02/20/2021
by   Yoav Wald, et al.
0

Out-of-domain (OOD) generalization is a significant challenge for machine learning models. To overcome it, many novel techniques have been proposed, often focused on learning models with certain invariance properties. In this work, we draw a link between OOD performance and model calibration, arguing that calibration across multiple domains can be viewed as a special case of an invariant representation leading to better OOD generalization. Specifically, we prove in a simplified setting that models which achieve multi-domain calibration are free of spurious correlations. This leads us to propose multi-domain calibration as a measurable surrogate for the OOD performance of a classifier. An important practical benefit of calibration is that there are many effective tools for calibrating classifiers. We show that these tools are easy to apply and adapt for a multi-domain setting. Using five datasets from the recently proposed WILDS OOD benchmark we demonstrate that simply re-calibrating models across multiple domains in a validation set leads to significantly improved performance on unseen test domains. We believe this intriguing connection between calibration and OOD generalization is promising from a practical point of view and deserves further research from a theoretical point of view.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/20/2013

Failure of Calibration is Typical

Schervish (1985b) showed that every forecasting system is noncalibrated ...
research
03/30/2021

Progressive Domain Expansion Network for Single Domain Generalization

Single domain generalization is a challenging case of model generalizati...
research
05/16/2021

Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization

Post-training quantization methods use a set of calibration data to comp...
research
11/25/2020

Batch Normalization Embeddings for Deep Domain Generalization

Domain generalization aims at training machine learning models to perfor...
research
05/23/2022

Feature-Distribution Perturbation and Calibration for Generalized Person ReID

Person Re-identification (ReID) has been advanced remarkably over the la...
research
03/16/2022

On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers

Every uncalibrated classifier has a corresponding true calibration map t...
research
07/27/2020

A calibration-free method for biosensing in cell manufacturing

Chimeric antigen receptor T cell therapy has demonstrated innovative the...

Please sign up or login with your details

Forgot password? Click here to reset