DeepAI
Log In Sign Up

Evaluating Robustness to Dataset Shift via Parametric Robustness Sets

05/31/2022
by   Nikolaj Thams, et al.
0

We give a method for proactively identifying small, plausible shifts in distribution which lead to large differences in model performance. To ensure that these shifts are plausible, we parameterize them in terms of interpretable changes in causal mechanisms of observed variables. This defines a parametric robustness set of plausible distributions and a corresponding worst-case loss. While the loss under an individual parametric shift can be estimated via reweighting techniques such as importance sampling, the resulting worst-case optimization problem is non-convex, and the estimate may suffer from large variance. For small shifts, however, we can construct a local second-order approximation to the loss under shift and cast the problem of finding a worst-case shift as a particular non-convex quadratic optimization problem, for which efficient algorithms are available. We demonstrate that this second-order approximation can be estimated directly for shifts in conditional exponential family models, and we bound the approximation error. We apply our approach to a computer vision task (classifying gender from images), revealing sensitivity to shifts in non-causal attributes.

READ FULL TEXT

page 33

page 35

10/28/2020

Evaluating Model Robustness to Dataset Shift

As the use of machine learning in safety-critical domains becomes widesp...
10/21/2020

Worst-case sensitivity

We introduce the notion of Worst-Case Sensitivity, defined as the worst-...
07/28/2017

Efficient Algorithms for Non-convex Isotonic Regression through Submodular Optimization

We consider the minimization of submodular functions subject to ordering...
07/05/2020

Robust Causal Inference Under Covariate Shift via Worst-Case Subpopulation Treatment Effects

We propose the worst-case treatment effect (WTE) across all subpopulatio...
05/19/2021

Provable Guarantees on the Robustness of Decision Rules to Causal Interventions

Robustness of decision rules to shifts in the data-generating process is...
02/18/2021

Reduced-Order Neural Network Synthesis with Robustness Guarantees

In the wake of the explosive growth in smartphones and cyberphysical sys...
09/11/2020

Computer-Aided Generation of N-shift RWS

Generating schedules for shift workers is essential for many employers, ...