Log In Sign Up

Evaluating Robustness to Dataset Shift via Parametric Robustness Sets

by   Nikolaj Thams, et al.

We give a method for proactively identifying small, plausible shifts in distribution which lead to large differences in model performance. To ensure that these shifts are plausible, we parameterize them in terms of interpretable changes in causal mechanisms of observed variables. This defines a parametric robustness set of plausible distributions and a corresponding worst-case loss. While the loss under an individual parametric shift can be estimated via reweighting techniques such as importance sampling, the resulting worst-case optimization problem is non-convex, and the estimate may suffer from large variance. For small shifts, however, we can construct a local second-order approximation to the loss under shift and cast the problem of finding a worst-case shift as a particular non-convex quadratic optimization problem, for which efficient algorithms are available. We demonstrate that this second-order approximation can be estimated directly for shifts in conditional exponential family models, and we bound the approximation error. We apply our approach to a computer vision task (classifying gender from images), revealing sensitivity to shifts in non-causal attributes.


page 33

page 35


Evaluating Model Robustness to Dataset Shift

As the use of machine learning in safety-critical domains becomes widesp...

Worst-case sensitivity

We introduce the notion of Worst-Case Sensitivity, defined as the worst-...

Efficient Algorithms for Non-convex Isotonic Regression through Submodular Optimization

We consider the minimization of submodular functions subject to ordering...

Robust Causal Inference Under Covariate Shift via Worst-Case Subpopulation Treatment Effects

We propose the worst-case treatment effect (WTE) across all subpopulatio...

Provable Guarantees on the Robustness of Decision Rules to Causal Interventions

Robustness of decision rules to shifts in the data-generating process is...

Reduced-Order Neural Network Synthesis with Robustness Guarantees

In the wake of the explosive growth in smartphones and cyberphysical sys...

Computer-Aided Generation of N-shift RWS

Generating schedules for shift workers is essential for many employers, ...