Minimax Optimal Estimation of Stability Under Distribution Shift

12/13/2022
by   Hongseok Namkoong, et al.
1

The performance of decision policies and prediction models often deteriorates when applied to environments different from the ones seen during training. To ensure reliable operation, we propose and analyze the stability of a system under distribution shift, which is defined as the smallest change in the underlying environment that causes the system's performance to deteriorate beyond a permissible threshold. In contrast to standard tail risk measures and distributionally robust losses that require the specification of a plausible magnitude of distribution shift, the stability measure is defined in terms of a more intuitive quantity: the level of acceptable performance degradation. We develop a minimax optimal estimator of stability and analyze its convergence rate, which exhibits a fundamental phase shift behavior. Our characterization of the minimax convergence rate shows that evaluating stability against large performance degradation incurs a statistical cost. Empirically, we demonstrate the practical utility of our stability framework by using it to compare system designs on problems where robustness to distribution shift is critical.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/22/2019

Minimax Rate Optimal Adaptive Nearest Neighbor Classification and Regression

k Nearest Neighbor (kNN) method is a simple and popular statistical meth...
research
09/09/2020

Template Matching and Change Point Detection by M-estimation

We consider the fundamental problem of matching a template to a signal. ...
research
04/11/2022

Local convergence rates of the least squares estimator with applications to transfer learning

Convergence properties of empirical risk minimizers can be conveniently ...
research
05/06/2022

Optimally tackling covariate shift in RKHS-based nonparametric regression

We study the covariate shift problem in the context of nonparametric reg...
research
11/20/2018

Convergence rate of optimal quantization grids and application to empirical measure

We study the convergence rate of optimal quantization for a probability ...
research
02/17/2021

Outside the Echo Chamber: Optimizing the Performative Risk

In performative prediction, predictions guide decision-making and hence ...

Please sign up or login with your details

Forgot password? Click here to reset