Robustness Implies Generalization via Data-Dependent Generalization Bounds

06/27/2022
by   Kenji Kawaguchi, et al.
0

This paper proves that robustness implies generalization via data-dependent generalization bounds. As a result, robustness and generalization are shown to be connected closely in a data-dependent manner. Our bounds improve previous bounds in two directions, to solve an open problem that has seen little development since 2010. The first is to reduce the dependence on the covering number. The second is to remove the dependence on the hypothesis space. We present several examples, including ones for lasso and deep learning, in which our bounds are provably preferable. The experiments on real-world data and theoretical models demonstrate near-exponential improvements in various situations. To achieve these improvements, we do not require additional assumptions on the unknown distribution; instead, we only incorporate an observable and computable property of the training samples. A key technical innovation is an improved concentration bound for multinomial random variables that is of independent interest beyond robustness and generalization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/29/2019

Improved Generalisation Bounds for Deep Learning Through L^∞ Covering Numbers

Using proof techniques involving L^∞ covering numbers, we show generalis...
research
03/25/2022

Generalization bounds for learning under graph-dependence: A survey

Traditional statistical learning theory relies on the assumption that da...
research
09/05/2019

McDiarmid-Type Inequalities for Graph-Dependent Variables and Stability Bounds

A crucial assumption in most statistical learning theory is that samples...
research
10/16/2017

Generalization in Deep Learning

This paper explains why deep learning can generalize well, despite large...
research
03/09/2023

Data-dependent Generalization Bounds via Variable-Size Compressibility

In this paper, we establish novel data-dependent upper bounds on the gen...
research
12/31/2017

Benford's Law Beyond Independence: Tracking Benford Behavior in Copula Models

Benford's law describes a common phenomenon among many naturally occurri...
research
02/16/2022

Dependence comparisons of order statistics in the proportional hazards model

Let X_1, … , X_n be mutually independent exponential random variables wi...

Please sign up or login with your details

Forgot password? Click here to reset