Toward Better Generalization Bounds with Locally Elastic Stability

10/27/2020
by   Zhun Deng, et al.
0

Classical approaches in learning theory are often seen to yield very loose generalization bounds for deep neural networks. Using the example of "stability and generalization" <cit.>, however, we demonstrate that generalization bounds can be significantly improved by taking into account refined characteristics of modern neural networks. Specifically, this paper proposes a new notion of algorithmic stability termed locally elastic stability in light of a certain phenomenon in the training of neural networks <cit.>. We prove that locally elastic stability implies a tighter generalization bound than that derived based on uniform stability in many situations. When applied to deep neural networks, our new generalization bound attaches much more meaningful confidence statements to the performance on unseen data than existing algorithmic stability notions, thereby shedding light on the effectiveness of modern neural networks in real-world applications.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/16/2022

On the generalization of learning algorithms that do not converge

Generalization analyses of deep learning typically assume that the train...
research
12/12/2012

Almost-everywhere algorithmic stability and generalization error

We explore in some detail the notion of algorithmic stability as a viabl...
research
05/03/2019

Stability and Generalization of Graph Convolutional Neural Networks

Inspired by convolutional neural networks on 1D and 2D data, graph convo...
research
10/15/2019

The Local Elasticity of Neural Networks

This paper presents a phenomenon in neural networks that we refer to as ...
research
10/22/2020

Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity

As a popular approach to modeling the dynamics of training overparametri...
research
11/28/2021

On the Robustness and Generalization of Deep Learning Driven Full Waveform Inversion

The data-driven approach has been demonstrated as a promising technique ...
research
06/03/2019

A necessary and sufficient stability notion for adaptive generalization

We introduce a new notion of the stability of computations, which holds ...

Please sign up or login with your details

Forgot password? Click here to reset