On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond

06/13/2018
by   Xingguo Li, et al.
2

Our paper proposes a generalization error bound for a general family of deep neural networks based on the spectral norm of weight matrices. Through introducing a novel characterization of the Lipschitz properties of neural network family, we achieve a tighter generalization error bound for ultra-deep neural networks, whose depth is much larger than the square root of its width. Besides the general deep neural networks, our results can be applied to derive new bounds for several popular architectures, including convolutional neural networks (CNNs), residual networks (ResNets), and hyperspherical networks (SphereNets). In the regime that the depth of these architectures is dominating, our bounds allow for the choice of much larger parameter spaces of weight matrices, inducing potentially stronger expressive ability.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/03/2018

Understanding Weight Normalized Deep Neural Networks with Rectified Linear Units

This paper presents a general framework for norm-based capacity control ...
research
10/03/2019

Generalization Bounds for Convolutional Neural Networks

Convolutional neural networks (CNNs) have achieved breakthrough performa...
research
05/26/2016

Robust Large Margin Deep Neural Networks

The generalization error of deep neural networks via their classificatio...
research
04/02/2019

Why ResNet Works? Residuals Generalize

Residual connections significantly boost the performance of deep neural ...
research
02/12/2023

Koopman-Based Bound for Generalization: New Aspect of Neural Networks Regarding Nonlinear Noise Filtering

We propose a new bound for generalization of neural networks using Koopm...
research
05/29/2019

Improved Generalisation Bounds for Deep Learning Through L^∞ Covering Numbers

Using proof techniques involving L^∞ covering numbers, we show generalis...
research
08/04/2018

On Lipschitz Bounds of General Convolutional Neural Networks

Many convolutional neural networks (CNNs) have a feed-forward structure....

Please sign up or login with your details

Forgot password? Click here to reset