
Onepass Stochastic Gradient Descent in Overparametrized Twolayer Neural Networks
There has been a recent surge of interest in understanding the convergen...
read it

When Do Neural Networks Outperform Kernel Methods?
For a certain scaling of the initialization of stochastic gradient desce...
read it

Tensor Programs III: Neural Matrix Laws
In a neural network (NN), weight matrices linearly transform inputs into...
read it

Disentangling trainability and generalization in deep learning
A fundamental goal in deep learning is the characterization of trainabil...
read it

Tensor Programs II: Neural Tangent Kernel for Any Architecture
We prove that a randomly initialized neural network of *any architecture...
read it

The Dynamics of Learning: A Random Matrix Approach
Understanding the learning dynamics of neural networks is one of the key...
read it

Batch Normalization Orthogonalizes Representations in Deep Random Networks
This paper underlines a subtle property of batchnormalization (BN): Suc...
read it
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
Several recent trends in machine learning theory and practice, from the design of stateoftheart Gaussian Process to the convergence analysis of deep neural nets (DNNs) under stochastic gradient descent (SGD), have found it fruitful to study wide random neural networks. Central to these approaches are certain scaling limits of such networks. We unify these results by introducing a notion of a straightline tensor program that can express most neural network computations, and we characterize its scaling limit when its tensors are large and randomized. From our framework follows (1) the convergence of random neural networks to Gaussian processes for architectures such as recurrent neural networks, convolutional neural networks, residual networks, attention, and any combination thereof, with or without batch normalization; (2) conditions under which the gradient independence assumption  that weights in backpropagation can be assumed to be independent from weights in the forward pass  leads to correct computation of gradient dynamics, and corrections when it does not; (3) the convergence of the Neural Tangent Kernel, a recently proposed kernel used to predict training dynamics of neural networks under gradient descent, at initialization for all architectures in (1) without batch normalization. Mathematically, our framework is general enough to rederive classical random matrix results such as the semicircle and the MarchenkoPastur laws, as well as recent results in neural network Jacobian singular values. We hope our work opens a way toward design of even stronger Gaussian Processes, initialization schemes to avoid gradient explosion/vanishing, and deeper understanding of SGD dynamics in modern architectures.
READ FULL TEXT
Comments
There are no comments yet.