
Training Dynamics of Deep Networks using Stochastic Gradient Descent via Neural Tangent Kernel
Stochastic Gradient Descent (SGD) is widely used to train deep neural ne...
read it

Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
Wide neural networks with random weights and biases are Gaussian process...
read it

Tensor Programs II: Neural Tangent Kernel for Any Architecture
We show that a randomly initialized neural network of *any architecture*...
read it

When Do Neural Networks Outperform Kernel Methods?
For a certain scaling of the initialization of stochastic gradient desce...
read it

Disentangling trainability and generalization in deep learning
A fundamental goal in deep learning is the characterization of trainabil...
read it

The Dynamics of Learning: A Random Matrix Approach
Understanding the learning dynamics of neural networks is one of the key...
read it

A Gaussian Process perspective on Convolutional Neural Networks
In this paper we cast the wellknown convolutional neural network in a G...
read it
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
Several recent trends in machine learning theory and practice, from the design of stateoftheart Gaussian Process to the convergence analysis of deep neural nets (DNNs) under stochastic gradient descent (SGD), have found it fruitful to study wide random neural networks. Central to these approaches are certain scaling limits of such networks. We unify these results by introducing a notion of a straightline tensor program that can express most neural network computations, and we characterize its scaling limit when its tensors are large and randomized. From our framework follows (1) the convergence of random neural networks to Gaussian processes for architectures such as recurrent neural networks, convolutional neural networks, residual networks, attention, and any combination thereof, with or without batch normalization; (2) conditions under which the gradient independence assumption  that weights in backpropagation can be assumed to be independent from weights in the forward pass  leads to correct computation of gradient dynamics, and corrections when it does not; (3) the convergence of the Neural Tangent Kernel, a recently proposed kernel used to predict training dynamics of neural networks under gradient descent, at initialization for all architectures in (1) without batch normalization. Mathematically, our framework is general enough to rederive classical random matrix results such as the semicircle and the MarchenkoPastur laws, as well as recent results in neural network Jacobian singular values. We hope our work opens a way toward design of even stronger Gaussian Processes, initialization schemes to avoid gradient explosion/vanishing, and deeper understanding of SGD dynamics in modern architectures.
READ FULL TEXT
Comments
There are no comments yet.