DeepAI AI Chat
Log In Sign Up

Correlation Functions in Random Fully Connected Neural Networks at Finite Width

by   Boris Hanin, et al.

This article considers fully connected neural networks with Gaussian random weights and biases and L hidden layers, each of width proportional to a large parameter n. For polynomially bounded non-linearities we give sharp estimates in powers of 1/n for the joint correlation functions of the network output and its derivatives. Moreover, we obtain exact layerwise recursions for these correlation functions and solve a number of special cases for classes of non-linearities including ReLU and tanh. We find in both cases that the depth-to-width ratio L/n plays the role of an effective network depth, controlling both the scale of fluctuations at individual neurons and the size of inter-neuron correlations. We use this to study a somewhat simplified version of the so-called exploding and vanishing gradient problem, proving that this particular variant occurs if and only if L/n is large. Several of the key ideas in this article were first developed at a physics level of rigor in a recent monograph with Roberts and Yaida.


page 1

page 2

page 3

page 4


On the rate of convergence of fully connected very deep neural network regression estimates

Recent results in nonparametric regression show that deep learning, i.e....

Random Neural Networks in the Infinite Width Limit as Gaussian Processes

This article gives a new proof that fully connected neural networks with...

Neural Network Architecture Beyond Width and Depth

This paper proposes a new neural network architecture by introducing an ...

The Separation Capacity of Random Neural Networks

Neural networks with random weights appear in a variety of machine learn...

Approximation power of random neural networks

This paper investigates the approximation power of three types of random...

Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective

A series of recent works suggest that deep neural networks (DNNs), of fi...

Slope and generalization properties of neural networks

Neural networks are very successful tools in for example advanced classi...