DeepAI AI Chat
Log In Sign Up

Correlation Functions in Random Fully Connected Neural Networks at Finite Width

04/03/2022
by   Boris Hanin, et al.
0

This article considers fully connected neural networks with Gaussian random weights and biases and L hidden layers, each of width proportional to a large parameter n. For polynomially bounded non-linearities we give sharp estimates in powers of 1/n for the joint correlation functions of the network output and its derivatives. Moreover, we obtain exact layerwise recursions for these correlation functions and solve a number of special cases for classes of non-linearities including ReLU and tanh. We find in both cases that the depth-to-width ratio L/n plays the role of an effective network depth, controlling both the scale of fluctuations at individual neurons and the size of inter-neuron correlations. We use this to study a somewhat simplified version of the so-called exploding and vanishing gradient problem, proving that this particular variant occurs if and only if L/n is large. Several of the key ideas in this article were first developed at a physics level of rigor in a recent monograph with Roberts and Yaida.

READ FULL TEXT

page 1

page 2

page 3

page 4

08/29/2019

On the rate of convergence of fully connected very deep neural network regression estimates

Recent results in nonparametric regression show that deep learning, i.e....
07/04/2021

Random Neural Networks in the Infinite Width Limit as Gaussian Processes

This article gives a new proof that fully connected neural networks with...
05/19/2022

Neural Network Architecture Beyond Width and Depth

This paper proposes a new neural network architecture by introducing an ...
07/31/2021

The Separation Capacity of Random Neural Networks

Neural networks with random weights appear in a variety of machine learn...
06/18/2019

Approximation power of random neural networks

This paper investigates the approximation power of three types of random...
06/12/2019

Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective

A series of recent works suggest that deep neural networks (DNNs), of fi...
07/03/2021

Slope and generalization properties of neural networks

Neural networks are very successful tools in for example advanced classi...