Mean Field Analysis of Neural Networks: A Central Limit Theorem

08/28/2018
by   Justin Sirignano, et al.
4

Machine learning has revolutionized fields such as image, text, and speech recognition. There's also growing interest in applying machine and deep learning methods in science, engineering, medicine, and finance. Despite their immense success in practice, there is limited mathematical understanding of neural networks. We mathematically study neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously prove that the neural network satisfies a central limit theorem. Our result describes the neural network's fluctuations around its mean-field limit. The fluctuations have a Gaussian distribution and satisfy a stochastic partial differential equation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/11/2019

Mean Field Analysis of Deep Neural Networks

We analyze multi-layer neural networks in the asymptotic regime of simul...
research
07/12/2022

Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent

The convergence of stochastic interacting particle systems in the mean-f...
research
07/09/2019

Scaling Limit of Neural Networks with the Xavier Initialization and Convergence to a Global Minimum

We analyze single-layer neural networks with the Xavier initialization i...
research
10/13/2022

Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence

The stochastic heavy ball method (SHB), also known as stochastic gradien...
research
10/11/2020

Convergence to the fixed-node limit in deep variational Monte Carlo

Variational quantum Monte Carlo (QMC) is an ab-initio method for solving...
research
02/22/2022

A duality connecting neural network and cosmological dynamics

We demonstrate that the dynamics of neural networks trained with gradien...

Please sign up or login with your details

Forgot password? Click here to reset