Stable behaviour of infinitely wide deep neural networks

03/01/2020
by   Stefano Favaro, et al.
0

We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/07/2021

Infinite-channel deep stable convolutional neural networks

The interplay between infinite-width neural networks (NNs) and classes o...
research
02/20/2021

Large-width functional asymptotics for deep Gaussian neural networks

In this paper, we consider fully connected feed-forward deep neural netw...
research
06/18/2021

α-Stable convergence of heavy-tailed infinitely-wide neural networks

We consider infinitely-wide multi-layer perceptrons (MLPs) which are lim...
research
01/11/2021

Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks

Infinite width limits of deep neural networks often have tractable forms...
research
10/05/2021

On the Impact of Stable Ranks in Deep Nets

A recent line of work has established intriguing connections between the...
research
04/08/2023

Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions

There is a growing literature on the study of large-width properties of ...
research
07/10/2020

Characteristics of Monte Carlo Dropout in Wide Neural Networks

Monte Carlo (MC) dropout is one of the state-of-the-art approaches for u...

Please sign up or login with your details

Forgot password? Click here to reset