Gaussian Process Behaviour in Wide Deep Neural Networks

Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties. In this paper, we study the relationship between Gaussian processes with a recursive kernel definition and random wide fully connected feedforward networks with more than one hidden layer. We show that, under broad conditions, as we make the architecture increasingly wide, the implied random function converges in distribution to a Gaussian process, formalising and extending existing results by Neal (1996) to deep networks. To evaluate convergence rates empirically, we use maximum mean discrepancy. We then exhibit situations where existing Bayesian deep networks are close to Gaussian processes in terms of the key quantities of interest. Any Gaussian process has a flat representation. Since this behaviour may be undesirable in certain situations we discuss ways in which it might be prevented.

READ FULL TEXT
research
02/20/2021

Large-width functional asymptotics for deep Gaussian neural networks

In this paper, we consider fully connected feed-forward deep neural netw...
research
03/14/2022

Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks

Given any deep fully connected neural network, initialized with random G...
research
10/25/2018

A Gaussian Process perspective on Convolutional Neural Networks

In this paper we cast the well-known convolutional neural network in a G...
research
02/24/2014

Avoiding pathologies in very deep networks

Choosing appropriate architectures and regularization strategies for dee...
research
06/29/2018

Bayesian Deep Learning on a Quantum Computer

Bayesian methods in machine learning, such as Gaussian processes, have g...
research
08/11/2022

Gaussian process surrogate models for neural networks

The lack of insight into deep learning systems hinders their systematic ...
research
02/17/2021

Non-asymptotic approximations of neural networks by Gaussian processes

We study the extent to which wide neural networks may be approximated by...

Please sign up or login with your details

Forgot password? Click here to reset