Wide Neural Networks with Bottlenecks are Deep Gaussian Processes

01/03/2020
by   Devanshu Agrawal, et al.
0

There is recently much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain narrow. In this paper, we consider the wide limit of BNNs where some hidden layers, called "bottlenecks", are held at finite width. The result is a composition of GPs that we term a "bottleneck neural network Gaussian process" (bottleneck NNGP). Although intuitive, the subtlety of the proof is in showing that the wide limit of a composition of networks is in fact the composition of the limiting GPs. We also analyze theoretically a single-bottleneck NNGP, finding that the bottleneck induces dependence between the outputs of a multi-output network that persists through infinite post-bottleneck depth, and prevents the kernel of the network from losing discriminative power at infinite post-bottleneck depth.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 13

11/29/2021

Dependence between Bayesian neural network units

The connection between Bayesian neural networks and Gaussian processes g...
07/01/2021

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

We analyze the learning dynamics of infinitely wide neural networks with...
01/07/2021

Infinitely Wide Tensor Networks as Gaussian Process

Gaussian Process is a non-parametric prior which can be understood as a ...
04/02/2020

Predicting the outputs of finite networks trained with noisy gradients

A recent line of studies has focused on the infinite width limit of deep...
11/23/2021

Depth induces scale-averaging in overparameterized linear Bayesian neural networks

Inference in deep Bayesian neural networks is only fully understood in t...
08/29/2021

Neural Network Gaussian Processes by Increasing Depth

Recent years have witnessed an increasing interest in the correspondence...
06/11/2021

The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective

Large width limits have been a recent focus of deep learning research: m...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.