A note on Linear Bottleneck networks and their Transition to Multilinearity

06/30/2022
by   Libin Zhu, et al.
14

Randomly initialized wide neural networks transition to linear functions of weights as the width grows, in a ball of radius O(1) around initialization. A necessary condition for this result is that all layers of the network are wide enough, i.e., all widths tend to infinity. However, the transition to linearity breaks down when this infinite width assumption is violated. In this work we show that linear networks with a bottleneck layer learn bilinear functions of the weights, in a ball of radius O(1) around initialization. In general, for B-1 bottleneck layers, the network is a degree B multilinear function of weights. Importantly, the degree only depends on the number of bottlenecks and not the total depth of the network.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/03/2020

Wide Neural Networks with Bottlenecks are Deep Gaussian Processes

There is recently much work on the "wide limit" of neural networks, wher...
research
05/24/2022

Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture

In this paper we show that feedforward neural networks corresponding to ...
research
06/18/2019

Approximation power of random neural networks

This paper investigates the approximation power of three types of random...
research
03/27/2022

On the Neural Tangent Kernel Analysis of Randomly Pruned Wide Neural Networks

We study the behavior of ultra-wide neural networks when their weights a...
research
10/02/2020

On the linearity of large non-linear models: when and why the tangent kernel is constant

The goal of this work is to shed light on the remarkable phenomenon of t...
research
06/07/2021

Batch Normalization Orthogonalizes Representations in Deep Random Networks

This paper underlines a subtle property of batch-normalization (BN): Suc...
research
07/01/2021

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

We analyze the learning dynamics of infinitely wide neural networks with...

Please sign up or login with your details

Forgot password? Click here to reset