Exact Solutions of a Deep Linear Network

02/10/2022

∙

by   Liu Ziyin, et al.

∙

0

∙

share

This work finds the exact solutions to a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that weight decay strongly interacts with the model architecture and can create bad minima in a network with more than 1 hidden layer, qualitatively different for a network with only 1 hidden layer. As an application, we also analyze stochastic nets and show that their prediction variance vanishes to zero as the stochasticity, the width, or the depth tends to infinity.