
A Deep Conditioning Treatment of Neural Networks
We study the role of depth in training randomly initialized overparamete...
read it

A Quantum Field Theory of Representation Learning
Continuous symmetries and their breaking play a prominent role in contem...
read it

Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning
In nature, symmetry governs regularities, while symmetry breaking brings...
read it

Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?
Deep neural networks are typically initialized with random weights, with...
read it

Capacity allocation through neural network layers
Capacity analysis has been recently introduced as a way to analyze how l...
read it

The Early Phase of Neural Network Training
Recent studies have shown that many important aspects of neural network ...
read it

Response to Comment on "Alloptical machine learning using diffractive deep neural networks"
In their Comment, Wei et al. (arXiv:1809.08360v1 [cs.LG]) claim that our...
read it
Spontaneous Symmetry Breaking in Neural Networks
We propose a framework to understand the unprecedented performance and robustness of deep neural networks using field theory. Correlations between the weights within the same layer can be described by symmetries in that layer, and networks generalize better if such symmetries are broken to reduce the redundancies of the weights. Using a two parameter field theory, we find that the network can break such symmetries itself towards the end of training in a process commonly known in physics as spontaneous symmetry breaking. This corresponds to a network generalizing itself without any user input layers to break the symmetry, but by communication with adjacent layers. In the layer decoupling limit applicable to residual networks (He et al., 2015), we show that the remnant symmetries that survive the nonlinear layers are spontaneously broken. The Lagrangian for the nonlinear and weight layers together has striking similarities with the one in quantum field theory of a scalar. Using results from quantum field theory we show that our framework is able to explain many experimentally observed phenomena,such as training on random labels with zero error (Zhang et al., 2017), the information bottleneck, the phase transition out of it and gradient variance explosion (ShwartzZiv & Tishby, 2017), shattered gradients (Balduzzi et al., 2017), and many more.
READ FULL TEXT
Comments
There are no comments yet.