DeepAI AI Chat
Log In Sign Up

Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning

by   Hidenori Tanaka, et al.

In nature, symmetry governs regularities, while symmetry breaking brings texture. Here, we reveal a novel role of symmetry breaking behind efficiency and stability in learning, a critical issue in machine learning. Recent experiments suggest that the symmetry of the loss function is closely related to the learning performance. This raises a fundamental question. Is such symmetry beneficial, harmful, or irrelevant to the success of learning? Here, we demystify this question and pose symmetry breaking as a new design principle by considering the symmetry of the learning rule in addition to the loss function. We model the discrete learning dynamics using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. We identify kinetic asymmetry unique to learning systems, where the kinetic energy often does not have the same symmetry as the potential (loss) function reflecting the non-physical symmetries of the loss function and the non-Euclidean metric used in learning rules. We generalize Noether's theorem known in physics to explicitly take into account this kinetic asymmetry and derive the resulting motion of the Noether charge. Finally, we apply our theory to modern deep networks with normalization layers and reveal a mechanism of implicit adaptive optimization induced by the kinetic symmetry breaking.


page 1

page 2

page 3

page 4


A Quantum Field Theory of Representation Learning

Continuous symmetries and their breaking play a prominent role in contem...

Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems

Binary perceptron is a fundamental model of supervised learning for the ...

Statistical physics of unsupervised learning with prior knowledge in neural networks

Integrating sensory inputs with prior beliefs from past experiences in u...

Fiducial Symmetry in Action

Symmetry is key in classical and modern physics. A striking example is c...

Time-Reversal Symmetric ODE Network

Time-reversal symmetry, which requires that the dynamics of a system sho...

Spontaneous Symmetry Breaking in Neural Networks

We propose a framework to understand the unprecedented performance and r...

Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks

Some interesting recent advances in the theoretical understanding of neu...