
A Quantum Field Theory of Representation Learning
Continuous symmetries and their breaking play a prominent role in contem...
read it

On the Complexity of Breaking Symmetry
We can break symmetry by eliminating solutions within a symmetry class t...
read it

Statistical physics of unsupervised learning with prior knowledge in neural networks
Integrating sensory inputs with prior beliefs from past experiences in u...
read it

Fiducial Symmetry in Action
Symmetry is key in classical and modern physics. A striking example is c...
read it

Spontaneous Symmetry Breaking in Neural Networks
We propose a framework to understand the unprecedented performance and r...
read it

Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
Some interesting recent advances in the theoretical understanding of neu...
read it

Replica Symmetry and Replica Symmetry Breaking for the Traveling Salesperson Problem
We study the energy landscape of the Traveling Salesperson problem (TSP)...
read it
Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning
In nature, symmetry governs regularities, while symmetry breaking brings texture. Here, we reveal a novel role of symmetry breaking behind efficiency and stability in learning, a critical issue in machine learning. Recent experiments suggest that the symmetry of the loss function is closely related to the learning performance. This raises a fundamental question. Is such symmetry beneficial, harmful, or irrelevant to the success of learning? Here, we demystify this question and pose symmetry breaking as a new design principle by considering the symmetry of the learning rule in addition to the loss function. We model the discrete learning dynamics using a continuoustime Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. We identify kinetic asymmetry unique to learning systems, where the kinetic energy often does not have the same symmetry as the potential (loss) function reflecting the nonphysical symmetries of the loss function and the nonEuclidean metric used in learning rules. We generalize Noether's theorem known in physics to explicitly take into account this kinetic asymmetry and derive the resulting motion of the Noether charge. Finally, we apply our theory to modern deep networks with normalization layers and reveal a mechanism of implicit adaptive optimization induced by the kinetic symmetry breaking.
READ FULL TEXT
Comments
There are no comments yet.