Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems

11/26/2021
by   fcq, et al.
0

Binary perceptron is a fundamental model of supervised learning for the non-convex optimization, which is a root of the popular deep learning. Binary perceptron is able to achieve a classification of random high-dimensional data by computing the marginal probabilities of binary synapses. The relationship between the algorithmic instability and the equilibrium analysis of the model remains elusive. Here, we establish the relationship by showing that the instability condition around the algorithmic fixed point is identical to the instability for breaking the replica symmetric saddle point solution of the free energy function. Therefore, our analysis provides insights towards bridging the gap between non-convex learning dynamics and statistical mechanics properties of more complex neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/06/2021

Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning

In nature, symmetry governs regularities, while symmetry breaking brings...
research
05/30/2022

Non-convex online learning via algorithmic equivalence

We study an algorithmic equivalence technique between nonconvex gradient...
research
09/08/2019

On the connections between algorithmic regularization and penalization for convex losses

In this work we establish the equivalence of algorithmic regularization ...
research
11/15/2019

Clustering of solutions in the symmetric binary perceptron

The geometrical features of the (non-convex) loss landscape of neural ne...
research
05/31/2023

Spontaneous symmetry breaking in generative diffusion models

Generative diffusion models have recently emerged as a leading approach ...
research
11/27/2018

A Scalable Optimization Mechanism for Pairwise based Discrete Hashing

Maintaining the pair similarity relationship among originally high-dimen...
research
02/21/2019

Active online learning in the binary perceptron problem

The binary perceptron is the simplest artificial neural network formed b...

Please sign up or login with your details

Forgot password? Click here to reset