The Hidden-Manifold Hopfield Model and a learning phase transition
share
The Hopfield model has a long-standing tradition in statistical physics, being one of the few neural networks for which a theory is available. Extending the theory of Hopfield models for correlated data could help understand the success of deep neural networks, for instance describing how they extract features from data. Motivated by this, we propose and investigate a generalized Hopfield model that we name Hidden-Manifold Hopfield Model: we generate the couplings from P=α N examples with the Hebb rule using a non-linear transformation of D=α_D N random vectors that we call factors, with N the number of neurons. Using the replica method, we obtain a phase diagram for the model that shows a phase transition where the factors hidden in the examples become attractors of the dynamics; this phase exists above a critical value of α and below a critical value of α_D. We call this behaviour learning transition.
READ FULL TEXT