Large Associative Memory Problem in Neurobiology and Machine Learning

08/16/2020
by   Dmitry Krotov, et al.
11

Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in ”Hopfield Networks is All You Need” paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/14/2021

Hierarchical Associative Memory

Dense Associative Memories or Modern Hopfield Networks have many appeali...
05/31/2021

A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]

In their recent paper titled "Large Associative Memory Problem in Neurob...
09/29/2020

Enforcing nonholonomic constraints in Aerobat, a roosting flapping wing model

Flapping wing flight is a challenging dynamical problem and is also a ve...
12/09/2016

Towards deep learning with spiking neurons in energy based models with contrastive Hebbian plasticity

In machine learning, error back-propagation in multi-layer neural networ...
08/04/2020

The world as a neural network

We discuss a possibility that the entire universe on its most fundamenta...
08/21/2020

Exploiting Heterogeneity in Operational Neural Networks by Synaptic Plasticity

The recently proposed network model, Operational Neural Networks (ONNs),...
06/01/2015

Learning with hidden variables

Learning and inferring features that generate sensory input is a task co...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.