DeepAI AI Chat
Log In Sign Up

Learning Attractor Dynamics for Generative Memory

by   Yan Wu, et al.

A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporating attractor dynamics into modern deep learning systems poses difficulties: attractor basins are characterised by vanishing gradients, which are known to make training neural networks difficult. In this work, we avoid the vanishing gradient problem by training a generative distributed memory without simulating the attractor dynamics. Based on the idea of memory writing as inference, as proposed in the Kanerva Machine, we show that a likelihood-based Lyapunov function emerges from maximising the variational lower-bound of a generative memory. Experiments shows it converges to correct patterns upon iterative retrieval and achieves competitive performance as both a memory model and a generative model.


Meta-Learning Deep Energy-Based Memory Models

We study the problem of learning associative memory – a system which is ...

Memory Retrieval in the B-Matrix Neural Network

This paper is an extension to the memory retrieval procedure of the B-Ma...

Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold

Content-addressable memory (CAM) networks, so-called because stored item...

Noise Facilitation in Associative Memories of Exponential Capacity

Recent advances in associative memory design through structured pattern ...

Memory and attention in deep learning

Intelligence necessitates memory. Without memory, humans fail to perform...

The Kanerva Machine: A Generative Distributed Memory

We present an end-to-end trained memory system that quickly adapts to ne...

Entropic Associative Memory for Manuscript Symbols

Manuscript symbols can be stored, recognized and retrieved from an entro...