DeepAI AI Chat
Log In Sign Up

Learning Attractor Dynamics for Generative Memory

11/23/2018
by   Yan Wu, et al.
Google
0

A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporating attractor dynamics into modern deep learning systems poses difficulties: attractor basins are characterised by vanishing gradients, which are known to make training neural networks difficult. In this work, we avoid the vanishing gradient problem by training a generative distributed memory without simulating the attractor dynamics. Based on the idea of memory writing as inference, as proposed in the Kanerva Machine, we show that a likelihood-based Lyapunov function emerges from maximising the variational lower-bound of a generative memory. Experiments shows it converges to correct patterns upon iterative retrieval and achieves competitive performance as both a memory model and a generative model.

READ FULL TEXT
10/07/2019

Meta-Learning Deep Energy-Based Memory Models

We study the problem of learning associative memory – a system which is ...
03/14/2011

Memory Retrieval in the B-Matrix Neural Network

This paper is an extension to the memory retrieval procedure of the B-Ma...
02/01/2022

Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold

Content-addressable memory (CAM) networks, so-called because stored item...
03/13/2014

Noise Facilitation in Associative Memories of Exponential Capacity

Recent advances in associative memory design through structured pattern ...
07/03/2021

Memory and attention in deep learning

Intelligence necessitates memory. Without memory, humans fail to perform...
04/05/2018

The Kanerva Machine: A Generative Distributed Memory

We present an end-to-end trained memory system that quickly adapts to ne...
02/17/2022

Entropic Associative Memory for Manuscript Symbols

Manuscript symbols can be stored, recognized and retrieved from an entro...