Fast Weight Long Short-Term Memory

04/18/2018
by   T. Anderson Keller, et al.
0

Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs). As recent studies introduced fast weights only to regular RNNs, it is unknown whether fast weight memory is beneficial to gated RNNs. In this work, we report a significant synergy between long short-term memory (LSTM) networks and fast weight associative memories. We show that this combination, in learning associative retrieval tasks, results in much faster training and lower test error, a performance boost most prominent at high memory task difficulties.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/14/2017

Simplified Long Short-term Memory Recurrent Neural Networks: part III

This is part III of three-part work. In parts I and II, we have presente...
research
04/23/2016

Memory and Information Processing in Recurrent Neural Networks

Recurrent neural networks (RNN) are simple dynamical systems whose compu...
research
02/09/2016

Associative Long Short-Term Memory

We investigate a new method to augment recurrent neural networks with ex...
research
04/11/2021

Memory Capacity of Neural Turing Machines with Matrix Representation

It is well known that recurrent neural networks (RNNs) faced limitations...
research
08/09/2017

Tikhonov Regularization for Long Short-Term Memory Networks

It is a well-known fact that adding noise to the input data often improv...
research
11/17/2022

Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks

Short-term memory in standard, general-purpose, sequence-processing recu...
research
09/11/2018

Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networks

In this paper, we introduce a novel method to interpret recurrent neural...

Please sign up or login with your details

Forgot password? Click here to reset