Towards mental time travel: a hierarchical memory for reinforcement learning agents

05/28/2021
by   Andrew Kyle Lampinen, et al.
0

Reinforcement learning agents often forget details of the past, especially after delays or distractor tasks. Agents with common memory architectures struggle to recall and integrate across multiple timesteps of a past event, or even to recall the details of a single timestep that is followed by distractor tasks. To address these limitations, we propose a Hierarchical Transformer Memory (HTM), which helps agents to remember the past in detail. HTM stores memories by dividing the past into chunks, and recalls by first performing high-level attention over coarse summaries of the chunks, and then performing detailed attention within only the most relevant chunks. An agent with HTM can therefore "mentally time-travel" – remember past events in detail without attending to all intervening events. We show that agents with HTM substantially outperform agents with other memory architectures at tasks requiring long-term recall, retention, or reasoning over memory. These include recalling where an object is hidden in a 3D environment, rapidly learning to navigate efficiently in a new neighborhood, and rapidly learning and retaining new object names. Agents with HTM can extrapolate to task sequences an order of magnitude longer than they were trained on, and can even generalize zero-shot from a meta-learning setting to maintaining knowledge across episodes. HTM improves agent sample efficiency, generalization, and generality (by solving tasks that previously required specialized architectures). Our work is a step towards agents that can learn, interact, and adapt in complex and temporally-extended environments.

READ FULL TEXT

page 4

page 6

research
05/24/2018

Been There, Done That: Meta-Learning with Episodic Recall

Meta-learning agents excel at rapidly learning new tasks from open-ended...
research
08/01/2022

e-Genia3 An AgentSpeak extension for empathic agents

In this paper, we present e-Genia3 an extension of AgentSpeak to provide...
research
08/24/2021

Quantum adaptive agents with efficient long-term memories

Central to the success of adaptive systems is their ability to interpret...
research
05/20/2022

BayesPCN: A Continually Learnable Predictive Coding Associative Memory

Associative memory plays an important role in human intelligence and its...
research
06/01/2018

Integrating Episodic Memory into a Reinforcement Learning Agent using Reservoir Sampling

Episodic memory is a psychology term which refers to the ability to reca...
research
06/23/2021

Evolving Hierarchical Memory-Prediction Machines in Multi-Task Reinforcement Learning

A fundamental aspect of behaviour is the ability to encode salient featu...
research
09/03/2020

Grounded Language Learning Fast and Slow

Recent work has shown that large text-based neural language models, trai...

Please sign up or login with your details

Forgot password? Click here to reset