Retrieval-Augmented Reinforcement Learning

02/17/2022
by   Anirudh Goyal, et al.
0

Most deep reinforcement learning (RL) algorithms distill experience into parametric behavior policies or value functions via gradient updates. While effective, this approach has several disadvantages: (1) it is computationally expensive, (2) it can take many updates to integrate experiences into the parametric model, (3) experiences that are not fully integrated do not appropriately influence the agent's behavior, and (4) behavior is limited by the capacity of the model. In this paper we explore an alternative paradigm in which we train a network to map a dataset of past experiences to optimal behavior. Specifically, we augment an RL agent with a retrieval process (parameterized as a neural network) that has direct access to a dataset of experiences. This dataset can come from the agent's past experiences, expert demonstrations, or any other relevant source. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context, to help the agent achieve its goal faster and more efficiently. We integrate our method into two different RL agents: an offline DQN agent and an online R2D2 agent. In offline multi-task problems, we show that the retrieval-augmented DQN agent avoids task interference and learns faster than the baseline DQN agent. On Atari, we show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores. We run extensive ablations to measure the contributions of the components of our proposed method.

READ FULL TEXT
research
06/10/2022

Large-Scale Retrieval for Reinforcement Learning

Effective decision making involves flexibly relating past experiences an...
research
06/05/2020

Balancing Reinforcement Learning Training Experiences in Interactive Information Retrieval

Interactive Information Retrieval (IIR) and Reinforcement Learning (RL) ...
research
01/26/2023

Which Experiences Are Influential for Your Agent? Policy Iteration with Turn-over Dropout

In reinforcement learning (RL) with experience replay, experiences store...
research
06/09/2023

Large Language Model Is Semi-Parametric Reinforcement Learning Agent

Inspired by the insights in cognitive science with respect to human memo...
research
01/13/2021

Continuous Deep Q-Learning with Simulator for Stabilization of Uncertain Discrete-Time Systems

Applications of reinforcement learning (RL) to stabilization problems of...
research
08/24/2020

Improved Memories Learning

We propose Improved Memories Learning (IMeL), a novel algorithm that tur...
research
09/18/2023

Contrastive Initial State Buffer for Reinforcement Learning

In Reinforcement Learning, the trade-off between exploration and exploit...

Please sign up or login with your details

Forgot password? Click here to reset