Influence-aware Memory for Deep Reinforcement Learning

11/18/2019
by   Miguel Suau de Castro, et al.
0

Making the right decisions when some of the state variables are hidden, involves reasoning about all the possible states of the environment. An agent receiving only partial observations needs to infer the true values of these hidden variables based on the history of experiences. Recent deep reinforcement learning methods use recurrent models to keep track of past information. However, these models are sometimes expensive to train and have convergence difficulties, especially when dealing with high dimensional input spaces. Taking inspiration from influence-based abstraction, we show that effective policies can be learned in the presence of uncertainty by only memorizing a small subset of input variables. We also incorporate a mechanism in our network that learns to automatically choose the important pieces of information that need to be remembered. The results indicate that, by forcing the agent's internal memory to focus on the selected regions while treating the rest of the observable variables as Markovian, we can outperform ordinary recurrent architectures in situations where the amount of information that the agent needs to retain represents a small fraction of the entire observation input. The method also reduces training time and obtains better scores than methods that stack multiple observations to remove partial observability in domains where long-term memory is required.

READ FULL TEXT

page 5

page 7

research
10/25/2021

Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning

Reinforcement Learning faces an important challenge in partial observabl...
research
06/02/2022

Deep Transformer Q-Networks for Partially Observable Reinforcement Learning

Real-world reinforcement learning tasks often involve some form of parti...
research
09/10/2015

Recurrent Reinforcement Learning: A Hybrid Approach

Successful applications of reinforcement learning in real-world problems...
research
06/26/2020

Perception-Prediction-Reaction Agents for Deep Reinforcement Learning

We introduce a new recurrent agent architecture and associated auxiliary...
research
07/12/2023

PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks

Deep reinforcement learning (RL) has shown immense potential for learnin...
research
06/10/2018

Deep Reinforcement Learning for Chinese Zero pronoun Resolution

Deep neural network models for Chinese zero pronoun resolution learn sem...

Please sign up or login with your details

Forgot password? Click here to reset