DeepAI AI Chat
Log In Sign Up

Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization

by   Marco Pleines, et al.

At first sight it may seem straightforward to use recurrent layers in Deep Reinforcement Learning algorithms to enable agents to make use of memory in the setting of partially observable environments. Starting from widely used Proximal Policy Optimization (PPO), we highlight vital details that one must get right when adding recurrence to achieve a correct and efficient implementation, namely: properly shaping the neural net's forward pass, arranging the training data, correspondingly selecting hidden states for sequence beginnings and masking paddings for loss computation. We further explore the limitations of recurrent PPO by benchmarking the contributed novel environments Mortar Mayhem and Searing Spotlights that challenge the agent's memory beyond solely capacity and distraction tasks. Remarkably, we can demonstrate a transition to strong generalization in Mortar Mayhem when scaling the number of training seeds, while the agent does not succeed on Searing Spotlights, which seems to be a tough challenge for memory-based agents.


page 5

page 11

page 12

page 14

page 15

page 16


Proximal Policy Optimization with Mixed Distributed Training

Instability and slowness are two main problems in deep reinforcement lea...

EgoMap: Projective mapping and structured egocentric memory for Deep RL

Tasks involving localization, memorization and planning in partially obs...

A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings

In this work we explore an auxiliary loss useful for reinforcement learn...

Meta reinforcement learning as task inference

Humans achieve efficient learning by relying on prior knowledge about th...

Coordinated Proximal Policy Optimization

We present Coordinated Proximal Policy Optimization (CoPPO), an algorith...

Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization

Advances in reinforcement learning (RL) often rely on massive compute re...

Learning classifier systems with memory condition to solve non-Markov problems

In the family of Learning Classifier Systems, the classifier system XCS ...

Code Repositories


Baseline implementation of recurrent PPO using truncated BPTT

view repo