DeepAI AI Chat
Log In Sign Up

Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization

11/11/2022
by   Burcu Küçükoğlu, et al.
Radboud Universiteit
0

Advances in reinforcement learning (RL) often rely on massive compute resources and remain notoriously sample inefficient. In contrast, the human brain is able to efficiently learn effective control strategies using limited resources. This raises the question whether insights from neuroscience can be used to improve current RL methods. Predictive processing is a popular theoretical framework which maintains that the human brain is actively seeking to minimize surprise. We show that recurrent neural networks which predict their own sensory states can be leveraged to minimise surprise, yielding substantial gains in cumulative reward. Specifically, we present the Predictive Processing Proximal Policy Optimization (P4O) agent; an actor-critic reinforcement learning agent that applies predictive processing to a recurrent variant of the PPO algorithm by integrating a world model in its hidden state. P4O significantly outperforms a baseline recurrent variant of the PPO algorithm on multiple Atari games using a single GPU. It also outperforms other state-of-the-art agents given the same wall-clock time and exceeds human gamer performance on multiple games including Seaquest, which is a particularly challenging environment in the Atari domain. Altogether, our work underscores how insights from the field of neuroscience may support the development of more capable and efficient artificial agents.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/09/2019

Transformer Based Reinforcement Learning For Games

Recent times have witnessed sharp improvements in reinforcement learning...
12/14/2022

Efficient Exploration in Resource-Restricted Reinforcement Learning

In many real-world applications of reinforcement learning (RL), performi...
09/21/2022

Model-Free Reinforcement Learning for Asset Allocation

Asset allocation (or portfolio management) is the task of determining ho...
11/27/2020

Adaptable Automation with Modular Deep Reinforcement Learning and Policy Transfer

Recent advances in deep Reinforcement Learning (RL) have created unprece...
12/25/2022

Novel Reinforcement Learning Algorithm for Suppressing Synchronization in Closed Loop Deep Brain Stimulators

Parkinson's disease is marked by altered and increased firing characteri...
02/28/2020

Self-Tuning Deep Reinforcement Learning

Reinforcement learning (RL) algorithms often require expensive manual or...
02/22/2020

Reinforcement Learning Framework for Deep Brain Stimulation Study

Malfunctioning neurons in the brain sometimes operate synchronously, rep...