Predictive Information Accelerates Learning in RL

07/24/2020
by   Kuang-Huei Lee, et al.
21

The Predictive Information is the mutual information between the past and the future, I(X_past; X_future). We hypothesize that capturing the predictive information is useful in RL, since the ability to model what will happen next is necessary for success on many tasks. To test our hypothesis, we train Soft Actor-Critic (SAC) agents from pixels with an auxiliary task that learns a compressed representation of the predictive information of the RL environment dynamics using a contrastive version of the Conditional Entropy Bottleneck (CEB) objective. We refer to these as Predictive Information SAC (PI-SAC) agents. We show that PI-SAC agents can substantially improve sample efficiency over challenging baselines on tasks from the DM Control suite of continuous control environments. We evaluate PI-SAC agents by comparing against uncompressed PI-SAC agents, other compressed and uncompressed agents, and SAC agents directly trained from pixels.

READ FULL TEXT

page 5

page 6

page 13

page 14

page 16

page 17

page 19

10/09/2020

Deep RL With Information Constrained Policies: Generalization in Continuous Control

Biological agents learn and act intelligently in spite of a highly limit...
09/21/2022

Model-Free Reinforcement Learning for Asset Allocation

Asset allocation (or portfolio management) is the task of determining ho...
06/12/2020

Deep Reinforcement and InfoMax Learning

Our work is based on the hypothesis that a model-free agent whose repres...
09/10/2019

Discovery of Useful Questions as Auxiliary Tasks

Arguably, intelligent agents ought to be able to discover their own ques...
07/20/2021

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

We present DrQ-v2, a model-free reinforcement learning (RL) algorithm fo...
11/12/2019

Learning Representations in Reinforcement Learning:An Information Bottleneck Approach

The information bottleneck principle is an elegant and useful approach t...
06/30/2022

Denoised MDPs: Learning World Models Better Than the World Itself

The ability to separate signal from noise, and reason with clean abstrac...

Code Repositories

pisac

Tensorflow source code for the PI-SAC agent from "Predictive Information Accelerates Learning in RL" (NeurIPS 2020)


view repo