Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision

by   Julien Scholz, et al.

Using a model of the environment, reinforcement learning agents can plan their future moves and achieve superhuman performance in board games like Chess, Shogi, and Go, while remaining relatively sample-efficient. As demonstrated by the MuZero Algorithm, the environment model can even be learned dynamically, generalizing the agent to many more tasks while at the same time achieving state-of-the-art performance. Notably, MuZero uses internal state representations derived from real environment states for its predictions. In this paper, we bind the model's predicted internal state representation to the environment state via two additional terms: a reconstruction model loss and a simpler consistency loss, both of which work independently and unsupervised, acting as constraints to stabilize the learning process. Our experiments show that this new integration of reconstruction model loss and simpler consistency loss provide a significant performance increase in OpenAI Gym environments. Our modifications also enable self-supervised pretraining for MuZero, so the algorithm can learn about environment dynamics before a goal is made available.


page 1

page 2

page 3

page 4


Learning State Representations via Retracing in Reinforcement Learning

We propose learning via retracing, a novel self-supervised approach for ...

Visualizing MuZero Models

MuZero, a model-based reinforcement learning algorithm that uses a value...

ToyBox: Better Atari Environments for Testing Reinforcement Learning Agents

It is a widely accepted principle that software without tests has bugs. ...

Data-Efficient Reinforcement Learning with Momentum Predictive Representations

While deep reinforcement learning excels at solving tasks where large am...

Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments

Self-supervised methods, wherein an agent learns representations solely ...

Continuous Homeostatic Reinforcement Learning for Self-Regulated Autonomous Agents

Homeostasis is a prevalent process by which living beings maintain their...

Recurrent World Models Facilitate Policy Evolution

A generative recurrent neural network is quickly trained in an unsupervi...