DeepAI AI Chat
Log In Sign Up

The Information Geometry of Unsupervised Reinforcement Learning

by   Benjamin Eysenbach, et al.
Carnegie Mellon University

How can a reinforcement learning (RL) agent prepare to solve downstream tasks if those tasks are not known a priori? One approach is unsupervised skill discovery, a class of algorithms that learn a set of policies without access to a reward function. Such algorithms bear a close resemblance to representation learning algorithms (e.g., contrastive learning) in supervised learning, in that both are pretraining algorithms that maximize some approximation to a mutual information objective. While prior work has shown that the set of skills learned by such methods can accelerate downstream RL tasks, prior work offers little analysis into whether these skill learning algorithms are optimal, or even what notion of optimality would be appropriate to apply to them. In this work, we show that unsupervised skill discovery algorithms based on mutual information maximization do not learn skills that are optimal for every possible reward function. However, we show that the distribution over skills provides an optimal initialization minimizing regret against adversarially-chosen reward functions, assuming a certain type of adaptation procedure. Our analysis also provides a geometric perspective on these skill learning methods.


page 1

page 2

page 3

page 4


CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery

We introduce Contrastive Intrinsic Control (CIC), an algorithm for unsup...

Skill-Based Reinforcement Learning with Intrinsic Reward Matching

While unsupervised skill discovery has shown promise in autonomously acq...

Wasserstein Unsupervised Reinforcement Learning

Unsupervised reinforcement learning aims to train agents to learn a hand...

Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery

Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for ...

Learning Options via Compression

Identifying statistical regularities in solutions to some tasks in multi...

APS: Active Pretraining with Successor Features

We introduce a new unsupervised pretraining objective for reinforcement ...

ASPiRe:Adaptive Skill Priors for Reinforcement Learning

We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that l...