Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning

06/02/2021
by   Jongwook Choi, et al.
9

Learning to reach goal states and learning diverse skills through mutual information (MI) maximization have been proposed as principled frameworks for self-supervised reinforcement learning, allowing agents to acquire broadly applicable multitask policies with minimal reward engineering. Starting from a simple observation that the standard goal-conditioned RL (GCRL) is encapsulated by the optimization objective of variational empowerment, we discuss how GCRL and MI-based RL can be generalized into a single family of methods, which we name variational GCRL (VGCRL), interpreting variational MI maximization, or variational empowerment, as representation learning methods that acquire functionally-aware state representations for goal reaching. This novel perspective allows us to: (1) derive simple but unexplored variants of GCRL to study how adding small representation capacity can already expand its capabilities; (2) investigate how discriminator function capacity and smoothness determine the quality of discovered skills, or latent goals, through modifying latent dimensionality and applying spectral normalization; (3) adapt techniques such as hindsight experience replay (HER) from GCRL to MI-based RL; and lastly, (4) propose a novel evaluation metric, named latent goal reaching (LGR), for comparing empowerment algorithms with different choices of latent dimensionality and discriminator parameterization. Through principled mathematical derivations and careful experimental studies, our work lays a novel foundation from which to evaluate, analyze, and develop representation learning techniques in goal-based RL.

READ FULL TEXT

page 5

page 17

06/15/2022

Contrastive Learning as Goal-Conditioned Reinforcement Learning

In reinforcement learning (RL), it is easier to solve a task if given a ...
06/14/2022

Stein Variational Goal Generation For Reinforcement Learning in Hard Exploration Problems

Multi-goal Reinforcement Learning has recently attracted a large amount ...
03/08/2019

Skew-Fit: State-Covering Self-Supervised Reinforcement Learning

In standard reinforcement learning, each new skill requires a manually-d...
10/06/2021

The Information Geometry of Unsupervised Reinforcement Learning

How can a reinforcement learning (RL) agent prepare to solve downstream ...
11/17/2020

C-Learning: Learning to Achieve Goals via Recursive Classification

We study the problem of predicting and controlling the future state dist...
11/16/2020

Towards Learning Controllable Representations of Physical Systems

Learned representations of dynamical systems reduce dimensionality, pote...
10/10/2021

Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximization

The goal of continuous control is to synthesize desired behaviors. In re...