Log In Sign Up

DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policies

by   Soroush Nasiriany, et al.

Can we use reinforcement learning to learn general-purpose policies that can perform a wide range of different tasks, resulting in flexible and reusable skills? Contextual policies provide this capability in principle, but the representation of the context determines the degree of generalization and expressivity. Categorical contexts preclude generalization to entirely new tasks. Goal-conditioned policies may enable some generalization, but cannot capture all tasks that might be desired. In this paper, we propose goal distributions as a general and broadly applicable task representation suitable for contextual policies. Goal distributions are general in the sense that they can represent any state-based reward function when equipped with an appropriate distribution class, while the particular choice of distribution class allows us to trade off expressivity and learnability. We develop an off-policy algorithm called distribution-conditioned reinforcement learning (DisCo RL) to efficiently learn these policies. We evaluate DisCo RL on a variety of robot manipulation tasks and find that it significantly outperforms prior methods on tasks that require generalization to new goal distributions.


Visual Reinforcement Learning with Imagined Goals

For an autonomous agent to fulfill a wide range of user-specified goals ...

Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning

Building generalizable goal-conditioned agents from rich observations is...

Goal-Conditioned Generators of Deep Policies

Goal-conditioned Reinforcement Learning (RL) aims at learning optimal po...

Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation

A general-purpose intelligent robot must be able to learn autonomously a...

Non-Markovian policies occupancy measures

A central object of study in Reinforcement Learning (RL) is the Markovia...

Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning

We propose a graphical model framework for goal-conditioned RL, with an ...

Self-Paced Contextual Reinforcement Learning

Generalization and adaptation of learned skills to novel situations is a...