Log In Sign Up

Planning with Goal-Conditioned Policies

by   Soroush Nasiriany, et al.

Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors. However, planning requires suitable abstractions for the states and transitions, which typically need to be designed by hand. In contrast, model-free reinforcement learning (RL) can acquire behaviors from low-level inputs directly, but often struggles with temporally extended tasks. Can we utilize reinforcement learning to automatically form the abstractions needed for planning, thus obtaining the best of both approaches? We show that goal-conditioned policies learned with RL can be incorporated into planning, so that a planner can focus on which states to reach, rather than how those states are reached. However, with complex state observations such as images, not all inputs represent valid states. We therefore also propose using a latent variable model to compactly represent the set of valid states for the planner, so that the policies provide an abstraction of actions, and the latent variable model provides an abstraction of states. We compare our method with planning-based and model-free methods and find that our method significantly outperforms prior work when evaluated on image-based robot navigation and manipulation tasks that require non-greedy, multi-staged behavior.


page 7

page 8

page 13

page 14


Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning

Offline Reinforcement learning (RL) has shown potent in many safe-critic...

Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space

General-purpose robots require diverse repertoires of behaviors to compl...

Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models

We present a data-efficient framework for solving sequential decision-ma...

Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agents

In reinforcement learning applications like robotics, agents usually nee...

Language as an Abstraction for Hierarchical Deep Reinforcement Learning

Solving complex, temporally-extended tasks is a long-standing problem in...

Temporally Extended Successor Representations

We present a temporally extended variation of the successor representati...

Goal-Space Planning with Subgoal Models

This paper investigates a new approach to model-based reinforcement lear...

Code Repositories