Planning with Sequence Models through Iterative Energy Minimization

03/28/2023
by   Hongyi Chen, et al.
0

Recent works have shown that sequence modeling can be effectively used to train reinforcement learning (RL) policies. However, the success of applying existing sequence models to planning, in which we wish to obtain a trajectory of actions to reach some goal, is less straightforward. The typical autoregressive generation procedures of sequence models preclude sequential refinement of earlier steps, which limits the effectiveness of a predicted plan. In this paper, we suggest an approach towards integrating planning with sequence models based on the idea of iterative energy minimization, and illustrate how such a procedure leads to improved RL performance across different tasks. We train a masked language model to capture an implicit energy function over trajectories of actions, and formulate planning as finding a trajectory of actions with minimum energy. We illustrate how this procedure enables improved performance over recent approaches across BabyAI and Atari environments. We further demonstrate unique benefits of our iterative optimization procedure, involving new task generalization, test-time constraints adaptation, and the ability to compose plans together. Project website: https://hychen-naza.github.io/projects/LEAP

READ FULL TEXT

page 5

page 7

page 9

research
06/17/2022

Bootstrapped Transformer for Offline Reinforcement Learning

Offline reinforcement learning (RL) aims at learning policies from previ...
research
04/26/2018

Action Categorization for Computationally Improved Task Learning and Planning

This paper explores the problem of task learning and planning, contribut...
research
05/16/2021

Model-Based Offline Planning with Trajectory Pruning

Offline reinforcement learning (RL) enables learning policies using pre-...
research
03/01/2022

AI Planning Annotation for Sample Efficient Reinforcement Learning

AI planning and Reinforcement Learning (RL) both solve sequential decisi...
research
07/07/2023

Goal-Conditioned Predictive Coding as an Implicit Planner for Offline Reinforcement Learning

Recent work has demonstrated the effectiveness of formulating decision m...
research
07/24/2018

Learning Plannable Representations with Causal InfoGAN

In recent years, deep generative models have been shown to 'imagine' con...
research
05/26/2023

Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Models

There have been wide spread claims in the literature about the emergent ...

Please sign up or login with your details

Forgot password? Click here to reset