DeepAI
Log In Sign Up

Model Based Planning with Energy Based Models

09/15/2019
by   Yilun Du, et al.
5

Model-based planning holds great promise for improving both sample efficiency and generalization in reinforcement learning (RL). We show that energy-based models (EBMs) are a promising class of models to use for model-based planning. EBMs naturally support inference of intermediate states given start and goal state distributions. We provide an online algorithm to train EBMs while interacting with the environment, and show that EBMs allow for significantly better online learning than corresponding feed-forward networks. We further show that EBMs support maximum entropy state inference and are able to generate diverse state space plans. We show that inference purely in state space - without planning actions - allows for better generalization to previously unseen obstacles in the environment and prevents the planner from exploiting the dynamics model by applying uncharacteristic action sequences. Finally, we show that online EBM training naturally leads to intentionally planned state exploration which performs significantly better than random exploration.

READ FULL TEXT

page 5

page 6

page 7

page 8

10/21/2022

Planning with Uncertainty: Deep Exploration in Model-Based Reinforcement Learning

Deep model-based Reinforcement Learning (RL) has shown super-human perfo...
02/08/2018

Learning and Querying Fast Generative Models for Reinforcement Learning

A key challenge in model-based reinforcement learning (RL) is to synthes...
10/12/2019

Regularizing Model-Based Planning with Energy-Based Models

Model-based reinforcement learning could enable sample-efficient learnin...
11/09/2021

Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning

Identifying uncertainty and taking mitigating actions is crucial for saf...
09/29/2021

Learning Dynamics Models for Model Predictive Agents

Model-Based Reinforcement Learning involves learning a dynamics model fr...
11/28/2019

Hierarchical model-based policy optimization: from actions to action sequences and back

We develop a normative framework for hierarchical model-based policy opt...

Code Repositories