Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

by   Xiyao Wang, et al.

Model-based reinforcement learning is a framework in which an agent learns an environment model, makes planning and decision-making in this model, and finally interacts with the real environment. Model-based reinforcement learning has high sample efficiency compared with model-free reinforcement learning, and shows great potential in the real-world application. However, model-based reinforcement learning has been plagued by dynamics bottleneck. Dynamics bottleneck is the phenomenon that when the timestep to interact with the environment increases, the reward of the agent falls into the local optimum instead of increasing. In this paper, we analyze and explain how the coupling relationship between model and policy causes the dynamics bottleneck and shows improving the exploration ability of the agent can alleviate this issue. We then propose a new planning algorithm called Maximum Entropy Cross-Entropy Method (MECEM). MECEM can improve the agent's exploration ability by maximizing the distribution of action entropy in the planning process. We conduct experiments on fourteen well-recognized benchmark environments such as HalfCheetah, Ant and Swimmer. The results verify that our approach obtains the state-of-the-art performance on eleven benchmark environments and can effectively alleviate dynamics bottleneck on HalfCheetah, Ant and Walker2D.



There are no comments yet.


page 1

page 2

page 3

page 4


A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning

We present an end-to-end, model-based deep reinforcement learning agent ...

Maximum Entropy Model-based Reinforcement Learning

Recent advances in reinforcement learning have demonstrated its ability ...

Can Agents Learn by Analogy? An Inferable Model for PAC Reinforcement Learning

Model-based reinforcement learning algorithms make decisions by building...

Model-Based Active Exploration

Efficient exploration is an unsolved problem in Reinforcement Learning. ...

Multitask Adaptation by Retrospective Exploration with Learned World Models

Model-based reinforcement learning (MBRL) allows solving complex tasks i...

Model Based Planning with Energy Based Models

Model-based planning holds great promise for improving both sample effic...

Self-Correcting Models for Model-Based Reinforcement Learning

When an agent cannot represent a perfectly accurate model of its environ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.