On Effective Scheduling of Model-based Reinforcement Learning

by   Hang Lai, et al.
Shanghai Jiao Tong University

Model-based reinforcement learning has attracted wide attention due to its superior sample efficiency. Despite its impressive success so far, it is still unclear how to appropriately schedule the important hyperparameters to achieve adequate performance, such as the real data ratio for policy optimization in Dyna-style model-based algorithms. In this paper, we first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance. Inspired by the analysis, we propose a framework named AutoMBPO to automatically schedule the real data ratio as well as other hyperparameters in training model-based policy optimization (MBPO) algorithm, a representative running case of model-based methods. On several continuous control tasks, the MBPO instance trained with hyperparameters scheduled by AutoMBPO can significantly surpass the original one, and the real data ratio schedule found by AutoMBPO shows consistency with our theoretical analysis.


page 1

page 2

page 3

page 4


When to Trust Your Model: Model-Based Policy Optimization

Designing effective model-based reinforcement learning algorithms is dif...

Model Embedding Model-Based Reinforcement Learning

Model-based reinforcement learning (MBRL) has shown its advantages in sa...

Model-based Policy Optimization with Unsupervised Model Adaptation

Model-based reinforcement learning methods learn a dynamics model with r...

On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

Model-based Reinforcement Learning (MBRL) is a promising framework for l...

A Data-Efficient Model-Based Learning Framework for the Closed-Loop Control of Continuum Robots

Traditional dynamic models of continuum robots are in general computatio...

Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?

We contribute to micro-data model-based reinforcement learning (MBRL) by...

Please sign up or login with your details

Forgot password? Click here to reset