Asynchronous Methods for Model-Based Reinforcement Learning

by   Yunzhi Zhang, et al.

Significant progress has been made in the area of model-based reinforcement learning. State-of-the-art algorithms are now able to match the asymptotic performance of model-free methods while being significantly more data efficient. However, this success has come at a price: state-of-the-art model-based methods require significant computation interleaved with data collection, resulting in run times that take days, even if the amount of agent interaction might be just hours or even minutes. When considering the goal of learning in real-time on real robots, this means these state-of-the-art model-based algorithms still remain impractical. In this work, we propose an asynchronous framework for model-based reinforcement learning methods that brings down the run time of these algorithms to be just the data collection time. We evaluate our asynchronous framework on a range of standard MuJoCo benchmarks. We also evaluate our asynchronous framework on three real-world robotic manipulation tasks. We show how asynchronous learning not only speeds up learning w.r.t wall-clock time through parallelization, but also further reduces the sample complexity of model-based approaches by means of improving the exploration and by means of effectively avoiding the policy overfitting to the deficiencies of learned dynamics models.


Model-Based Reinforcement Learning via Meta-Policy Optimization

Model-based reinforcement learning approaches carry the promise of being...

Residual Model Learning for Microrobot Control

A majority of microrobots are constructed using compliant materials that...

Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots

An oft-ignored challenge of real-world reinforcement learning is that th...

Robo-PlaNet: Learning to Poke in a Day

Recently, the Deep Planning Network (PlaNet) approach was introduced as ...

Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?

We contribute to micro-data model-based reinforcement learning (MBRL) by...

Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

Using a high Update-To-Data (UTD) ratio, model-based methods have recent...

Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors

Model usage is the central challenge of model-based reinforcement learni...