Model Imitation for Model-Based Reinforcement Learning

09/25/2019
by   Yueh-Hua Wu, et al.
10

Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollouts in the learned model, especially those of long horizon, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRL. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on the claim, we propose to learn the synthesized model by matching the distributions of multi-step rollouts sampled from the synthesized model and the real ones via WGAN. We theoretically show that matching the two can minimize the difference of cumulative rewards between the real transition and the learned one. Our experiments also show that the proposed model imitation method outperforms the state-of-the-art in terms of sample complexity and average return.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/12/2022

Model-based Reinforcement Learning with Multi-step Plan Value Estimation

A promising way to improve the sample efficiency of reinforcement learni...
research
12/24/2019

Learning to Combat Compounding-Error in Model-Based Reinforcement Learning

Despite its potential to improve sample complexity versus model-free app...
research
09/23/2019

PAC Reinforcement Learning without Real-World Feedback

This work studies reinforcement learning in the Sim-to-Real setting, in ...
research
06/11/2020

PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processes

We consider the problem of batch multi-task reinforcement learning with ...
research
11/02/2022

Model-based Reinforcement Learning with a Hamiltonian Canonical ODE Network

Model-based reinforcement learning usually suffers from a high sample co...
research
10/31/2018

Towards a Simple Approach to Multi-step Model-based Reinforcement Learning

When environmental interaction is expensive, model-based reinforcement l...
research
09/18/2020

A Contraction Approach to Model-based Reinforcement Learning

Model-based Reinforcement Learning has shown considerable experimental s...

Please sign up or login with your details

Forgot password? Click here to reset