Learning Dynamics Models for Model Predictive Agents

by   Michael Lutter, et al.

Model-Based Reinforcement Learning involves learning a dynamics model from data, and then using this model to optimise behaviour, most often with an online planner. Much of the recent research along these lines presents a particular set of design choices, involving problem definition, model learning and planning. Given the multiple contributions, it is difficult to evaluate the effects of each. This paper sets out to disambiguate the role of different design choices for learning dynamics models, by comparing their performance to planning with a ground-truth model – the simulator. First, we collect a rich dataset from the training sequence of a model-free agent on 5 domains of the DeepMind Control Suite. Second, we train feed-forward dynamics models in a supervised fashion, and evaluate planner performance while varying and analysing different model design choices, including ensembling, stochasticity, multi-step training and timestep size. Besides the quantitative analysis, we describe a set of qualitative findings, rules of thumb, and future research directions for planning with learned dynamics models. Videos of the results are available at https://sites.google.com/view/learning-better-models.


Evaluating model-based planning and planner amortization for continuous control

There is a widespread intuition that model-based control methods should ...

Adaptive Online Planning for Continual Lifelong Learning

We study learning control in an online lifelong learning scenario, where...

SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning

Model-based reinforcement learning (RL) methods can be broadly categoriz...

Planning for Novelty: Width-Based Algorithms for Common Problems in Control, Planning and Reinforcement Learning

Width-based algorithms search for solutions through a general definition...

Model Based Planning with Energy Based Models

Model-based planning holds great promise for improving both sample effic...

GEM: Group Enhanced Model for Learning Dynamical Control Systems

Learning the dynamics of a physical system wherein an autonomous agent o...

RotorPy: A Python-based Multirotor Simulator with Aerodynamics for Education and Research

Simulators play a critical role in aerial robotics both in and out of th...

Please sign up or login with your details

Forgot password? Click here to reset