Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning

10/26/2020
by   Younggyo Seo, et al.
0

Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance. However, learning a generalizable dynamics model robust to changes in dynamics remains a challenge since the target transition dynamics follow a multi-modal distribution. In this paper, we present a new model-based RL algorithm, coined trajectory-wise multiple choice learning, that learns a multi-headed dynamics model for dynamics generalization. The main idea is updating the most accurate prediction head to specialize each head in certain environments with similar dynamics, i.e., clustering environments. Moreover, we incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector, enabling the model to perform online adaptation to unseen environments. Finally, to utilize the specialized prediction heads more effectively, we propose an adaptive planning method, which selects the most accurate prediction head over a recent experience. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods. Source code and videos are available at https://sites.google.com/view/trajectory-mcl.

READ FULL TEXT
06/09/2022

A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning

The generalization of model-based reinforcement learning (MBRL) methods ...
05/14/2020

Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning

Model-based reinforcement learning (RL) enjoys several benefits, such as...
05/13/2019

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning

While model-based deep reinforcement learning (RL) holds great promise f...
08/28/2018

SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning

Model-based reinforcement learning (RL) methods can be broadly categoriz...
06/28/2022

Masked World Models for Visual Control

Visual model-based reinforcement learning (RL) has the potential to enab...
04/07/2021

GEM: Group Enhanced Model for Learning Dynamical Control Systems

Learning the dynamics of a physical system wherein an autonomous agent o...
04/23/2020

Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads

Transporting suspended payloads is challenging for autonomous aerial veh...