A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning

06/09/2022
by   Jixian Guo, et al.
0

The generalization of model-based reinforcement learning (MBRL) methods to environments with unseen transition dynamics is an important yet challenging problem. Existing methods try to extract environment-specified information Z from past transition segments to make the dynamics prediction model generalizable to different dynamics. However, because environments are not labelled, the extracted information inevitably contains redundant information unrelated to the dynamics in transition segments and thus fails to maintain a crucial property of Z: Z should be similar in the same environment and dissimilar in different ones. As a result, the learned dynamics prediction function will deviate from the true one, which undermines the generalization ability. To tackle this problem, we introduce an interventional prediction module to estimate the probability of two estimated ẑ_i, ẑ_j belonging to the same environment. Furthermore, by utilizing the Z's invariance within a single environment, a relational head is proposed to enforce the similarity between Ẑ from the same environment. As a result, the redundant information will be reduced in Ẑ. We empirically show that Ẑ estimated by our method enjoy less redundant information than previous methods, and such Ẑ can significantly reduce dynamics prediction errors and improve the performance of model-based RL methods on zero-shot new environments with unseen dynamics. The codes of this method are available at <https://github.com/CR-Gjx/RIA>.

READ FULL TEXT

page 18

page 21

page 22

page 23

research
10/26/2020

Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning

Model-based reinforcement learning (RL) has shown great potential in var...
research
05/14/2020

Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning

Model-based reinforcement learning (RL) enjoys several benefits, such as...
research
12/06/2021

ED2: An Environment Dynamics Decomposition Framework for World Model Construction

Model-based reinforcement learning methods achieve significant sample ef...
research
02/08/2023

Predictable MDP Abstraction for Unsupervised Model-Based RL

A key component of model-based reinforcement learning (RL) is a dynamics...
research
05/01/2017

Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning

In this paper we study how to learn stochastic, multimodal transition dy...
research
11/23/2022

Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning

The latent world model provides a promising way to learn policies in a c...
research
04/16/2019

Object-Oriented Dynamics Learning through Multi-Level Abstraction

Object-based approaches for learning action-conditioned dynamics has dem...

Please sign up or login with your details

Forgot password? Click here to reset