Transfer RL via the Undo Maps Formalism

11/26/2022
by   Abhi Gupta, et al.
0

Transferring knowledge across domains is one of the most fundamental problems in machine learning, but doing so effectively in the context of reinforcement learning remains largely an open problem. Current methods make strong assumptions on the specifics of the task, often lack principled objectives, and – crucially – modify individual policies, which might be sub-optimal when the domains differ due to a drift in the state space, i.e., it is intrinsic to the environment and therefore affects every agent interacting with it. To address these drawbacks, we propose TvD: transfer via distribution matching, a framework to transfer knowledge across interactive domains. We approach the problem from a data-centric perspective, characterizing the discrepancy in environments by means of (potentially complex) transformation between their state spaces, and thus posing the problem of transfer as learning to undo this transformation. To accomplish this, we introduce a novel optimization objective based on an optimal transport distance between two distributions over trajectories – those generated by an already-learned policy in the source domain and a learnable pushforward policy in the target domain. We show this objective leads to a policy update scheme reminiscent of imitation learning, and derive an efficient algorithm to implement it. Our experiments in simple gridworlds show that this method yields successful transfer learning across a wide range of environment transformations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/22/2018

Cross-Domain Transfer in Reinforcement Learning using Target Apprentice

In this paper, we present a new approach to Transfer Learning (TL) in Re...
research
08/16/2016

A novel transfer learning method based on common space mapping and weighted domain matching

In this paper, we propose a novel learning framework for the problem of ...
research
06/21/2023

Introspective Action Advising for Interpretable Transfer Learning

Transfer learning can be applied in deep reinforcement learning to accel...
research
10/27/2021

Transfer learning with causal counterfactual reasoning in Decision Transformers

The ability to adapt to changes in environmental contingencies is an imp...
research
05/07/2023

A Framework for Characterizing Novel Environment Transformations in General Environments

To be robust to surprising developments, an intelligent agent must be ab...
research
06/04/2020

Visual Transfer for Reinforcement Learning via Wasserstein Domain Confusion

We introduce Wasserstein Adversarial Proximal Policy Optimization (WAPPO...
research
09/06/2022

Cross apprenticeship learning framework: Properties and solution approaches

Apprenticeship learning is a framework in which an agent learns a policy...

Please sign up or login with your details

Forgot password? Click here to reset