Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch

06/12/2020
by   Michael Wan, et al.
1

Deep reinforcement learning (RL) algorithms have achieved great success on a wide variety of sequential decision-making tasks. However, many of these algorithms suffer from high sample complexity when learning from scratch using environmental rewards, due to issues such as credit-assignment and high-variance gradients, among others. Transfer learning, in which knowledge gained on a source task is applied to more efficiently learn a different but related target task, is a promising approach to improve the sample complexity in RL. Prior work has considered using pre-trained teacher policies to enhance the learning of the student policy, albeit with the constraint that the teacher and the student MDPs share the state-space or the action-space. In this paper, we propose a new framework for transfer learning where the teacher and the student can have arbitrarily different state- and action-spaces. To handle this mismatch, we produce embeddings which can systematically extract knowledge from the teacher policy and value networks, and blend it into the student networks. To train the embeddings, we use a task-aligned loss and show that the representations could be enriched further by adding a mutual information loss. Using a set of challenging simulated robotic locomotion tasks involving many-legged centipedes, we demonstrate successful transfer learning in situations when the teacher and student have different state- and action-spaces.

READ FULL TEXT

page 6

page 7

page 8

research
06/21/2023

Introspective Action Advising for Interpretable Transfer Learning

Transfer learning can be applied in deep reinforcement learning to accel...
research
10/31/2022

Teacher-student curriculum learning for reinforcement learning

Reinforcement learning (rl) is a popular paradigm for sequential decisio...
research
09/05/2019

Learning Action-Transferable Policy with Action Embedding

Despite achieving great success on performance in various sequential dec...
research
04/14/2022

Methodical Advice Collection and Reuse in Deep Reinforcement Learning

Reinforcement learning (RL) has shown great success in solving many chal...
research
09/20/2022

Soft Action Priors: Towards Robust Policy Transfer

Despite success in many challenging problems, reinforcement learning (RL...
research
02/08/2019

Size Independent Neural Transfer for RDDL Planning

Neural planners for RDDL MDPs produce deep reactive policies in an offli...
research
04/07/2023

Bridging Action Space Mismatch in Learning from Demonstrations

Learning from demonstrations (LfD) methods guide learning agents to a de...

Please sign up or login with your details

Forgot password? Click here to reset