Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

01/20/2022
by   Sasha Salter, et al.
19

The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world. Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics. While hierarchical and KL-regularized RL individually hold promise here, arguably a hybrid approach could combine their respective benefits. Key to these fields is the use of information asymmetry to bias which skills are learnt. While asymmetric choice has a large influence on transferability, prior works have explored a narrow range of asymmetries, primarily motivated by intuition. In this paper, we theoretically and empirically show the crucial trade-off, controlled by information asymmetry, between the expressivity and transferability of skills across sequential tasks. Given this insight, we provide a principled approach towards choosing asymmetry and apply our approach to a complex, robotic block stacking domain, unsolvable by baselines, demonstrating the effectiveness of hierarchical KL-regularized RL, coupled with correct asymmetric choice, for sample-efficient transfer learning.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 18

page 19

10/22/2020

Accelerating Reinforcement Learning with Learned Skill Priors

Intelligent agents rely heavily on prior experience when learning a new ...
03/18/2019

Exploiting Hierarchy for Learning and Transfer in KL-regularized RL

As reinforcement learning agents are tasked with solving more challengin...
05/10/2021

Adaptive Policy Transfer in Reinforcement Learning

Efficient and robust policy transfer remains a key challenge for reinfor...
06/26/2019

Regularized Hierarchical Policies for Compositional Transfer in Robotics

The successful application of flexible, general learning algorithms -- s...
11/27/2020

Skill Transfer via Partially Amortized Hierarchical Planning

To quickly solve new tasks in complex environments, intelligent agents n...
08/14/2019

Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity

Transfer learning methods for reinforcement learning (RL) domains facili...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.