Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

10/10/2019
by   Siyuan Li, et al.
37

Hierarchical Reinforcement Learning (HRL) is a promising approach to solving long-horizon problems with sparse and delayed rewards. Many existing HRL algorithms either use pre-trained low-level skills that are unadaptable, or require domain-specific information to define low-level rewards. In this paper, we aim to adapt low-level skills to downstream tasks while maintaining the generality of reward design. We propose an HRL framework which sets auxiliary rewards for low-level skill training based on the advantage function of the high-level policy. This auxiliary reward enables efficient, simultaneous learning of the high-level policy and low-level skills without using task-specific knowledge. In addition, we also theoretically prove that optimizing low-level skills with this auxiliary reward will increase the task return for the joint policy. Experimental results show that our algorithm dramatically outperforms other state-of-the-art HRL methods in Mujoco domains. We also find both low-level and high-level policies trained by our algorithm transferable.

READ FULL TEXT

page 6

page 8

page 15

research
10/20/2021

Hierarchical Skills for Efficient Exploration

In reinforcement learning, pre-trained low-level skills have the potenti...
research
04/10/2017

Stochastic Neural Networks for Hierarchical Reinforcement Learning

Deep reinforcement learning has achieved many impressive results in rece...
research
09/21/2022

Hierarchical Decision Transformer

Sequence models in reinforcement learning require task knowledge to esti...
research
01/19/2023

Keyframe Demonstration Seeded and Bayesian Optimized Policy Search

This paper introduces a novel Learning from Demonstration framework to l...
research
02/19/2018

Learning High-level Representations from Demonstrations

Hierarchical learning (HL) is key to solving complex sequential decision...
research
06/13/2019

Sub-policy Adaptation for Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning is a promising approach to long-hori...
research
10/17/2016

Learning and Transfer of Modulated Locomotor Controllers

We study a novel architecture and training procedure for locomotion task...

Please sign up or login with your details

Forgot password? Click here to reset