Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning

06/20/2020
by   Tianren Zhang, et al.
10

Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is often large. Searching in a large goal space poses difficulties for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a k-step adjacency region centered by the current state using an adjacency constraint. We theoretically prove that the proposed adjacency constraint preserves the optimal hierarchical policy, and show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks show that our method outperforms the state-of-the-art HRL approaches.

READ FULL TEXT

page 7

page 8

page 13

page 18

page 19

research
10/30/2021

Adjacency constraint for efficient hierarchical reinforcement learning

Goal-conditioned Hierarchical Reinforcement Learning (HRL) is a promisin...
research
01/24/2022

Adversarially Guided Subgoal Generation for Hierarchical Reinforcement Learning

Hierarchical reinforcement learning (HRL) proposes to solve difficult ta...
research
10/26/2021

Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning

Goal-conditioned hierarchical reinforcement learning (HRL) has shown pro...
research
11/10/2021

Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving

Despite advances in hierarchical reinforcement learning, its application...
research
02/05/2019

Adjacency-constrained hierarchical clustering of a band similarity matrix with application to Genomics

Motivation: Genomic data analyses such as Genome-Wide Association Studie...
research
11/22/2018

Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning

In hierarchical reinforcement learning a major challenge is determining ...
research
07/17/2021

Hierarchical Reinforcement Learning with Optimal Level Synchronization based on a Deep Generative Model

The high-dimensional or sparse reward task of a reinforcement learning (...

Please sign up or login with your details

Forgot password? Click here to reset