DeepAI
Log In Sign Up

Abstract Value Iteration for Hierarchical Reinforcement Learning

10/29/2020
by   Kishor Jothimurugan, et al.
14

We propose a novel hierarchical reinforcement learning framework for control with continuous state and action spaces. In our framework, the user specifies subgoal regions which are subsets of states; then, we (i) learn options that serve as transitions between these subgoal regions, and (ii) construct a high-level plan in the resulting abstract decision process (ADP). A key challenge is that the ADP may not be Markov, which we address by proposing two algorithms for planning in the ADP. Our first algorithm is conservative, allowing us to prove theoretical guarantees on its performance, which help inform the design of subgoal regions. Our second algorithm is a practical one that interweaves planning at the abstract level and learning at the concrete level. In our experiments, we demonstrate that our approach outperforms state-of-the-art hierarchical reinforcement learning algorithms on several challenging benchmarks.

READ FULL TEXT
06/25/2021

Compositional Reinforcement Learning from Logical Specifications

We study the problem of learning control policies for complex tasks give...
12/16/2019

Planning with Abstract Learned Models While Learning Transferable Subtasks

We introduce an algorithm for model-based hierarchical reinforcement lea...
05/17/2016

Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clustering

This paper introduces an automated skill acquisition framework in reinfo...
03/10/2016

Hierarchical Linearly-Solvable Markov Decision Problems

We present a hierarchical reinforcement learning framework that formulat...
10/16/2018

Learning abstract planning domains and mappings to real world perceptions

Most of the works on planning and learning, e.g., planning by (model bas...
02/02/2021

Metrics and continuity in reinforcement learning

In most practical applications of reinforcement learning, it is untenabl...

Code Repositories

abstract-value-iteration

Code for experiments in the paper "Abstract Value Iteration for Hierarchical Reinforcement Learning" (https://arxiv.org/abs/2010.15638)


view repo