DeepAI AI Chat
Log In Sign Up

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

by   Ofir Nachum, et al.
berkeley college

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.


page 3

page 5

page 7


Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

Manipulation and locomotion are closely related problems that are often ...

Exploring Hierarchy-Aware Inverse Reinforcement Learning

We introduce a new generative model for human planning under the Bayesia...

First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation

Go-Explore achieved breakthrough performance on challenging reinforcemen...

Modulated Policy Hierarchies

Solving tasks with sparse rewards is a main challenge in reinforcement l...

Learning Representations for Control with Hierarchical Forward Models

Learning control from pixels is difficult for reinforcement learning (RL...

HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators

Most common navigation tasks in human environments require auxiliary arm...

Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

The ability to discover behaviours from past experience and transfer the...