DeepAI AI Chat
Log In Sign Up

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

09/23/2019
by   Ofir Nachum, et al.
Google
berkeley college
14

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

READ FULL TEXT

page 3

page 5

page 7

08/13/2019

Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

Manipulation and locomotion are closely related problems that are often ...
07/13/2018

Exploring Hierarchy-Aware Inverse Reinforcement Learning

We introduce a new generative model for human planning under the Bayesia...
12/06/2022

First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation

Go-Explore achieved breakthrough performance on challenging reinforcemen...
11/30/2018

Modulated Policy Hierarchies

Solving tasks with sparse rewards is a main challenge in reinforcement l...
06/22/2022

Learning Representations for Control with Hierarchical Forward Models

Learning control from pixels is difficult for reinforcement learning (RL...
10/24/2019

HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators

Most common navigation tasks in human environments require auxiliary arm...
01/20/2022

Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

The ability to discover behaviours from past experience and transfer the...