Log In Sign Up

Exploring Hierarchy-Aware Inverse Reinforcement Learning

by   Chris Cundy, et al.

We introduce a new generative model for human planning under the Bayesian Inverse Reinforcement Learning (BIRL) framework which takes into account the fact that humans often plan using hierarchical strategies. We describe the Bayesian Inverse Hierarchical RL (BIHRL) algorithm for inferring the values of hierarchical planners, and use an illustrative toy model to show that BIHRL retains accuracy where standard BIRL fails. Furthermore, BIHRL is able to accurately predict the goals of `Wikispeedia' game players, with inclusion of hierarchical structure in the model resulting in a large boost in accuracy. We show that BIHRL is able to significantly outperform BIRL even when we only have a weak prior on the hierarchical structure of the plans available to the agent, and discuss the significant challenges that remain for scaling up this framework to more realistic settings.


page 1

page 2

page 3

page 4


Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Hierarchical reinforcement learning has demonstrated significant success...

Modeling Human Inference of Others' Intentions in Complex Situations with Plan Predictability Bias

A recent approach based on Bayesian inverse planning for the "theory of ...

A Bayesian Hierarchical Model for Criminal Investigations

Potential violent criminals will often need to go through a sequence of ...

Robust Hierarchical Planning with Policy Delegation

We propose a novel framework and algorithm for hierarchical planning bas...

Hierarchical Reinforcement Learning with Optimal Level Synchronization based on a Deep Generative Model

The high-dimensional or sparse reward task of a reinforcement learning (...

Possibility Before Utility: Learning And Using Hierarchical Affordances

Reinforcement learning algorithms struggle on tasks with complex hierarc...