DeepAI AI Chat
Log In Sign Up

Context-Specific Representation Abstraction for Deep Option Learning

by   Marwa Abdulhai, et al.

Hierarchical reinforcement learning has focused on discovering temporally extended actions, such as options, that can provide benefits in problems requiring extensive exploration. One promising approach that learns these options end-to-end is the option-critic (OC) framework. We examine and show in this paper that OC does not decompose a problem into simpler sub-problems, but instead increases the size of the search over policy space with each option considering the entire state space during learning. This issue can result in practical limitations of this method, including sample inefficient learning. To address this problem, we introduce Context-Specific Representation Abstraction for Deep Option Learning (CRADOL), a new framework that considers both temporal abstraction and context-specific representation abstraction to effectively reduce the size of the search over policy space. Specifically, our method learns a factored belief state representation that enables each option to learn a policy over only a subsection of the state space. We test our method against hierarchical, non-hierarchical, and modular recurrent neural network baselines, demonstrating significant sample efficiency improvements in challenging partially observable environments.


page 2

page 6


Attention Option-Critic

Temporal abstraction in reinforcement learning is the ability of an agen...

Matching options to tasks using Option-Indexed Hierarchical Reinforcement Learning

The options framework in Hierarchical Reinforcement Learning breaks down...

Variational Temporal Abstraction

We introduce a variational approach to learning and inference of tempora...

Temporally Extended Successor Representations

We present a temporally extended variation of the successor representati...

MDP Abstraction with Successor Features

Abstraction plays an important role for generalisation of knowledge and ...

Hierarchical State Abstraction Based on Structural Information Principles

State abstraction optimizes decision-making by ignoring irrelevant envir...

Data-efficient Hindsight Off-policy Option Learning

Solutions to most complex tasks can be decomposed into simpler, intermed...