DeepAI
Log In Sign Up

On the Role of Weight Sharing During Deep Option Learning

12/31/2019
by   Matthew Riemer, et al.
0

The options framework is a popular approach for building temporally extended actions in reinforcement learning. In particular, the option-critic architecture provides general purpose policy gradient theorems for learning actions from scratch that are extended in time. However, past work makes the key assumption that each of the components of option-critic has independent parameters. In this work we note that while this key assumption of the policy gradient theorems of option-critic holds in the tabular case, it is always violated in practice for the deep function approximation setting. We thus reconsider this assumption and consider more general extensions of option-critic and hierarchical option-critic training that optimize for the full architecture with each update. It turns out that not assuming parameter independence challenges a belief in prior work that training the policy over options can be disentangled from the dynamics of the underlying options. In fact, learning can be sped up by focusing the policy over options on states where options are actually likely to terminate. We put our new algorithms to the test in application to sample efficient learning of Atari games, and demonstrate significantly improved stability and faster convergence when learning long options.

READ FULL TEXT
09/16/2016

The Option-Critic Architecture

Temporal abstraction is key to scaling up learning and planning in reinf...
11/20/2019

Hierarchical Average Reward Policy Gradient Algorithms

Option-critic learning is a general-purpose reinforcement learning (RL) ...
01/09/2017

On Reject and Refine Options in Multicategory Classification

In many real applications of statistical learning, a decision made from ...
10/27/2018

Learning Abstract Options

Building systems that autonomously create temporal abstractions from dat...
12/04/2018

Natural Option Critic

The recently proposed option-critic architecture Bacon et al. provide a ...
07/21/2018

Safe Option-Critic: Learning Safety in the Option-Critic Architecture

Designing hierarchical reinforcement learning algorithms that induce a n...
05/23/2019

Soft Options Critic

The option-critic paper and several variants have successfully demonstra...