Log In Sign Up

Soft Options Critic

by   Elita Lobo, et al.

The option-critic paper and several variants have successfully demonstrated the use of the options framework proposed by Barto et al to scale learning and planning in hierarchical tasks. Although most of these frameworks use entropy as a regularizer to improve exploration, they do not maximize entropy along with returns at every time step. In this paper we investigate the effect of maximizing entropy of each options and inter-option policy in options framework. We adopt the architecture of the recently introduced soft-actor critic algorithm to enable learning of robust options in continuous and discrete action spaces in a off-policy manner thus also making it sample efficient. In this paper we derive the soft options improvement theorem and propose a novel soft-options framework to incorporate maximization of entropy of actions and options in a constrained manner. Our experiments show that maximizing entropy of actions and options in a constrained manner with high learning rate does not harm the main objective of maximizing returns and hence outperforms vanilla options-critic framework in most hierarchical tasks. We also observe faster recovery when the environment is subject to perturbations.


page 1

page 2

page 3

page 4


Learnings Options End-to-End for Continuous Action Tasks

We present new results on learning temporally extended actions for conti...

The Eigenoption-Critic Framework

Eigenoptions (EOs) have been recently introduced as a promising idea for...

Diverse Exploration via InfoMax Options

In this paper, we study the problem of autonomously discovering temporal...

DAC: The Double Actor-Critic Architecture for Learning Options

We reformulate the option framework as two parallel augmented MDPs. Unde...

Safe Option-Critic: Learning Safety in the Option-Critic Architecture

Designing hierarchical reinforcement learning algorithms that induce a n...

SOAC: The Soft Option Actor-Critic Architecture

The option framework has shown great promise by automatically extracting...

Data-efficient Hindsight Off-policy Option Learning

Solutions to most complex tasks can be decomposed into simpler, intermed...