DeepAI AI Chat
Log In Sign Up

Adversarial Option-Aware Hierarchical Imitation Learning

by   Mingxuan Jing, et al.

It has been a challenge to learning skills for an agent from long-horizon unannotated demonstrations. Existing approaches like Hierarchical Imitation Learning(HIL) are prone to compounding errors or suboptimal solutions. In this paper, we propose Option-GAIL, a novel method to learn skills at long horizon. The key idea of Option-GAIL is modeling the task hierarchy by options and train the policy via generative adversarial optimization. In particular, we propose an Expectation-Maximization(EM)-style algorithm: an E-step that samples the options of expert conditioned on the current learned policy, and an M-step that updates the low- and high-level policies of agent simultaneously to minimize the newly proposed option-occupancy measurement between the expert and the agent. We theoretically prove the convergence of the proposed algorithm. Experiments show that Option-GAIL outperforms other counterparts consistently across a variety of tasks.


Provable Hierarchical Imitation Learning via EM

Due to recent empirical successes, the options framework for hierarchica...

Hierarchical Adversarial Inverse Reinforcement Learning

Hierarchical Imitation Learning (HIL) has been proposed to recover highl...

Online Baum-Welch algorithm for Hierarchical Imitation Learning

The options framework for hierarchical reinforcement learning has increa...

DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations

An option is a short-term skill consisting of a control policy for a spe...

PODNet: A Neural Network for Discovery of Plannable Options

Learning from demonstration has been widely studied in machine learning ...

Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates

Environments with sparse rewards and long horizons pose a significant ch...

Option Encoder: A Framework for Discovering a Policy Basis in Reinforcement Learning

Option discovery and skill acquisition frameworks are integral to the fu...