DeepAI AI Chat
Log In Sign Up

On Value Discrepancy of Imitation Learning

by   Tian Xu, et al.
Nanjing University

Imitation learning trains a policy from expert demonstrations. Imitation learning approaches have been designed from various principles, such as behavioral cloning via supervised learning, apprenticeship learning via inverse reinforcement learning, and GAIL via generative adversarial learning. In this paper, we propose a framework to analyze the theoretical property of imitation learning approaches based on discrepancy propagation analysis. Under the infinite-horizon setting, the framework leads to the value discrepancy of behavioral cloning in an order of O((1-γ)^-2). We also show that the framework leads to the value discrepancy of GAIL in an order of O((1-γ)^-1). It implies that GAIL has less compounding errors than behavioral cloning, which is also verified empirically in this paper. To the best of our knowledge, we are the first one to analyze GAIL's performance theoretically. The above results indicate that the proposed framework is a general tool to analyze imitation learning approaches. We hope our theoretical results can provide insights for future improvements in imitation learning algorithms.


page 1

page 2

page 3

page 4


Error Bounds of Imitating Policies and Environments

Imitation learning trains a policy by mimicking expert demonstrations. V...

f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning

Imitation learning (IL) aims to learn a policy from expert demonstration...

A Critique of Strictly Batch Imitation Learning

Recent work by Jarrett et al. attempts to frame the problem of offline i...

Asking for Help: Failure Prediction in Behavioral Cloning through Value Approximation

Recent progress in end-to-end Imitation Learning approaches has shown pr...

Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms

We formulate the problem of learning to imitate multiple, non-determinis...

Merge or Not? Learning to Group Faces via Imitation Learning

Given a large number of unlabeled face images, face grouping aims at clu...

Neuroprosthetic decoder training as imitation learning

Neuroprosthetic brain-computer interfaces function via an algorithm whic...