Causal Imitation Learning with Unobserved Confounders

08/12/2022
by   Junzhe Zhang, et al.
0

One of the common ways children learn is by mimicking adults. Imitation learning focuses on learning policies with suitable performance from demonstrations generated by an expert, with an unspecified performance measure, and unobserved reward signal. Popular methods for imitation learning start by either directly mimicking the behavior policy of an expert (behavior cloning) or by learning a reward function that prioritizes observed expert trajectories (inverse reinforcement learning). However, these methods rely on the assumption that covariates used by the expert to determine her/his actions are fully observed. In this paper, we relax this assumption and study imitation learning when sensory inputs of the learner and the expert differ. First, we provide a non-parametric, graphical criterion that is complete (both necessary and sufficient) for determining the feasibility of imitation from the combinations of demonstration data and qualitative assumptions about the underlying environment, represented in the form of a causal model. We then show that when such a criterion does not hold, imitation could still be feasible by exploiting quantitative knowledge of the expert trajectories. Finally, we develop an efficient procedure for learning the imitating policy from experts' trajectories.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/10/2021

Imitation Learning by Reinforcement Learning

Imitation Learning algorithms learn a policy from demonstrations of expe...
research
05/30/2022

TaSIL: Taylor Series Imitation Learning

We propose Taylor Series Imitation Learning (TaSIL), a simple augmentati...
research
06/18/2019

RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstration

Imitation learning has long been an approach to alleviate the tractabili...
research
08/12/2022

Sequential Causal Imitation Learning with Unobserved Confounders

"Monkey see monkey do" is an age-old adage, referring to naïve imitation...
research
08/03/2022

Sequence Model Imitation Learning with Unobserved Contexts

We consider imitation learning problems where the expert has access to a...
research
10/13/2021

On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning

We consider the problem of using expert data with unobserved confounders...
research
07/29/2023

Initial State Interventions for Deconfounded Imitation Learning

Imitation learning suffers from causal confusion. This phenomenon occurs...

Please sign up or login with your details

Forgot password? Click here to reset