Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning

by   Jongjin Park, et al.
KAIST 수리과학과

Behavioral cloning has proven to be effective for learning sequential decision-making policies from expert demonstrations. However, behavioral cloning often suffers from the causal confusion problem where a policy relies on the noticeable effect of expert actions due to the strong correlation but not the cause we desire. This paper presents Object-aware REgularizatiOn (OREO), a simple technique that regularizes an imitation policy in an object-aware manner. Our main idea is to encourage a policy to uniformly attend to all semantic objects, in order to prevent the policy from exploiting nuisance variables strongly correlated with expert actions. To this end, we introduce a two-stage approach: (a) we extract semantic objects from images by utilizing discrete codes from a vector-quantized variational autoencoder, and (b) we randomly drop the units that share the same discrete code together, i.e., masking out semantic objects. Our experiments demonstrate that OREO significantly improves the performance of behavioral cloning, outperforming various other regularization and causality-based methods on a variety of Atari environments and a self-driving CARLA environment. We also show that our method even outperforms inverse reinforcement learning methods trained with a considerable amount of environment interaction.


page 2

page 3

page 4

page 6

page 9


Causal Confusion in Imitation Learning

Behavioral cloning reduces policy learning to supervised learning by tra...

Causal Imitative Model for Autonomous Driving

Imitation learning is a powerful approach for learning autonomous drivin...

Regularizing Adversarial Imitation Learning Using Causal Invariance

Imitation learning methods are used to infer a policy in a Markov decisi...

IQ-Learn: Inverse soft-Q Learning for Imitation

In many sequential decision-making problems (e.g., robotics control, gam...

Hierarchical Adversarial Inverse Reinforcement Learning

Hierarchical Imitation Learning (HIL) has been proposed to recover highl...

Concurrent Training Improves the Performance of Behavioral Cloning from Observation

Learning from demonstration is widely used as an efficient way for robot...

SQIL: Imitation Learning via Regularized Behavioral Cloning

Learning to imitate expert behavior given action demonstrations containi...

Please sign up or login with your details

Forgot password? Click here to reset