Log In Sign Up

Unsupervised Perceptual Rewards for Imitation Learning

by   Pierre Sermanet, et al.

Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a reward function takes considerable hand engineering and often requires additional sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple implicit intermediate steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide feedback on these intermediate steps. To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit specification of sub-goals. The resulting reward functions can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also show that our method can be used to learn a real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task. Supplementary material and data are available at


page 1

page 2

page 7

page 8

page 12

page 13

page 14

page 15


Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

Reinforcement learning provides a general framework for learning robotic...

Dynamical Distance Learning for Unsupervised and Semi-Supervised Skill Discovery

Reinforcement learning requires manual specification of a reward functio...

Automatic Evaluation of Excavator Operators using Learned Reward Functions

Training novice users to operate an excavator for learning different ski...

Generalizing Skills with Semi-Supervised Reinforcement Learning

Deep reinforcement learning (RL) can acquire complex behaviors from low-...

A Framework for Data-Driven Robotics

We present a framework for data-driven robotics that makes use of a larg...

Robot Sound Interpretation: Learning Visual-Audio Representations for Voice-Controlled Robots

Inspired by sensorimotor theory, we propose a novel pipeline for voice-c...

Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft

Real-world tasks of interest are generally poorly defined by human-reada...

Code Repositories


Unsupervised Perceptual Rewards for Imitation Learning

view repo