Language-Conditioned Imitation Learning for Robot Manipulation Tasks

10/22/2020
by   Simon Stepputtis, et al.
0

Imitation learning is a popular approach for teaching motor skills to robots. However, most approaches focus on extracting policy parameters from execution traces alone (i.e., motion trajectories and perceptual data). No adequate communication channel exists between the human expert and the robot to describe critical aspects of the task, such as the properties of the target object or the intended shape of the motion. Motivated by insights into the human teaching process, we introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent (e.g., "go to the large green bowl"). The training process then interrelates these two modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at runtime on new human commands and instructions, which allows for more fine-grained control over the trained policies while also reducing situational ambiguity. We demonstrate in a set of simulation experiments how our approach can learn language-conditioned manipulation policies for a seven-degree-of-freedom robot arm and compare the results to a variety of alternative methods.

READ FULL TEXT

page 6

page 8

research
09/05/2023

Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation

We tackle the problem of developing humanoid loco-manipulation skills wi...
research
05/25/2023

Imitating Task and Motion Planning with Visuomotor Transformers

Imitation learning is a powerful tool for training robot manipulation po...
research
05/10/2023

Learning Video-Conditioned Policies for Unseen Manipulation Tasks

The ability to specify robot commands by a non-expert user is critical f...
research
11/26/2019

Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration

In this work we propose a novel end-to-end imitation learning approach w...
research
12/16/2022

Learning and Extrapolation of Robotic Skills using Task-Parameterized Equation Learner Networks

Imitation learning approaches achieve good generalization within the ran...
research
04/13/2022

What Matters in Language Conditioned Robotic Imitation Learning

A long-standing goal in robotics is to build robots that can perform a w...
research
11/07/2017

Learning Robust Bed Making using Deep Imitation Learning with DART

Bed-making is a universal home task that can be challenging for senior c...

Please sign up or login with your details

Forgot password? Click here to reset