Generalizable task representation learning from human demonstration videos: a geometric approach

02/28/2022
by   Jun Jin, et al.
0

We study the problem of generalizable task learning from human demonstration videos without extra training on the robot or pre-recorded robot motions. Given a set of human demonstration videos showing a task with different objects/tools (categorical objects), we aim to learn a representation of visual observation that generalizes to categorical objects and enables efficient controller design. We propose to introduce a geometric task structure to the representation learning problem that geometrically encodes the task specification from human demonstration videos, and that enables generalization by building task specification correspondence between categorical objects. Specifically, we propose CoVGS-IL, which uses a graph-structured task function to learn task representations under structural constraints. Our method enables task generalization by selecting geometric features from different objects whose inner connection relationships define the same task in geometric constraints. The learned task representation is then transferred to a robot controller using uncalibrated visual servoing (UVS); thus, the need for extra robot training or pre-recorded robot motions is removed.

READ FULL TEXT

page 1

page 5

page 6

research
09/29/2018

Robot eye-hand coordination learning by watching human demonstrations: a task function approximation approach

We present a robot eye-hand coordination learning method that can direct...
research
09/07/2022

K-VIL: Keypoints-based Visual Imitation Learning

Visual imitation learning provides efficient and intuitive solutions for...
research
03/02/2019

Evaluation of state representation methods in robot hand-eye coordination learning from demonstration

We evaluate different state representation methods in robot hand-eye coo...
research
03/17/2021

Learning Descriptor of Constrained Task from Demonstration

Constrained objects, such as doors and drawers are often complex and sha...
research
05/17/2022

Conditional Visual Servoing for Multi-Step Tasks

Visual Servoing has been effectively used to move a robot into specific ...
research
11/08/2019

Visual Geometric Skill Inference by Watching Human Demonstration

We study the problem of learning manipulation skills from human demonstr...
research
03/03/2021

Semantic constraints to represent common sense required in household actions for multi-modal Learning-from-observation robot

The paradigm of learning-from-observation (LfO) enables a robot to learn...

Please sign up or login with your details

Forgot password? Click here to reset