HandMime: Sign Language Fingerspelling Acquisition via Imitation Learning

09/12/2022
by   Federico Tavella, et al.
0

Learning fine-grained movements is among the most challenging topics in robotics. This holds true especially for robotic hands. Robotic sign language acquisition or, more specifically, fingerspelling sign language acquisition in robots can be considered a specific instance of such challenge. In this paper, we propose an approach for learning dexterous motor imitation from videos examples, without the use of any additional information. We build an URDF model of a robotic hand with a single actuator for each joint. By leveraging pre-trained deep vision models, we extract the 3D pose of the hand from RGB videos. Then, using state-of-the-art reinforcement learning algorithms for motion imitation (namely, proximal policy optimisation), we train a policy to reproduce the movement extracted from the demonstrations. We identify the best set of hyperparameters to perform imitation based on a reference motion. Additionally, we demonstrate the ability of our approach to generalise over 6 different fingerspelled letters.

READ FULL TEXT
research
07/11/2017

Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation

Imitation learning is an effective approach for autonomous systems to ac...
research
10/03/2018

Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation

Human hand actions are quite complex, especially when they involve objec...
research
02/25/2020

Metric-Based Imitation Learning Between Two Dissimilar Anthropomorphic Robotic Arms

The development of autonomous robotic systems that can learn from human ...
research
05/21/2021

A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Control

We present a simple and intuitive approach for interactive control of ph...
research
04/07/2022

3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

Automatic laparoscope motion control is fundamentally important for surg...
research
07/27/2019

Self-Imitation Learning of Locomotion Movements through Termination Curriculum

Animation and machine learning research have shown great advancements in...
research
03/19/2020

3D Ego-Pose Estimation via Imitation Learning

Ego-pose estimation, i.e., estimating a person's 3D pose with a single w...

Please sign up or login with your details

Forgot password? Click here to reset