GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB

12/04/2017
by   Franziska Mueller, et al.
0

We address the highly challenging problem of real-time 3D hand tracking based on a monocular RGB-only sequence. Our tracking method combines a convolutional neural network with a kinematic 3D hand model, such that it generalizes well to unseen data, is robust to occlusions and varying camera viewpoints, and leads to anatomically plausible as well as temporally smooth hand motions. For training our CNN we propose a novel approach for the synthetic generation of training data that is based on a geometrically consistent image-to-image translation network. To be more specific, we use a neural network that translates synthetic images to "real" images, such that the so-generated images follow the same statistical distribution as real-world hand images. For training this translation network we combine an adversarial loss and a cycle-consistency loss with a geometric consistency loss in order to preserve geometric properties (such as hand pose) during translation. We demonstrate that our hand tracking system outperforms the current state-of-the-art on challenging RGB-only footage.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 8

page 10

research
03/10/2020

Unpaired Image-to-Image Translation using Adversarial Consistency Loss

Unpaired image-to-image translation is a class of vision problems whose ...
research
05/03/2017

VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera

We present the first real-time method to capture the full global 3D skel...
research
06/04/2018

Synthetic data generation for end-to-end thermal infrared tracking

The usage of both off-the-shelf and end-to-end trained deep networks hav...
research
05/11/2020

Combining Deep Learning with Geometric Features for Image based Localization in the Gastrointestinal Tract

Tracking monocular colonoscope in the Gastrointestinal tract (GI) is a c...
research
07/05/2018

A Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking

We propose an algorithm for real-time 6DOF pose tracking of rigid 3D obj...
research
02/11/2016

Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model

Real-time marker-less hand tracking is of increasing importance in human...
research
04/03/2017

Capturing Hand Motion with an RGB-D Sensor, Fusing a Generative Model with Salient Points

Hand motion capture has been an active research topic in recent years, f...

Please sign up or login with your details

Forgot password? Click here to reset