RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

06/22/2021
by   Jiayi Wang, et al.
11

Tracking and reconstructing the 3D pose and geometry of two hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, or sign language recognition. Existing works are either limited to simpler tracking settings (e.g., considering only a single hand or two spatially separated hands), or rely on less ubiquitous sensors, such as depth cameras. In contrast, in this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera that explicitly considers close interactions. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN that regresses multiple complementary pieces of information, including segmentation, dense matchings to a 3D hand model, and 2D keypoint positions, together with newly proposed intra-hand relative depth and inter-hand distance maps. These predictions are subsequently used in a generative model fitting framework in order to estimate pose and shape parameters of a 3D hand model for both hands. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline through an extensive ablation study. Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions. Moreover, our method even performs on-par with depth-based real-time methods.

READ FULL TEXT

page 1

page 4

page 6

page 8

page 12

page 13

page 14

page 15

research
05/15/2017

Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo

We present a novel solution to the problem of 3D tracking of the articul...
research
06/15/2021

Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

We present a novel method for real-time pose and shape reconstruction of...
research
02/11/2016

Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model

Real-time marker-less hand tracking is of increasing importance in human...
research
04/03/2017

Capturing Hand Motion with an RGB-D Sensor, Fusing a Generative Model with Salient Points

Hand motion capture has been an active research topic in recent years, f...
research
10/11/2022

HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances

Monocular 3D human performance capture is indispensable for many applica...
research
06/06/2015

Capturing Hands in Action using Discriminative Salient Points and Physics Simulation

Hand motion capture is a popular research field, recently gaining more a...
research
08/16/2023

DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field

Reconstructing hand-held objects from a single RGB image is an important...

Please sign up or login with your details

Forgot password? Click here to reset