Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

06/15/2021
by   Franziska Mueller, et al.
0

We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. In order to achieve this, we embed a recent parametric hand pose and shape model and a dense correspondence predictor based on a deep neural network into a suitable energy minimization framework. For training the correspondence prediction network, we synthesize a two-hand dataset based on physical simulations that includes both hand pose and shape annotations while at the same time avoiding inter-hand penetrations. To achieve real-time rates, we phrase the model fitting in terms of a nonlinear least-squares problem so that the energy can be optimized based on a highly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work, including tight two-hand grasps, significant inter-hand occlusions, and gesture interaction.

READ FULL TEXT

page 1

page 3

page 5

page 8

page 9

page 10

research
06/22/2021

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

Tracking and reconstructing the 3D pose and geometry of two hands in int...
research
03/17/2023

ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty

Despite their potential, markerless hand tracking technologies are not y...
research
02/11/2016

Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model

Real-time marker-less hand tracking is of increasing importance in human...
research
02/16/2018

Real-Time 3D Shape of Micro-Details

Motivated by the growing demand for interactive environments, we propose...
research
05/15/2017

Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo

We present a novel solution to the problem of 3D tracking of the articul...
research
04/07/2017

Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

We present an approach for real-time, robust and accurate hand pose esti...
research
10/07/2018

Deep Model-Based 6D Pose Refinement in RGB

We present a novel approach for model-based 6D pose refinement in color ...

Please sign up or login with your details

Forgot password? Click here to reset