HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

07/02/2021
by   Jameel Malik, et al.
29

3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. Existing methods addressing it directly regress hand meshes via 2D convolutional neural networks, which leads to artifacts due to perspective distortions in the images. To address the limitations of the existing methods, we develop HandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. The input to our network is a 3D voxelized-depth-map-based on the truncated signed distance function (TSDF). HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology and which is the most accurate representation. The second representation is the hand surface that preserves the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which does not rely on training data. In extensive evaluations on three public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and HO-3D, the proposed HandVoxNet++ achieves the state-of-the-art performance. In this journal extension of our previous approach presented at CVPR 2020, we gain 41.09 datasets, respectively. Our method is ranked first on the HANDS19 challenge dataset (Task 1: Depth-Based 3D Hand Pose Estimation) at the moment of the submission of our results to the portal in August 2020.

READ FULL TEXT

page 3

page 8

page 9

page 10

research
04/03/2020

HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map

3D hand shape and pose estimation from a single depth map is a new and c...
research
11/20/2017

V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map

Most of the existing deep learning-based methods for 3D hand and human p...
research
04/07/2017

Hand3D: Hand Pose Estimation using 3D Neural Network

We propose a novel 3D neural network architecture for 3D hand pose estim...
research
12/26/2018

Structure-Aware 3D Hourglass Network for Hand Pose Estimation from Single Depth Image

In this paper, we propose a novel structure-aware 3D hourglass network f...
research
03/26/2023

Lightweight Estimation of Hand Mesh and Biomechanically Feasible Kinematic Parameters

3D hand pose estimation is a long-standing challenge in both robotics an...
research
08/01/2019

Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images

In this paper, we tackle the problem of 3D human shape estimation from s...
research
04/10/2016

Direction matters: hand pose estimation from local surface normals

We present a hierarchical regression framework for estimating hand joint...

Please sign up or login with your details

Forgot password? Click here to reset