HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

by   Jameel Malik, et al.

3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. Existing methods addressing it directly regress hand meshes via 2D convolutional neural networks, which leads to artifacts due to perspective distortions in the images. To address the limitations of the existing methods, we develop HandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. The input to our network is a 3D voxelized-depth-map-based on the truncated signed distance function (TSDF). HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology and which is the most accurate representation. The second representation is the hand surface that preserves the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which does not rely on training data. In extensive evaluations on three public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and HO-3D, the proposed HandVoxNet++ achieves the state-of-the-art performance. In this journal extension of our previous approach presented at CVPR 2020, we gain 41.09 datasets, respectively. Our method is ranked first on the HANDS19 challenge dataset (Task 1: Depth-Based 3D Hand Pose Estimation) at the moment of the submission of our results to the portal in August 2020.


page 3

page 8

page 9

page 10


HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map

3D hand shape and pose estimation from a single depth map is a new and c...

V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map

Most of the existing deep learning-based methods for 3D hand and human p...

Hand3D: Hand Pose Estimation using 3D Neural Network

We propose a novel 3D neural network architecture for 3D hand pose estim...

Structure-Aware 3D Hourglass Network for Hand Pose Estimation from Single Depth Image

In this paper, we propose a novel structure-aware 3D hourglass network f...

Lightweight Estimation of Hand Mesh and Biomechanically Feasible Kinematic Parameters

3D hand pose estimation is a long-standing challenge in both robotics an...

Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images

In this paper, we tackle the problem of 3D human shape estimation from s...

Direction matters: hand pose estimation from local surface normals

We present a hierarchical regression framework for estimating hand joint...

Please sign up or login with your details

Forgot password? Click here to reset