HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation

11/30/2020
by   Jiefeng Li, et al.
0

Model-based 3D pose and shape estimation methods reconstruct a full 3D mesh for the human body by estimating several parameters. However, learning the abstract parameters is a highly non-linear process and suffers from image-model misalignment, leading to mediocre model performance. In contrast, 3D keypoint estimation methods combine deep CNN network with the volumetric representation to achieve pixel-level localization accuracy but may predict unrealistic body structure. In this paper, we address the above issues by bridging the gap between body mesh estimation and 3D keypoint estimation. We propose a novel hybrid inverse kinematics solution (HybrIK). HybrIK directly transforms accurate 3D joints to relative body-part rotations for 3D body mesh reconstruction, via the twist-and-swing decomposition. The swing rotation is analytically solved with 3D joints, and the twist rotation is derived from the visual cues through the neural network. We show that HybrIK preserves both the accuracy of 3D pose and the realistic body structure of the parametric human model, leading to a pixel-aligned 3D body mesh and a more accurate 3D pose than the pure 3D keypoint estimation methods. Without bells and whistles, the proposed method surpasses the state-of-the-art methods by a large margin on various 3D human pose and shape benchmarks. As an illustrative example, HybrIK outperforms all the previous methods by 13.2 mm MPJPE and 21.9 mm PVE on 3DPW dataset. Our code is available at https://github.com/Jeff-sjtu/HybrIK.

READ FULL TEXT

page 12

page 13

08/20/2020

Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose

Most of the recent deep learning-based 3D human pose and mesh estimation...
04/27/2021

KAMA: 3D Keypoint Aware Body Mesh Articulation

We present KAMA, a 3D Keypoint Aware Mesh Articulation approach that all...
09/21/2022

Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms

3D human pose and shape estimation (a.k.a. "human mesh recovery") has ac...
10/25/2019

TRB: A Novel Triplet Representation for Understanding 2D Human Body

Human pose and shape are two important components of 2D human body. Howe...
03/09/2022

Structure-Aware Flow Generation for Human Body Reshaping

Body reshaping is an important procedure in portrait photo retouching. D...
09/27/2019

Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop

Model-based human pose estimation is currently approached through two di...
12/31/2019

Learning 3D Human Shape and Pose from Dense Body Parts

Reconstructing 3D human shape and pose from a monocular image is challen...