HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances

10/11/2022
by   Yue Jiang, et al.
0

Monocular 3D human performance capture is indispensable for many applications in computer graphics and vision for enabling immersive experiences. However, detailed capture of humans requires tracking of multiple aspects, including the skeletal pose, the dynamic surface, which includes clothing, hand gestures as well as facial expressions. No existing monocular method allows joint tracking of all these components. To this end, we propose HiFECap, a new neural human performance capture approach, which simultaneously captures human pose, clothing, facial expression, and hands just from a single RGB video. We demonstrate that our proposed network architecture, the carefully designed training strategy, and the tight integration of parametric face and hand models to a template mesh enable the capture of all these individual aspects. Importantly, our method also captures high-frequency details, such as deforming wrinkles on the clothes, better than the previous works. Furthermore, we show that HiFECap outperforms the state-of-the-art human performance capture approaches qualitatively and quantitatively while for the first time capturing all aspects of the human.

READ FULL TEXT

page 1

page 4

page 8

page 9

page 10

page 19

page 21

page 22

research
03/05/2021

Real-time RGBD-based Extended Body Pose Estimation

We present a system for real-time RGBD-based estimation of 3D human pose...
research
03/08/2023

X-Avatar: Expressive Human Avatars

We present X-Avatar, a novel avatar model that captures the full express...
research
06/22/2021

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

Tracking and reconstructing the 3D pose and geometry of two hands in int...
research
04/20/2023

Reconstructing Signing Avatars From Video Using Linguistic Priors

Sign language (SL) is the primary method of communication for the 70 mil...
research
09/21/2016

Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks

We present a real-time deep learning framework for video-based facial pe...
research
03/30/2022

Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation

We propose an approach to estimate arm and hand dynamics from monocular ...
research
10/21/2022

HDHumans: A Hybrid Approach for High-fidelity Digital Humans

Photo-real digital human avatars are of enormous importance in graphics,...

Please sign up or login with your details

Forgot password? Click here to reset