High-Fidelity 3D Digital Human Creation from RGB-D Selfies

by   Xiangkai Lin, et al.

We present a fully automatic system that can produce high-fidelity, photo-realistic 3D digital human characters with a consumer RGB-D selfie camera. The system only needs the user to take a short selfie RGB-D video while rotating his/her head, and can produce a high quality reconstruction in less than 30 seconds. Our main contribution is a new facial geometry modeling and reflectance synthesis procedure that significantly improves the state-of-the-art. Specifically, given the input video a two-stage frame selection algorithm is first employed to select a few high-quality frames for reconstruction. A novel, differentiable renderer based 3D Morphable Model (3DMM) fitting method is then applied to recover facial geometries from multiview RGB-D data, which takes advantages of extensive data generation and perturbation. Our 3DMM has much larger expressive capacities than conventional 3DMM, allowing us to recover more accurate facial geometry using merely linear bases. For reflectance synthesis, we present a hybrid approach that combines parametric fitting and CNNs to synthesize high-resolution albedo/normal maps with realistic hair/pore/wrinkle details. Results show that our system can produce faithful 3D characters with extremely realistic details. Code and the constructed 3DMM is publicly available.


page 1

page 9

page 11

page 12

page 13

page 14

page 15

page 16


CrossHuman: Learning Cross-Guidance from Multi-Frame Images for Human Reconstruction

We propose CrossHuman, a novel method that learns cross-guidance from pa...

HDHumans: A Hybrid Approach for High-fidelity Digital Humans

Photo-real digital human avatars are of enormous importance in graphics,...

High-Res Facial Appearance Capture from Polarized Smartphone Images

We propose a novel method for high-quality facial texture reconstruction...

StyleFaceV: Face Video Generation via Decomposing and Recomposing Pretrained StyleGAN3

Realistic generative face video synthesis has long been a pursuit in bot...

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars

Synthesizing high-fidelity head avatars is a central problem for compute...

High-fidelity Interpretable Inverse Rig: An Accurate and Sparse Solution Optimizing the Quartic Blendshape Model

We propose a method to fit arbitrarily accurate blendshape rig models by...

3D Face Reconstruction Using Color Photometric Stereo with Uncalibrated Near Point Lights

We present a new color photometric stereo (CPS) method that can recover ...

Code Repositories


Code and data for our paper "High-Fidelity 3D Digital Human Creation from RGB-D Selfies".

view repo


Project page for our paper "High-Fidelity 3D Digital Human Creation from RGB-D Selfies".

view repo

Please sign up or login with your details

Forgot password? Click here to reset