Learning Detailed Radiance Manifolds for High-Fidelity and 3D-Consistent Portrait Synthesis from Monocular Image

11/25/2022
by   Yu Deng, et al.
0

A key challenge for novel view synthesis of monocular portrait images is 3D consistency under continuous pose variations. Most existing methods rely on 2D generative models which often leads to obvious 3D inconsistency artifacts. We present a 3D-consistent novel view synthesis approach for monocular portrait images based on a recent proposed 3D-aware GAN, namely Generative Radiance Manifolds (GRAM), which has shown strong 3D consistency at multiview image generation of virtual subjects via the radiance manifolds representation. However, simply learning an encoder to map a real image into the latent space of GRAM can only reconstruct coarse radiance manifolds without faithful fine details, while improving the reconstruction fidelity via instance-specific optimization is time-consuming. We introduce a novel detail manifolds reconstructor to learn 3D-consistent fine details on the radiance manifolds from monocular images, and combine them with the coarse radiance manifolds for high-fidelity reconstruction. The 3D priors derived from the coarse radiance manifolds are used to regulate the learned details to ensure reasonable synthesized results at novel views. Trained on in-the-wild 2D images, our method achieves high-fidelity and 3D-consistent portrait synthesis largely outperforming the prior art.

READ FULL TEXT

page 7

page 18

page 19

page 20

page 21

page 22

page 23

page 24

research
11/28/2022

High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

High-fidelity facial avatar reconstruction from a monocular video is a s...
research
03/15/2022

S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image

We present a novel face reconstruction method capable of reconstructing ...
research
01/13/2020

High-Fidelity Synthesis with Disentangled Representation

Learning disentangled representation of data without supervision is an i...
research
04/12/2022

GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation

Despite Neural Radiance Fields (NeRF) showing compelling results in phot...
research
01/03/2023

Class-Continuous Conditional Generative Neural Radiance Field

The 3D-aware image synthesis focuses on conserving spatial consistency b...
research
05/09/2022

NeuralHDHair: Automatic High-fidelity Hair Modeling from a Single Image Using Implicit Neural Representations

Undoubtedly, high-fidelity 3D hair plays an indispensable role in digita...
research
06/17/2023

MA-NeRF: Motion-Assisted Neural Radiance Fields for Face Synthesis from Sparse Images

We address the problem of photorealistic 3D face avatar synthesis from s...

Please sign up or login with your details

Forgot password? Click here to reset