Half-body Portrait Relighting with Overcomplete Lighting Representation

06/25/2021
by   Guoxian Song, et al.
0

We present a neural-based model for relighting a half-body portrait image by simply referring to another portrait image with the desired lighting condition. Rather than following classical inverse rendering methodology that involves estimating normals, albedo and environment maps, we implicitly encode the subject and lighting in a latent space, and use these latent codes to generate relighted images by neural rendering. A key technical innovation is the use of a novel overcomplete lighting representation, which facilitates lighting interpolation in the latent space, as well as helping regularize the self-organization of the lighting latent space during training. In addition, we propose a novel multiplicative neural render that more effectively combines the subject and lighting latent codes for rendering. We also created a large-scale photorealistic rendered relighting dataset for training, which allows our model to generalize well to real images. Extensive experiments demonstrate that our system not only outperforms existing methods for referral-based portrait relighting, but also has the capability generate sequences of relighted images via lighting rotations.

READ FULL TEXT

page 1

page 4

page 6

page 7

page 8

page 10

research
06/11/2018

Learning to Estimate Indoor Lighting from 3D Objects

In this work, we propose a step towards a more accurate prediction of th...
research
04/01/2022

LASER: LAtent SpacE Rendering for 2D Visual Localization

We present LASER, an image-based Monte Carlo Localization (MCL) framewor...
research
08/31/2022

Dual-Space NeRF: Learning Animatable Avatars and Scene Lighting in Separate Spaces

Modeling the human body in a canonical space is a common practice for ca...
research
02/26/2020

Learning to Shade Hand-drawn Sketches

We present a fully automatic method to generate detailed and accurate ar...
research
02/26/2020

Learning to Shadow Hand-drawn Sketches

We present a fully automatic method to generate detailed and accurate ar...
research
11/26/2020

Lifting 2D StyleGAN for 3D-Aware Face Generation

We propose a framework, called LiftedGAN, that disentangles and lifts a ...
research
06/17/2022

Texture Generation Using Graph Generative Adversarial Network And Differentiable Rendering

Novel texture synthesis for existing 3D mesh models is an important step...

Please sign up or login with your details

Forgot password? Click here to reset