Learning Illumination from Diverse Portraits

08/05/2020
by   Chloe LeGendre, et al.
9

We present a learning-based technique for estimating high dynamic range (HDR), omnidirectional illumination from a single low dynamic range (LDR) portrait image captured under arbitrary indoor or outdoor lighting conditions. We train our model using portrait photos paired with their ground truth environmental illumination. We generate a rich set of such photos by using a light stage to record the reflectance field and alpha matte of 70 diverse subjects in various expressions. We then relight the subjects using image-based relighting with a database of one million HDR lighting environments, compositing the relit subjects onto paired high-resolution background imagery recorded during the lighting acquisition. We train the lighting estimation model using rendering-based loss functions and add a multi-scale adversarial loss to estimate plausible high frequency lighting detail. We show that our technique outperforms the state-of-the-art technique for portrait-based lighting estimation, and we also show that our method reliably handles the inherent ambiguity between overall lighting strength and surface albedo, recovering a similar scale of illumination for subjects with diverse skin tones. We demonstrate that our method allows virtual objects and digital characters to be added to a portrait photograph with consistent illumination. Our lighting inference runs in real-time on a smartphone, enabling realistic rendering and compositing of virtual objects into live video for augmented reality applications.

READ FULL TEXT

page 1

page 2

page 3

page 5

page 6

page 7

page 10

page 12

research
04/02/2019

DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality

We present a learning-based method to infer plausible high dynamic range...
research
06/15/2020

A study of the effect of the illumination model on the generation of synthetic training datasets

The use of computer generated images to train Deep Neural Networks is a ...
research
07/12/2020

IllumiNet: Transferring Illumination from Planar Surfaces to Virtual Objects in Augmented Reality

This paper presents an illumination estimation method for virtual object...
research
04/18/2023

Deep Dynamic Cloud Lighting

Sky illumination is a core source of lighting in rendering, and a substa...
research
05/12/2022

HDR Lighting Dilation for Dynamic Range Reduction on Virtual Production Stages

We present a technique to reduce the dynamic range of an HDRI lighting e...
research
04/26/2023

EverLight: Indoor-Outdoor Editable HDR Lighting Estimation

Because of the diversity in lighting environments, existing illumination...
research
05/19/2017

A Lightweight Approach for On-the-Fly Reflectance Estimation

Estimating surface reflectance (BRDF) is one key component for complete ...

Please sign up or login with your details

Forgot password? Click here to reset