Light Stage Super-Resolution: Continuous High-Frequency Relighting

10/17/2020
by   Tiancheng Sun, et al.
3

The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects.

READ FULL TEXT

page 1

page 7

page 8

page 9

page 10

page 11

research
08/09/2020

Neural Light Transport for Relighting and View Synthesis

The light transport (LT) of a scene describes how it appears under diffe...
research
11/23/2019

Joint Spatial and Angular Super-Resolution from a Single Image

Synthesizing a densely sampled light field from a single image is highly...
research
03/16/2023

Depth Super-Resolution from Explicit and Implicit High-Frequency Features

We propose a novel multi-stage depth super-resolution network, which pro...
research
05/23/2018

A hybrid approach of interpolations and CNN to obtain super-resolution

We propose a novel architecture that learns an end-to-end mapping functi...
research
06/26/2020

High Resolution Zero-Shot Domain Adaptation of Synthetically Rendered Face Images

Generating photorealistic images of human faces at scale remains a prohi...
research
05/18/2020

Portrait Shadow Manipulation

Casually-taken portrait photographs often suffer from unflattering light...
research
03/21/2023

Learning a 3D Morphable Face Reflectance Model from Low-cost Data

Modeling non-Lambertian effects such as facial specularity leads to a mo...

Please sign up or login with your details

Forgot password? Click here to reset