Spatiotemporally Consistent HDR Indoor Lighting Estimation

05/07/2023
by   Zhengqin Li, et al.
0

We propose a physically-motivated deep learning framework to solve a general version of the challenging indoor lighting estimation problem. Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position. Particularly, when the input is an LDR video sequence, our framework not only progressively refines the lighting prediction as it sees more regions, but also preserves temporal consistency by keeping the refinement smooth. Our framework reconstructs a spherical Gaussian lighting volume (SGLV) through a tailored 3D encoder-decoder, which enables spatially consistent lighting prediction through volume ray tracing, a hybrid blending network for detailed environment maps, an in-network Monte-Carlo rendering layer to enhance photorealism for virtual object insertion, and recurrent neural networks (RNN) to achieve temporally consistent lighting prediction with a video sequence as the input. For training, we significantly enhance the OpenRooms public dataset of photorealistic synthetic indoor scenes with around 360K HDR environment maps of much higher resolution and 38K video sequences, rendered with GPU-based path tracing. Experiments show that our framework achieves lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods, leading to photorealistic AR applications such as object insertion.

READ FULL TEXT

page 1

page 4

page 7

page 8

page 11

page 12

page 13

page 14

research
02/13/2022

Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction

Lighting prediction from a single image is becoming increasingly importa...
research
09/13/2021

Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

In this work, we address the problem of jointly estimating albedo, norma...
research
06/10/2019

Fast Spatially-Varying Indoor Lighting Estimation

We propose a real-time method to estimate spatiallyvarying indoor lighti...
research
06/11/2018

Learning to Estimate Indoor Lighting from 3D Objects

In this work, we propose a step towards a more accurate prediction of th...
research
10/30/2021

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

We consider the challenging problem of predicting intrinsic object prope...
research
09/21/2022

Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation

Given a portrait image of a person and an environment map of the target ...
research
04/09/2021

Spatially-Varying Outdoor Lighting Estimation from Intrinsics

We present SOLID-Net, a neural network for spatially-varying outdoor lig...

Please sign up or login with your details

Forgot password? Click here to reset