Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes

04/06/2023
by   Zian Wang, et al.
0

Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and virtual object insertion. Recent NeRF based methods achieve impressive fidelity of 3D reconstruction, but bake the lighting and shadows into the radiance field, while mesh-based methods that facilitate intrinsic decomposition through differentiable rendering have not yet scaled to the complexity and scale of outdoor scenes. We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows. By faithfully disentangling complex geometry and materials from lighting effects, our method enables photorealistic relighting with specular and shadow effects on several outdoor datasets. Moreover, it supports physics-based scene manipulations such as virtual object insertion with ray-traced shadow casting.

READ FULL TEXT

page 1

page 3

page 7

page 8

page 15

research
08/19/2022

Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion

We consider the challenging problem of outdoor lighting estimation for t...
research
08/06/2020

Learning to Factorize and Relight a City

We propose a learning-based framework for disentangling outdoor scenes i...
research
06/15/2023

UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video

We show how to build a model that allows realistic, free-viewpoint rende...
research
10/15/2022

IBL-NeRF: Image-Based Lighting Formulation of Neural Radiance Fields

We propose IBL-NeRF, which decomposes the neural radiance fields (NeRF) ...
research
04/28/2021

DeRenderNet: Intrinsic Image Decomposition of Urban Scenes with Shape-(In)dependent Shading Rendering

We propose DeRenderNet, a deep neural network to decompose the albedo an...
research
07/20/2020

Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images

We present a deep learning approach to reconstruct scene appearance from...
research
04/08/2021

Appearance-Driven Automatic 3D Model Simplification

We present a suite of techniques for jointly optimizing triangle meshes ...

Please sign up or login with your details

Forgot password? Click here to reset