Towards Learning Neural Representations from Shadows

03/29/2022
by   Kushagra Tiwary, et al.
0

We present a method that learns neural scene representations from only shadows present in the scene. While traditional shape-from-shadow (SfS) algorithms reconstruct geometry from shadows, they assume a fixed scanning setup and fail to generalize to complex scenes. Neural rendering algorithms, on the other hand, rely on photometric consistency between RGB images but largely ignore physical cues such as shadows, which have been shown to provide valuable information about the scene. We observe that shadows are a powerful cue that can constrain neural scene representations to learn SfS, and even outperform NeRF to reconstruct otherwise hidden geometry. We propose a graphics-inspired differentiable approach to render accurate shadows with volumetric rendering, predicting a shadow map that can be compared to the ground truth shadow. Even with just binary shadow maps, we show that neural rendering can localize the object and estimate coarse geometry. Our approach reveals that sparse cues in images can be used to estimate geometry using differentiable volumetric rendering. Moreover, our framework is highly generalizable and can work alongside existing 3D reconstruction techniques that otherwise only use photometric consistency. Our code is made available in our supplementary materials.

READ FULL TEXT

page 2

page 12

page 13

page 18

page 20

page 21

research
04/16/2020

Fast Differentiable Raycasting for Neural Rendering using Sphere-based Representations

Differentiable rendering in combination with deep learning promises grea...
research
03/01/2021

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Recent work has demonstrated that volumetric scene representations combi...
research
05/27/2023

NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images

We present a neural rendering-based method called NeRO for reconstructin...
research
10/25/2021

Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering

Simultaneous reconstruction of geometry and reflectance properties in un...
research
05/11/2021

Vision-based Neural Scene Representations for Spacecraft

In advanced mission concepts with high levels of autonomy, spacecraft ne...
research
04/15/2021

GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

We present GANcraft, an unsupervised neural rendering framework for gene...
research
06/13/2022

SNeS: Learning Probably Symmetric Neural Surfaces from Incomplete Data

We present a method for the accurate 3D reconstruction of partly-symmetr...

Please sign up or login with your details

Forgot password? Click here to reset