NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

09/02/2021
by   Yi Wei, et al.
1

In this work, we present a new multi-view depth estimation method that utilizes both conventional SfM reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene by finetuning on its sparse SfM reconstruction. Then, we show that the shape-radiance ambiguity of NeRF still exists in indoor environments and propose to address the issue by employing the adapted depth priors to monitor the sampling process of volume rendering. Finally, a per-pixel confidence map acquired by error computation on the rendered image can be used to further improve the depth quality. Experiments show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes, with surprising findings presented on the effectiveness of correspondence-based optimization and NeRF-based optimization over the adapted depth priors. In addition, we show that the guided optimization scheme does not sacrifice the original synthesis capability of neural radiance fields, improving the rendering quality on both seen and novel views. Code is available at https://github.com/weiyithu/NerfingMVS.

READ FULL TEXT

page 1

page 3

page 4

page 6

page 7

page 8

research
04/13/2023

NeRFVS: Neural Radiance Fields for Free View Synthesis via Geometry Scaffolds

We present NeRFVS, a novel neural radiance fields (NeRF) based method to...
research
03/23/2023

SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates

Neural radiance fields (NeRFs) have enabled high fidelity 3D reconstruct...
research
03/30/2023

S-VolSDF: Sparse Multi-View Stereo Regularization of Neural Implicit Surfaces

Neural rendering of implicit surfaces performs well in 3D vision applica...
research
05/30/2023

DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation

Neural radiance fields (NeRF) shows powerful performance in novel view s...
research
09/13/2023

Exploiting Multiple Priors for Neural 3D Indoor Reconstruction

Neural implicit modeling permits to achieve impressive 3D reconstruction...
research
02/28/2023

HelixSurf: A Robust and Efficient Neural Implicit Surface Learning of Indoor Scenes with Iterative Intertwined Regularization

Recovery of an underlying scene geometry from multiview images stands as...
research
07/27/2023

NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection

We present NeRF-Det, a novel method for indoor 3D detection with posed R...

Please sign up or login with your details

Forgot password? Click here to reset