Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

04/14/2021
by   Julian Chibane, et al.
0

Recent neural view synthesis methods have achieved impressive quality and realism, surpassing classical pipelines which rely on multi-view reconstruction. State-of-the-Art methods, such as NeRF, are designed to learn a single scene with a neural network and require dense multi-view inputs. Testing on a new scene requires re-training from scratch, which takes 2-3 days. In this work, we introduce Stereo Radiance Fields (SRF), a neural view synthesis approach that is trained end-to-end, generalizes to new scenes, and requires only sparse views at test time. The core idea is a neural architecture inspired by classical multi-view stereo methods, which estimates surface points by finding similar image regions in stereo images. In SRF, we predict color and density for each 3D point given an encoding of its stereo correspondence in the input images. The encoding is implicitly learned by an ensemble of pair-wise similarities – emulating classical stereo. Experiments show that SRF learns structure instead of overfitting on a scene. We train on multiple scenes of the DTU dataset and generalize to new ones without re-training, requiring only 10 sparse and spread-out views as input. We show that 10-15 minutes of fine-tuning further improve the results, achieving significantly sharper, more detailed results than scene-specific models. The code, model, and videos are available at https://virtualhumans.mpi-inf.mpg.de/srf/.

READ FULL TEXT

page 1

page 3

page 6

page 7

page 8

research
08/09/2021

NeuralMVS: Bridging Multi-View Stereo and Novel View Synthesis

Multi-View Stereo (MVS) is a core task in 3D computer vision. With the s...
research
03/29/2021

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

We present MVSNeRF, a novel neural rendering approach that can efficient...
research
05/24/2018

Stereo Magnification: Learning View Synthesis using Multiplane Images

The view synthesis problem--generating novel views of a scene from known...
research
04/19/2023

Single-View View Synthesis with Self-Rectified Pseudo-Stereo

Synthesizing novel views from a single view image is a highly ill-posed ...
research
08/09/2023

WaveNeRF: Wavelet-based Generalizable Neural Radiance Fields

Neural Radiance Field (NeRF) has shown impressive performance in novel v...
research
07/21/2022

Neural Pixel Composition: 3D-4D View Synthesis from Multi-Views

We present Neural Pixel Composition (NPC), a novel approach for continuo...
research
12/02/2022

Fast Non-Rigid Radiance Fields from Monocularized Data

3D reconstruction and novel view synthesis of dynamic scenes from collec...

Please sign up or login with your details

Forgot password? Click here to reset