Generalizable Neural Radiance Fields for Novel View Synthesis with Transformer

06/10/2022
by   Dan Wang, et al.
0

We propose a Transformer-based NeRF (TransNeRF) to learn a generic neural radiance field conditioned on observed-view images for the novel view synthesis task. By contrast, existing MLP-based NeRFs are not able to directly receive observed views with an arbitrary number and require an auxiliary pooling-based operation to fuse source-view information, resulting in the missing of complicated relationships between source views and the target rendering view. Furthermore, current approaches process each 3D point individually and ignore the local consistency of a radiance field scene representation. These limitations potentially can reduce their performance in challenging real-world applications where large differences between source views and a novel rendering view may exist. To address these challenges, our TransNeRF utilizes the attention mechanism to naturally decode deep associations of an arbitrary number of source views into a coordinate-based scene representation. Local consistency of shape and appearance are considered in the ray-cast space and the surrounding-view space within a unified Transformer network. Experiments demonstrate that our TransNeRF, trained on a wide variety of scenes, can achieve better performance in comparison to state-of-the-art image-based neural rendering methods in both scene-agnostic and per-scene finetuning scenarios especially when there is a considerable gap between source views and a rendering view.

READ FULL TEXT

page 1

page 2

page 9

page 10

research
02/25/2021

IBRNet: Learning Multi-View Image-Based Rendering

We present a method that synthesizes novel views of complex scenes by in...
research
07/27/2022

Is Attention All NeRF Needs?

We present Generalizable NeRF Transformer (GNT), a pure, unified transfo...
research
05/11/2023

SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Views

We study to generate novel views of indoor scenes given sparse input vie...
research
04/28/2022

NeurMiPs: Neural Mixture of Planar Experts for View Synthesis

We present Neural Mixtures of Planar Experts (NeurMiPs), a novel planar-...
research
03/07/2023

Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis

Rendering novel views from captured multi-view images has made considera...
research
07/29/2022

End-to-end View Synthesis via NeRF Attention

In this paper, we present a simple seq2seq formulation for view synthesi...
research
04/09/2020

Sequential Neural Rendering with Transformer

This paper address the problem of novel view synthesis by means of neura...

Please sign up or login with your details

Forgot password? Click here to reset