360FusionNeRF: Panoramic Neural Radiance Fields with Joint Guidance

09/28/2022
by   Shreyas Kulkarni, et al.
0

We present a method to synthesize novel views from a single 360^∘ panorama image based on the neural radiance field (NeRF). Prior studies in a similar setting rely on the neighborhood interpolation capability of multi-layer perceptions to complete missing regions caused by occlusion, which leads to artifacts in their predictions. We propose 360FusionNeRF, a semi-supervised learning framework where we introduce geometric supervision and semantic consistency to guide the progressive training process. Firstly, the input image is re-projected to 360^∘ images, and auxiliary depth maps are extracted at other camera positions. The depth supervision, in addition to the NeRF color guidance, improves the geometry of the synthesized views. Additionally, we introduce a semantic consistency loss that encourages realistic renderings of novel views. We extract these semantic features using a pre-trained visual encoder such as CLIP, a Vision Transformer trained on hundreds of millions of diverse 2D photographs mined from the web with natural language supervision. Experiments indicate that our proposed method can produce plausible completions of unobserved regions while preserving the features of the scene. When trained across various scenes, 360FusionNeRF consistently achieves the state-of-the-art performance when transferring to synthetic Structured3D dataset (PSNR 5 dataset (PSNR 3 LPIPS 18

READ FULL TEXT

page 1

page 2

page 5

page 6

research
03/18/2022

Enhancement of Novel View Synthesis Using Omnidirectional Image Completion

We present a method for synthesizing novel views from a single 360-degre...
research
04/01/2021

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

We present DietNeRF, a 3D neural scene representation estimated from a f...
research
12/02/2021

Zero-Shot Text-Guided Object Generation with Dream Fields

We combine neural rendering with multi-modal image and text representati...
research
04/02/2022

SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image

Despite the rapid development of Neural Radiance Field (NeRF), the neces...
research
05/11/2023

SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Views

We study to generate novel views of indoor scenes given sparse input vie...
research
06/10/2022

NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors

Though Neural Radiance Field (NeRF) demonstrates compelling novel view s...
research
04/28/2023

ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields

Neural radiance fields (NeRF) have achieved impressive performances in v...

Please sign up or login with your details

Forgot password? Click here to reset