Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines

05/02/2019
by   Ben Mildenhall, et al.
6

We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration. Previous approaches either require intractably dense view sampling or provide little to no guidance for how users should sample views of a scene to reliably render high-quality novel views. Instead, we propose an algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. We extend traditional plenoptic sampling theory to derive a bound that specifies precisely how densely users should sample views of a given scene when using our algorithm. In practice, we apply this bound to capture and render views of real world scenes that achieve the perceptual quality of Nyquist rate view sampling while using up to 4000x fewer views. We demonstrate our approach's practicality with an augmented reality smartphone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.

READ FULL TEXT

page 1

page 5

page 6

page 7

page 10

research
05/15/2021

NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field

In this paper, we present an efficient and robust deep learning solution...
research
01/26/2021

LIGHTS: LIGHT Specularity Dataset for specular detection in Multi-view

Specular highlights are commonplace in images, however, methods for dete...
research
05/24/2018

Stereo Magnification: Learning View Synthesis using Multiplane Images

The view synthesis problem--generating novel views of a scene from known...
research
05/01/2018

Adaptive View Planning for Aerial 3D Reconstruction of Complex Scenes

With the proliferation of small aerial vehicles, acquiring close up aeri...
research
06/22/2015

DeepStereo: Learning to Predict New Views from the World's Imagery

Deep networks have recently enjoyed enormous success when applied to rec...
research
11/29/2022

DINER: Depth-aware Image-based NEural Radiance fields

We present Depth-aware Image-based NEural Radiance fields (DINER). Given...
research
09/02/2022

CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations

Recent advances in neural radiance fields (NeRFs) achieve state-of-the-a...

Please sign up or login with your details

Forgot password? Click here to reset