DeepStereo: Learning to Predict New Views from the World's Imagery

06/22/2015
by   John Flynn, et al.
0

Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision, but their use in graphics problems has been limited. In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to traditional approaches which consist of multiple complex stages of processing, each of which require careful tuning and can fail in unexpected ways, our system is trained end-to-end. The pixels from neighboring views of a scene are presented to the network which then directly produces the pixels of the unseen view. The benefits of our approach include generality (we only require posed image sets and can easily apply our method to different domains), and high quality results on traditionally difficult scenes. We believe this is due to the end-to-end nature of our system which is able to plausibly generate pixels according to color, depth, and texture priors learnt automatically from the training data. To verify our method we show that it can convincingly reproduce known test views from nearby imagery. Additionally we show images rendered from novel viewpoints. To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 7

research
11/25/2021

Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

A classical problem in computer vision is to infer a 3D scene representa...
research
01/27/2023

Inter-View Depth Consistency Testing in Depth Difference Subspace

Multiview depth imagery will play a critical role in free-viewpoint tele...
research
05/02/2019

Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines

We present a practical and robust deep learning solution for capturing a...
research
11/17/2017

Depth Assisted Full Resolution Network for Single Image-based View Synthesis

Researches in novel viewpoint synthesis majorly focus on interpolation f...
research
10/08/2019

Sky pixel detection in outdoor imagery using an adaptive algorithm and machine learning

Computer vision techniques allow automated detection of sky pixels in ou...
research
06/21/2021

Moving in a 360 World: Synthesizing Panoramic Parallaxes from a Single Panorama

We present Omnidirectional Neural Radiance Fields (OmniNeRF), the first ...
research
02/08/2018

Hole Filling with Multiple Reference Views in DIBR View Synthesis

Depth-image-based rendering (DIBR) oriented view synthesis has been wide...

Please sign up or login with your details

Forgot password? Click here to reset