DeepAI AI Chat
Log In Sign Up

NeuralPassthrough: Learned Real-Time View Synthesis for VR

by   Lei Xiao, et al.

Virtual reality (VR) headsets provide an immersive, stereoscopic visual experience, but at the cost of blocking users from directly observing their physical environment. Passthrough techniques are intended to address this limitation by leveraging outward-facing cameras to reconstruct the images that would otherwise be seen by the user without the headset. This is inherently a real-time view synthesis challenge, since passthrough cameras cannot be physically co-located with the eyes. Existing passthrough techniques suffer from distracting reconstruction artifacts, largely due to the lack of accurate depth information (especially for near-field and disoccluded objects), and also exhibit limited image quality (e.g., being low resolution and monochromatic). In this paper, we propose the first learned passthrough method and assess its performance using a custom VR headset that contains a stereo pair of RGB cameras. Through both simulations and experiments, we demonstrate that our learned passthrough method delivers superior image quality compared to state-of-the-art methods, while meeting strict VR requirements for real-time, perspective-correct stereoscopic view synthesis over a wide field of view for desktop-connected headsets.


page 1

page 2

page 4

page 6

page 7


A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality

We present a system for acquiring, processing, and rendering panoramic l...

MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

We introduce a method to convert stereo 360 (omnidirectional stereo) ima...

Deep Billboards towards Lossless Real2Sim in Virtual Reality

An aspirational goal for virtual reality (VR) is to bring in a rich dive...

Stereo Unstructured Magnification: Multiple Homography Image for View Synthesis

This paper studies the problem of view synthesis with certain amount of ...

BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning

Providing omnidirectional depth along with RGB information is important ...

360^∘ Stereo Image Composition with Depth Adaption

360^∘ images and videos have become an economic and popular way to provi...