NeuralPassthrough: Learned Real-Time View Synthesis for VR

07/05/2022
by   Lei Xiao, et al.
0

Virtual reality (VR) headsets provide an immersive, stereoscopic visual experience, but at the cost of blocking users from directly observing their physical environment. Passthrough techniques are intended to address this limitation by leveraging outward-facing cameras to reconstruct the images that would otherwise be seen by the user without the headset. This is inherently a real-time view synthesis challenge, since passthrough cameras cannot be physically co-located with the eyes. Existing passthrough techniques suffer from distracting reconstruction artifacts, largely due to the lack of accurate depth information (especially for near-field and disoccluded objects), and also exhibit limited image quality (e.g., being low resolution and monochromatic). In this paper, we propose the first learned passthrough method and assess its performance using a custom VR headset that contains a stereo pair of RGB cameras. Through both simulations and experiments, we demonstrate that our learned passthrough method delivers superior image quality compared to state-of-the-art methods, while meeting strict VR requirements for real-time, perspective-correct stereoscopic view synthesis over a wide field of view for desktop-connected headsets.

READ FULL TEXT

page 1

page 2

page 4

page 6

page 7

research
10/20/2018

A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality

We present a system for acquiring, processing, and rendering panoramic l...
research
01/04/2023

Towards a Pipeline for Real-Time Visualization of Faces for VR-based Telepresence and Live Broadcasting Utilizing Neural Rendering

While head-mounted displays (HMDs) for Virtual Reality (VR) have become ...
research
08/14/2020

MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

We introduce a method to convert stereo 360 (omnidirectional stereo) ima...
research
08/08/2022

Deep Billboards towards Lossless Real2Sim in Virtual Reality

An aspirational goal for virtual reality (VR) is to bring in a rich dive...
research
04/01/2022

Stereo Unstructured Magnification: Multiple Homography Image for View Synthesis

This paper studies the problem of view synthesis with certain amount of ...
research
12/19/2020

Digital Reconstruction of Elmina Castle for Mobile Virtual Reality via Point-based Detail Transfer

Reconstructing 3D models from large, dense point clouds is critical to e...
research
07/12/2021

Monoscopic vs. Stereoscopic Views and Display Types in the Teleoperation of Unmanned Ground Vehicles for Object Avoidance

Virtual reality (VR) head-mounted displays (HMD) have recently been used...

Please sign up or login with your details

Forgot password? Click here to reset