MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

08/14/2020
by   Benjamin Attal, et al.
3

We introduce a method to convert stereo 360 (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360 imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.

READ FULL TEXT

page 2

page 4

page 7

page 11

page 12

page 13

page 19

page 22

research
03/10/2021

Learning to compose 6-DoF omnidirectional videos using multi-sphere images

Omnidirectional video is an essential component of Virtual Reality. Alth...
research
06/24/2019

Gaze-Contingent Ocular Parallax Rendering for Virtual Reality

Immersive computer graphics systems strive to generate perceptually real...
research
09/08/2023

Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended Reality

Real-time Stereo Matching is a cornerstone algorithm for many Extended R...
research
07/05/2022

NeuralPassthrough: Learned Real-Time View Synthesis for VR

Virtual reality (VR) headsets provide an immersive, stereoscopic visual ...
research
09/23/2021

Low-Latency Immersive 6D Televisualization with Spherical Rendering

We present a method for real-time stereo scene capture and remote VR vis...
research
01/08/2020

OO-VR: NUMA Friendly Object-Oriented VR Rendering Framework For Future NUMA-Based Multi-GPU Systems

With the strong computation capability, NUMA-based multi-GPU system is a...
research
10/12/2020

Omni-Directional Image Generation from Single Snapshot Image

An omni-directional image (ODI) is the image that has a field of view co...

Please sign up or login with your details

Forgot password? Click here to reset