Foveated Neural Radiance Fields for Real-Time and Egocentric Virtual Reality

03/30/2021
by   Nianchen Deng, et al.
0

Traditional high-quality 3D graphics requires large volumes of fine-detailed scene data for rendering. This demand compromises computational efficiency and local storage resources. Specifically, it becomes more concerning for future wearable and portable virtual and augmented reality (VR/AR) displays. Recent approaches to combat this problem include remote rendering/streaming and neural representations of 3D assets. These approaches have redefined the traditional local storage-rendering pipeline by distributed computing or compression of large data. However, these methods typically suffer from high latency or low quality for practical visualization of large immersive virtual scenes, notably with extra high resolution and refresh rate requirements for VR applications such as gaming and design. Tailored for the future portable, low-storage, and energy-efficient VR platforms, we present the first gaze-contingent 3D neural representation and view synthesis method. We incorporate the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. Furthermore, we jointly optimize the latency/performance and visual quality, while mutually bridging human perception and neural scene synthesis, to achieve perceptually high-quality immersive interaction. Both objective analysis and subjective study demonstrate the effectiveness of our approach in significantly reducing local storage volume and synthesis latency (up to 99 both data size and computational time), while simultaneously presenting high-fidelity rendering, with perceptual quality identical to that of fully locally stored and rendered high-quality imagery.

READ FULL TEXT

page 1

page 2

page 3

page 5

page 8

page 9

page 10

page 11

research
02/25/2021

Q-VR: System-Level Design for Future Mobile Collaborative Virtual Reality

High Quality Mobile Virtual Reality (VR) is what the incoming graphics t...
research
05/09/2022

Foveated Rendering: Motivation, Taxonomy, and Research Directions

With the recent interest in virtual reality and augmented reality, there...
research
08/28/2021

Towards Retina-Quality VR Video Streaming: 15ms Could Save You 80 Your Bandwidth

Virtual reality systems today cannot yet stream immersive, retina-qualit...
research
11/07/2022

QRF: Implicit Neural Representations with Quantum Radiance Fields

Photorealistic rendering of real-world scenes is a tremendous challenge ...
research
03/10/2021

A Distributed, Decoupled System for Losslessly Streaming Dynamic Light Probes to Thin Clients

We present a networked, high performance graphics system that combines d...
research
08/10/2021

Gaze-Contingent Retinal Speckle Suppression for Perceptually-Matched Foveated Holographic Displays

Computer-generated holographic (CGH) displays show great potential and a...
research
05/03/2021

Thinking Outside the Lab: VR Size Depth Perception in the Wild

Size and distance perception in Virtual Reality (VR) have been widely st...

Please sign up or login with your details

Forgot password? Click here to reset