LitAR: Visually Coherent Lighting for Mobile Augmented Reality

01/15/2023
by   Yiqin Zhao, et al.
0

An accurate understanding of omnidirectional environment lighting is crucial for high-quality virtual object rendering in mobile augmented reality (AR). In particular, to support reflective rendering, existing methods have leveraged deep learning models to estimate or have used physical light probes to capture physical lighting, typically represented in the form of an environment map. However, these methods often fail to provide visually coherent details or require additional setups. For example, the commercial framework ARKit uses a convolutional neural network that can generate realistic environment maps; however the corresponding reflective rendering might not match the physical environments. In this work, we present the design and implementation of a lighting reconstruction framework called LitAR that enables realistic and visually-coherent rendering. LitAR addresses several challenges of supporting lighting information for mobile AR. First, to address the spatial variance problem, LitAR uses two-field lighting reconstruction to divide the lighting reconstruction task into the spatial variance-aware near-field reconstruction and the directional-aware far-field reconstruction. The corresponding environment map allows reflective rendering with correct color tones. Second, LitAR uses two noise-tolerant data capturing policies to ensure data quality, namely guided bootstrapped movement and motion-based automatic capturing. Third, to handle the mismatch between the mobile computation capability and the high computation requirement of lighting reconstruction, LitAR employs two novel real-time environment map rendering techniques called multi-resolution projection and anchor extrapolation. These two techniques effectively remove the need of time-consuming mesh reconstruction while maintaining visual quality.

READ FULL TEXT

page 1

page 7

page 13

page 16

page 19

page 21

page 23

page 24

research
01/15/2023

Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality

Lighting understanding plays an important role in virtual object composi...
research
05/30/2021

Xihe: A 3D Vision-based Lighting Estimation Framework for Mobile Augmented Reality

Omnidirectional lighting provides the foundation for achieving spatially...
research
09/21/2018

Dynamic Environment Mapping for Augmented Reality Applications on Mobile Devices

Augmented Reality is a topic of foremost interest nowadays. Its main goa...
research
07/07/2022

Privacy-preserving Reflection Rendering for Augmented Reality

Many augmented reality (AR) applications rely on omnidirectional environ...
research
11/12/2018

LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering

Motivated by augmented and virtual reality applications such as telepres...
research
09/17/2018

Radiative Transport Based Flame Volume Reconstruction from Videos

We introduce a novel approach for flame volume reconstruction from video...
research
03/30/2020

PointAR: Efficient Lighting Estimation for Mobile Augmented Reality

We propose an efficient lighting estimation pipeline that is suitable to...

Please sign up or login with your details

Forgot password? Click here to reset