LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering

by   Ricardo Martin-Brualla, et al.

Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real-time. We call this approach neural (re-)rendering, and our live system "LookinGood". Our deep architecture is trained to produce high resolution and high quality images from a coarse rendering in real-time. First, we propose a self-supervised training method that does not require manual ground-truth annotation. We contribute a specialized reconstruction error that uses semantic information to focus on relevant parts of the subject, e.g. the face. We also introduce a salient reweighing scheme of the loss function that is able to discard outliers. We specifically design the system for virtual and augmented reality headsets where the consistency between the left and right eye plays a crucial role in the final user experience. Finally, we generate temporally stable results by explicitly minimizing the difference between two consecutive frames. We tested the proposed system in two different scenarios: one involving a single RGB-D sensor, and upper body reconstruction of an actor, the second consisting of full body 360 degree capture. Through extensive experimentation, we demonstrate how our system generalizes across unseen sequences and subjects. The supplementary video is available at http://youtu.be/Md3tdAKoLGU.


page 1

page 3

page 5

page 9

page 10

page 11

page 12

page 13


Foveated Rendering: Motivation, Taxonomy, and Research Directions

With the recent interest in virtual reality and augmented reality, there...

LitAR: Visually Coherent Lighting for Mobile Augmented Reality

An accurate understanding of omnidirectional environment lighting is cru...

Deep Appearance Models for Face Rendering

We introduce a deep appearance model for rendering the human face. Inspi...

Development and Evaluation of a Learning-based Model for Real-time Haptic Texture Rendering

Current Virtual Reality (VR) environments lack the rich haptic signals t...

Super-Resolution Appearance Transfer for 4D Human Performances

A common problem in the 4D reconstruction of people from multi-view vide...

Radiative Transport Based Flame Volume Reconstruction from Videos

We introduce a novel approach for flame volume reconstruction from video...

Photo2Relief: Let Human in the Photograph Stand Out

In this paper, we propose a technique for making humans in photographs p...

Please sign up or login with your details

Forgot password? Click here to reset