DeepAI AI Chat
Log In Sign Up

Pixel Codec Avatars

by   Shugao Ma, et al.

Telecommunication with photorealistic avatars in virtual or augmented reality is a promising path for achieving authentic face-to-face communication in 3D over remote physical distances. In this work, we present the Pixel Codec Avatars (PiCA): a deep generative model of 3D human faces that achieves state of the art reconstruction performance while being computationally efficient and adaptive to the rendering conditions during execution. Our model combines two core ideas: (1) a fully convolutional architecture for decoding spatially varying features, and (2) a rendering-adaptive per-pixel decoder. Both techniques are integrated via a dense surface representation that is learned in a weakly-supervised manner from low-topology mesh tracking over training images. We demonstrate that PiCA improves reconstruction over existing techniques across testing expressions and views on persons of different gender and skin tone. Importantly, we show that the PiCA model is much smaller than the state-of-art baseline model, and makes multi-person telecommunicaiton possible: on a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.


page 1

page 4

page 6

page 8


Pixel-Face: A Large-Scale, High-Resolution Benchmark for 3D Face Reconstruction

3D face reconstruction is a fundamental task that can facilitate numerou...

FastMESH: Fast Surface Reconstruction by Hexagonal Mesh-based Neural Rendering

Despite the promising results of multi-view reconstruction, the recent n...

Generating 3D faces using Convolutional Mesh Autoencoders

Learned 3D representations of human faces are useful for computer vision...

Weakly-supervised Single-view Image Relighting

We present a learning-based approach to relight a single image of Lamber...

Privacy-preserving Reflection Rendering for Augmented Reality

Many augmented reality (AR) applications rely on omnidirectional environ...