DeepAI AI Chat
Log In Sign Up

Driving-Signal Aware Full-Body Avatars

by   Timur Bagautdinov, et al.

We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driving signals, such as human pose and facial keypoints, and produces a high-quality representation of human geometry and view-dependent appearance. The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation. To this end, we explicitly account for information deficiency in the driving signal by introducing a latent space that exclusively captures the remaining information, thus enabling the imputation of the missing factors required during full-body animation, while remaining faithful to the driving signal. We also propose a learnable localized compression for the driving signal which promotes better generalization, and helps minimize the influence of global chance-correlations often found in real datasets. For a given driving signal, the resulting variational model produces a compact space of uncertainty for missing factors that allows for an imputation strategy best suited to a particular application. We demonstrate the efficacy of our approach on the challenging problem of full-body animation for virtual telepresence with driving signals acquired from minimal sensors placed in the environment and mounted on a VR-headset.


page 1

page 6

page 9

page 10

page 11

page 13

page 14

page 15


Garment Avatars: Realistic Cloth Driving using Pattern Registration

Virtual telepresence is the future of online communication. Clothing is ...

Drivable Volumetric Avatars using Texel-Aligned Features

Photorealistic telepresence requires both high-fidelity body modeling an...

DANBO: Disentangled Articulated Neural Body Representations via Graph Neural Networks

Deep learning greatly improved the realism of animatable human models by...

FLAG: Flow-based 3D Avatar Generation from Sparse Observations

To represent people in mixed reality applications for collaboration and ...

NeuWigs: A Neural Dynamic Model for Volumetric Hair Capture and Animation

The capture and animation of human hair are two of the major challenges ...

Auto-CARD: Efficient and Robust Codec Avatar Driving for Real-time Mobile Telepresence

Real-time and robust photorealistic avatars for telepresence in AR/VR ha...

HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving

Autonomous driving is an exciting new industry, posing important researc...