Realistic Face Reenactment via Self-Supervised Disentangling of Identity and Pose

03/29/2020
by   Xianfang Zeng, et al.
0

Recent works have shown how realistic talking face images can be obtained under the supervision of geometry guidance, e.g., facial landmark or boundary. To alleviate the demand for manual annotations, in this paper, we propose a novel self-supervised hybrid model (DAE-GAN) that learns how to reenact face naturally given large amounts of unlabeled videos. Our approach combines two deforming autoencoders with the latest advances in the conditional generation. On the one hand, we adopt the deforming autoencoder to disentangle identity and pose representations. A strong prior in talking face videos is that each frame can be encoded as two parts: one for video-specific identity and the other for various poses. Inspired by that, we utilize a multi-frame deforming autoencoder to learn a pose-invariant embedded face for each video. Meanwhile, a multi-scale deforming autoencoder is proposed to extract pose-related information for each frame. On the other hand, the conditional generator allows for enhancing fine details and overall reality. It leverages the disentangled features to generate photo-realistic and pose-alike face images. We evaluate our model on VoxCeleb1 and RaFD dataset. Experiment results demonstrate the superior quality of reenacted images and the flexibility of transferring facial movements between identities.

READ FULL TEXT

page 1

page 3

page 4

page 6

page 7

research
05/28/2019

FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment

Recent face reenactment studies have achieved remarkable success either ...
research
05/15/2023

Identity-Preserving Talking Face Generation with Landmark and Appearance Priors

Generating talking face videos from audio attracts lots of research inte...
research
10/09/2020

Learning 3D Face Reconstruction with a Pose Guidance Network

We present a self-supervised learning approach to learning monocular 3D ...
research
10/20/2019

LinesToFacePhoto: Face Photo Generation from Lines with Conditional Self-Attention Generative Adversarial Network

In this paper, we explore the task of generating photo-realistic face im...
research
08/26/2019

Learning Disentangled Representations via Independent Subspaces

Image generating neural networks are mostly viewed as black boxes, where...
research
03/19/2022

ALAP-AE: As-Lite-as-Possible Auto-Encoder

We present a novel algorithm to reduce tensor compute required by a cond...
research
09/10/2023

MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment

We present a novel end-to-end identity-agnostic face reenactment system,...

Please sign up or login with your details

Forgot password? Click here to reset