APB2Face: Audio-guided face reenactment with auxiliary pose and blink signals

04/30/2020
by   Jiangning Zhang, et al.
4

Audio-guided face reenactment aims at generating photorealistic faces using audio information while maintaining the same facial movement as when speaking to a real person. However, existing methods can not generate vivid face images or only reenact low-resolution faces, which limits the application value. To solve those problems, we propose a novel deep neural network named APB2Face, which consists of GeometryPredictor and FaceReenactor modules. GeometryPredictor uses extra head pose and blink state signals as well as audio to predict the latent landmark geometry information, while FaceReenactor inputs the face landmark image to reenact the photorealistic face. A new dataset AnnVI collected from YouTube is presented to support the approach, and experimental results indicate the superiority of our method than state-of-the-arts, whether in authenticity or controllability.

READ FULL TEXT

page 2

page 4

research
10/25/2020

APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment

Audio-guided face reenactment aims to generate a photorealistic face tha...
research
11/12/2018

Hallucinating very low-resolution and obscured face images

Most of the face hallucination methods are often designed for complete i...
research
05/28/2019

FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment

Recent face reenactment studies have achieved remarkable success either ...
research
09/14/2023

DiffTalker: Co-driven audio-image diffusion for talking faces via intermediate landmarks

Generating realistic talking faces is a complex and widely discussed tas...
research
05/15/2023

Identity-Preserving Talking Face Generation with Landmark and Appearance Priors

Generating talking face videos from audio attracts lots of research inte...
research
05/23/2019

Speech2Face: Learning the Face Behind a Voice

How much can we infer about a person's looks from the way they speak? In...
research
07/07/2022

AV-Gaze: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-Profilic Faces

In challenging real-life conditions such as extreme head-pose, occlusion...

Please sign up or login with your details

Forgot password? Click here to reset