Mesh Guided One-shot Face Reenactment using Graph Convolutional Networks

08/18/2020
by   Guangming Yao, et al.
10

Face reenactment aims to animate a source face image to a different pose and expression provided by a driving image. Existing approaches are either designed for a specific identity, or suffer from the identity preservation problem in the one-shot or few-shot scenarios. In this paper, we introduce a method for one-shot face reenactment, which uses the reconstructed 3D meshes (i.e., the source mesh and driving mesh) as guidance to learn the optical flow needed for the reenacted face synthesis. Technically, we explicitly exclude the driving face's identity information in the reconstructed driving mesh. In this way, our network can focus on the motion estimation for the source face without the interference of driving face shape. We propose a motion net to learn the face motion, which is an asymmetric autoencoder. The encoder is a graph convolutional network (GCN) that learns a latent motion vector from the meshes, and the decoder serves to produce an optical flow image from the latent vector with CNNs. Compared to previous methods using sparse keypoints to guide the optical flow learning, our motion net learns the optical flow directly from 3D dense meshes, which provide the detailed shape and pose information for the optical flow, so it can achieve more accurate expression and pose on the reenacted face. Extensive experiments show that our method can generate high-quality results and outperforms state-of-the-art methods in both qualitative and quantitative comparisons.

READ FULL TEXT

page 1

page 3

page 6

page 8

04/07/2021

Single Source One Shot Reenactment using Weighted motion From Paired Feature Points

Image reenactment is a task where the target object in the source image ...
05/04/2021

Self-Supervised Approach for Facial Movement Based Optical Flow

Computing optical flow is a fundamental problem in computer vision. Howe...
03/27/2022

Thin-Plate Spline Motion Model for Image Animation

Image animation brings life to the static object in the source image acc...
07/19/2017

DenseNet for Dense Flow

Classical approaches for estimating optical flow have achieved rapid pro...
07/02/2021

Deep Mesh Prior: Unsupervised Mesh Restoration using Graph Convolutional Networks

This paper addresses mesh restoration problems, i.e., denoising and comp...
05/02/2022

Emotion-Controllable Generalized Talking Face Generation

Despite the significant progress in recent years, very few of the AI-bas...
08/05/2019

One-shot Face Reenactment

To enable realistic shape (e.g. pose and expression) transfer, existing ...