3D to 4D Facial Expressions Generation Guided by Landmarks

05/16/2021
by   Naima Otberdout, et al.
11

While deep learning-based 3D face generation has made a progress recently, the problem of dynamic 3D (4D) facial expression synthesis is less investigated. In this paper, we propose a novel solution to the following question: given one input 3D neutral face, can we generate dynamic 3D (4D) facial expressions from it? To tackle this problem, we first propose a mesh encoder-decoder architecture (Expr-ED) that exploits a set of 3D landmarks to generate an expressive 3D face from its neutral counterpart. Then, we extend it to 4D by modeling the temporal dynamics of facial expressions using a manifold-valued GAN capable of generating a sequence of 3D landmarks from an expression label (Motion3DGAN). The generated landmarks are fed into the mesh encoder-decoder, ultimately producing a sequence of 3D expressive faces. By decoupling the two steps, we separately address the non-linearity induced by the mesh deformation and motion dynamics. The experimental results on the CoMA dataset show that our mesh encoder-decoder guided by landmarks brings a significant improvement with respect to other landmark-based 3D fitting approaches, and that we can generate high quality dynamic facial expressions. This framework further enables the 3D expression intensity to be continuously adapted from low to high intensity. Finally, we show our framework can be applied to other tasks, such as 2D-3D facial expression transfer.

READ FULL TEXT

page 6

page 7

page 8

page 9

page 10

research
03/29/2023

4D Facial Expression Diffusion Model

Facial expression generation is one of the most challenging and long-sou...
research
07/29/2022

Generating Complex 4D Expression Transitions by Learning Face Landmark Trajectories

In this work, we address the problem of 4D facial expressions generation...
research
02/06/2018

Every Smile is Unique: Landmark-Guided Diverse Smile Generation

Each smile is unique: one person surely smiles in different ways (e.g., ...
research
07/19/2020

Learning to Generate Customized Dynamic 3D Facial Expressions

Recent advances in deep learning have significantly pushed the state-of-...
research
09/05/2022

Facial Expression Translation using Landmark Guided GANs

We propose a simple yet powerful Landmark guided Generative Adversarial ...
research
04/13/2022

Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions

Recently, talking-face video generation has received considerable attent...
research
10/12/2020

Fat Pad Cages for Facial Posing

We introduce Fat Pad cages for posing facial meshes. It combines cage re...

Please sign up or login with your details

Forgot password? Click here to reset