Deep Semantic Manipulation of Facial Videos

11/15/2021
by   Girish Kumar Solanki, et al.
4

Editing and manipulating facial features in videos is an interesting and important field of research with a plethora of applications, ranging from movie post-production and visual effects to realistic avatars for video games and virtual assistants. To the best of our knowledge, this paper proposes the first method to perform photorealistic manipulation of facial expressions in videos. Our method supports semantic video manipulation based on neural rendering and 3D-based facial expression modelling. We focus on interactive manipulation of the videos by altering and controlling the facial expressions, achieving promising photorealistic results. The proposed method is based on a disentangled representation and estimation of the 3D facial shape and activity, providing the user with intuitive and easy-to-use control of the facial expressions in the input video. We also introduce a user-friendly, interactive AI tool that processes human-readable semantic labels about the desired emotion manipulations in specific parts of the input video and synthesizes photorealistic manipulated videos. We achieve that by mapping the emotion labels to valence-arousal (VA) values, which in turn are mapped to disentangled 3D facial expressions through an especially designed and trained expression decoder network. The paper presents detailed qualitative and quantitative experiments, which demonstrate the effectiveness of our system and the promising results it achieves. Additional results and videos can be found at the supplementary material (https://github.com/Girish-03/DeepSemManipulation).

READ FULL TEXT

page 1

page 2

page 3

page 4

page 5

page 6

research
12/01/2021

Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos

In this paper, we introduce a novel deep learning method for photo-reali...
research
06/22/2020

Facial Expression Editing with Continuous Emotion Labels

Recently deep generative models have achieved impressive results in the ...
research
05/12/2020

Real-time Facial Expression Recognition "In The Wild” by Disentangling 3D Expression from Identity

Human emotions analysis has been the focus of many studies, especially i...
research
09/16/2021

Invertable Frowns: Video-to-Video Facial Emotion Translation

We present Wav2Lip-Emotion, a video-to-video translation architecture th...
research
07/19/2022

Structure-aware Editable Morphable Model for 3D Facial Detail Animation and Manipulation

Morphable models are essential for the statistical modeling of 3D faces....
research
07/05/2018

PortraitGAN for Flexible Portrait Manipulation

Previous methods have dealt with discrete manipulation of facial attribu...
research
10/12/2022

Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video

We present a novel semantic model for human head defined with neural rad...

Please sign up or login with your details

Forgot password? Click here to reset