A Unified Architecture of Semantic Segmentation and Hierarchical Generative Adversarial Networks for Expression Manipulation

12/08/2021
by   Rumeysa Bodur, et al.
3

Editing facial expressions by only changing what we want is a long-standing research problem in Generative Adversarial Networks (GANs) for image manipulation. Most of the existing methods that rely only on a global generator usually suffer from changing unwanted attributes along with the target attributes. Recently, hierarchical networks that consist of both a global network dealing with the whole image and multiple local networks focusing on local parts are showing success. However, these methods extract local regions by bounding boxes centred around the sparse facial key points which are non-differentiable, inaccurate and unrealistic. Hence, the solution becomes sub-optimal, introduces unwanted artefacts degrading the overall quality of the synthetic images. Moreover, a recent study has shown strong correlation between facial attributes and local semantic regions. To exploit this relationship, we designed a unified architecture of semantic segmentation and hierarchical GANs. A unique advantage of our framework is that on forward pass the semantic segmentation network conditions the generative model, and on backward pass gradients from hierarchical GANs are propagated to the semantic segmentation network, which makes our framework an end-to-end differentiable architecture. This allows both architectures to benefit from each other. To demonstrate its advantages, we evaluate our method on two challenging facial expression translation benchmarks, AffectNet and RaFD, and a semantic segmentation benchmark, CelebAMask-HQ across two popular architectures, BiSeNet and UNet. Our extensive quantitative and qualitative evaluations on both face semantic segmentation and face expression manipulation tasks validate the effectiveness of our work over existing state-of-the-art methods.

READ FULL TEXT

page 1

page 4

page 8

page 11

page 12

page 13

page 14

page 15

research
04/20/2021

Semantic Segmentation by Improved Generative Adversarial Networks

While most existing segmentation methods usually combined the powerful f...
research
01/27/2020

FakeLocator: Robust Localization of GAN-Based Face Manipulations via Semantic Segmentation Networks with Bells and Whistles

Nowadays, full face synthesis and partial face manipulation by virtue of...
research
06/27/2019

Adversarial Pixel-Level Generation of Semantic Images

Generative Adversarial Networks (GANs) have obtained extraordinary succe...
research
10/27/2020

Pixel-based Facial Expression Synthesis

Facial expression synthesis has achieved remarkable advances with the ad...
research
03/09/2018

Cross-View Image Synthesis using Conditional GANs

Learning to generate natural scenes has always been a challenging task i...
research
11/25/2020

Enhanced 3DMM Attribute Control via Synthetic Dataset Creation Pipeline

While facial attribute manipulation of 2D images via Generative Adversar...
research
07/24/2018

GANimation: Anatomically-aware Facial Animation from a Single Image

Recent advances in Generative Adversarial Networks (GANs) have shown imp...

Please sign up or login with your details

Forgot password? Click here to reset