PIE: Portrait Image Embedding for Semantic Control

09/20/2020
by   Ayush Tewari, et al.
1

Editing of portrait images is a very popular and important research topic with a large variety of applications. For ease of use, control should be provided via a semantically meaningful parameterization that is akin to computer animation controls. The vast majority of existing techniques do not provide such intuitive and fine-grained control, or only enable coarse editing of a single isolated control parameter. Very recently, high-quality semantically controlled editing has been demonstrated, however only on synthetically created StyleGAN images. We present the first approach for embedding real portrait images in the latent space of StyleGAN, which allows for intuitive editing of the head pose, facial expression, and scene illumination in the image. Semantic editing in parameter space is achieved based on StyleRig, a pretrained neural network that maps the control space of a 3D morphable face model to the latent space of the GAN. We design a novel hierarchical non-linear optimization problem to obtain the embedding. An identity preservation energy term allows spatially coherent edits while maintaining facial integrity. Our approach runs at interactive frame rates and thus allows the user to explore the space of possible edits. We evaluate our approach on a wide set of portrait photos, compare it to the current state of the art, and validate the effectiveness of its components in an ablation study.

READ FULL TEXT

page 1

page 5

page 7

page 8

page 9

page 11

page 12

research
11/30/2016

Semantic Facial Expression Editing using Autoencoded Flow

High-level manipulation of facial expressions in images --- such as chan...
research
03/13/2021

PhotoApp: Photorealistic Appearance Editing of Head Portraits

Photorealistic editing of portraits is a challenging task as humans are ...
research
05/27/2022

Video2StyleGAN: Disentangling Local and Global Variations in a Video

Image editing using a pretrained StyleGAN generator has emerged as a pow...
research
05/24/2022

Hierarchical Vectorization for Portrait Images

Aiming at developing intuitive and easy-to-use portrait editing tools, w...
research
09/17/2021

PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering

Generating portrait images by controlling the motions of existing faces ...
research
05/12/2022

Tensor-based Emotion Editing in the StyleGAN Latent Space

In this paper, we use a tensor model based on the Higher-Order Singular ...
research
12/13/2020

Improved StyleGAN Embedding: Where are the Good Latents?

StyleGAN is able to produce photorealistic images that are almost indist...

Please sign up or login with your details

Forgot password? Click here to reset