Image Processing Using Multi-Code GAN Prior

12/15/2019
by   Jinjin Gu, et al.
51

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, applying trained GAN models to real image processing remains challenging. Because the generator in GANs typically maps the latent space to the image space, there leaves no space for it to take a real image as the input. To make a trained GAN handle real images, existing methods attempt to invert a target image back to the latent space either by back-propagation or by learning an additional encoder. However, the reconstructions from both of the methods are far from ideal. In this work, we propose a new inversion approach to incorporate the well-trained GANs as effective prior to a variety of image processing tasks. In particular, to invert a given GAN model, we employ multiple latent codes to generate multiple feature maps at some intermediate layer of the generator, then compose them with adaptive channel importance to output the final image. Such an over-parameterization of the latent space significantly improves the image reconstruction quality, outperforming existing GAN inversion methods. The resulting high-fidelity image reconstruction enables the trained GAN models as prior to many real-world applications, such as image colorization, super-resolution, image inpainting, and semantic manipulation. We further analyze the properties of the layer-wise representation learned by GAN models and shed light on what knowledge each layer is capable of representing.

READ FULL TEXT

page 7

page 12

page 14

page 15

page 16

page 17

page 18

page 19

research
03/31/2020

In-Domain GAN Inversion for Real Image Editing

Recent work has shown that a variety of controllable semantics emerges i...
research
10/07/2021

StyleGAN-induced data-driven regularization for inverse problems

Recent advances in generative adversarial networks (GANs) have opened up...
research
03/23/2023

TriPlaneNet: An Encoder for EG3D Inversion

Recent progress in NeRF-based GANs has introduced a number of approaches...
research
03/21/2022

High-fidelity GAN Inversion with Padding Space

Inverting a Generative Adversarial Network (GAN) facilitates a wide rang...
research
08/23/2021

Adaptable GAN Encoders for Image Reconstruction via Multi-type Latent Vectors with Two-scale Attentions

Although current deep generative adversarial networks (GANs) could synth...
research
06/02/2021

Barbershop: GAN-based Image Compositing using Segmentation Masks

Seamlessly blending features from multiple images is extremely challengi...
research
08/25/2022

High-Fidelity Image Inpainting with GAN Inversion

Image inpainting seeks a semantically consistent way to recover the corr...

Please sign up or login with your details

Forgot password? Click here to reset