DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs

11/11/2020
by   Yaxing Wang, et al.
0

Image-to-image translation has recently achieved remarkable results. But despite current success, it suffers from inferior performance when translations between classes require large shape changes. We attribute this to the high-resolution bottlenecks which are used by current state-of-the-art image-to-image methods. Therefore, in this work, we propose a novel deep hierarchical Image-to-Image Translation method, called DeepI2I. We learn a model by leveraging hierarchical features: (a) structural information contained in the shallow layers and (b) semantic information extracted from the deep layers. To enable the training of deep I2I models on small datasets, we propose a novel transfer learning method, that transfers knowledge from pre-trained GANs. Specifically, we leverage the discriminator of a pre-trained GANs (i.e. BigGAN or StyleGAN) to initialize both the encoder and the discriminator and the pre-trained generator to initialize the generator of our model. Applying knowledge transfer leads to an alignment problem between the encoder and generator. We introduce an adaptor network to address this. On many-class image-to-image translation on three datasets (Animal faces, Birds, and Foods) we decrease mFID by at least 35 Furthermore, we qualitatively and quantitatively demonstrate that transfer learning significantly improves the performance of I2I systems, especially for small datasets. Finally, we are the first to perform I2I translations for domains with over 100 classes.

READ FULL TEXT

page 15

page 18

page 19

page 20

page 21

page 22

page 23

page 25

research
06/04/2019

Cross-Domain Cascaded Deep Feature Translation

In recent years we have witnessed tremendous progress in unpaired image-...
research
04/22/2022

SE-GAN: Skeleton Enhanced GAN-based Model for Brush Handwriting Font Generation

Previous works on font generation mainly focus on the standard print fon...
research
07/06/2021

Independent Encoder for Deep Hierarchical Unsupervised Image-to-Image Translation

The main challenges of image-to-image (I2I) translation are to make the ...
research
05/13/2021

TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets

Image-to-image (I2I) translation has matured in recent years and is able...
research
10/08/2019

Semi Few-Shot Attribute Translation

Recent studies have shown remarkable success in image-to-image translati...
research
03/30/2021

SPatchGAN: A Statistical Feature Based Discriminator for Unsupervised Image-to-Image Translation

For unsupervised image-to-image translation, we propose a discriminator ...
research
12/06/2019

Exploring Unlabeled Faces for Novel Attribute Discovery

Despite remarkable success in unpaired image-to-image translation, exist...

Please sign up or login with your details

Forgot password? Click here to reset