Constraining Multi-scale Pairwise Features between Encoder and Decoder Using Contrastive Learning for Unpaired Image-to-Image Translation

11/20/2022
by   Xiuding Cai, et al.
0

Contrastive learning (CL) has shown great potential in image-to-image translation (I2I). Current CL-based I2I methods usually re-exploit the encoder of the generator to maximize the mutual information between the input and generated images, which does not exert an active effect on the decoder part. In addition, though negative samples play a crucial role in CL, most existing methods adopt a random sampling strategy, which may be less effective. In this paper, we rethink the CL paradigm in the unpaired I2I tasks from two perspectives and propose a new one-sided image translation framework called EnCo. First, we present an explicit constraint on the multi-scale pairwise features between the encoder and decoder of the generator to guarantee the semantic consistency of the input and generated images. Second, we propose a discriminative attention-guided negative sampling strategy to replace the random negative sampling, which significantly improves the performance of the generative model with an almost negligible computational overhead. Compared with existing methods, EnCo acts more effective and efficient. Extensive experiments on several popular I2I datasets demonstrate the effectiveness and advantages of our proposed approach, and we achieve several state-of-the-art compared to previous methods.

READ FULL TEXT

page 1

page 2

page 6

page 10

page 13

page 14

page 15

page 16

research
08/10/2021

Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translation

Contrastive learning shows great potential in unpaired image-to-image tr...
research
04/24/2023

Multi-crop Contrastive Learning for Unsupervised Image-to-Image Translation

Recently, image-to-image translation methods based on contrastive learni...
research
09/17/2019

Multi-mapping Image-to-Image Translation via Learning Disentanglement

Recent advances of image-to-image translation focus on learning the one-...
research
07/30/2020

Contrastive Learning for Unpaired Image-to-Image Translation

In image-to-image translation, each patch in the output should reflect t...
research
07/24/2020

The Surprising Effectiveness of Linear Unsupervised Image-to-Image Translation

Unsupervised image-to-image translation is an inherently ill-posed probl...
research
04/04/2019

UU-Nets Connecting Discriminator and Generator for Image to Image Translation

Adversarial generative model have successfully manifest itself in image ...
research
04/18/2021

Multi-scale Self-calibrated Network for Image Light Source Transfer

Image light source transfer (LLST), as the most challenging task in the ...

Please sign up or login with your details

Forgot password? Click here to reset