Diffusion-based Image Translation using Disentangled Style and Content Representation

09/30/2022
by   Gihyun Kwon, et al.
0

Diffusion-based image translation guided by semantic texts or a single target image has enabled flexible style transfer which is not limited to the specific domains. Unfortunately, due to the stochastic nature of diffusion models, it is often difficult to maintain the original content of the image during the reverse diffusion. To address this, here we present a novel diffusion-based unsupervised image translation method using disentangled style and content representation. Specifically, inspired by the splicing Vision Transformer, we extract intermediate keys of multihead self attention layer from ViT model and used them as the content preservation loss. Then, an image guided style transfer is performed by matching the [CLS] classification token from the denoised samples and target image, whereas additional CLIP loss is used for the text-driven style transfer. To further accelerate the semantic change during the reverse diffusion, we also propose a novel semantic divergence loss and resampling strategy. Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks.

READ FULL TEXT

page 1

page 4

page 7

page 9

page 17

page 19

page 20

research
06/07/2023

Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance

Diffusion models have shown significant progress in image translation ta...
research
02/14/2023

DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models

Image-based fashion design with AI techniques has attracted increasing a...
research
07/06/2022

DCT-Net: Domain-Calibrated Translation for Portrait Stylization

This paper introduces DCT-Net, a novel image translation architecture fo...
research
11/19/2022

DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization

Despite the impressive results of arbitrary image-guided style transfer ...
research
03/15/2023

Class-Guided Image-to-Image Diffusion: Cell Painting from Brightfield Images with Class Labels

Image-to-image reconstruction problems with free or inexpensive metadata...
research
09/22/2022

MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation

We present a novel method for exemplar-based image translation, called m...
research
11/20/2021

Delving into Rectifiers in Style-Based Image Translation

While modern image translation techniques can create photorealistic synt...

Please sign up or login with your details

Forgot password? Click here to reset