DeepAI AI Chat
Log In Sign Up

MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation

09/22/2022
by   Junyoung Seo, et al.
0

We present a novel method for exemplar-based image translation, called matching interleaved diffusion models (MIDMs). Most existing methods for this task were formulated as GAN-based matching-then-generation framework. However, in this framework, matching errors induced by the difficulty of semantic matching across cross-domain, e.g., sketch and photo, can be easily propagated to the generation step, which in turn leads to degenerated results. Motivated by the recent success of diffusion models overcoming the shortcomings of GANs, we incorporate the diffusion models to overcome these limitations. Specifically, we formulate a diffusion-based matching-and-generation framework that interleaves cross-domain matching and diffusion steps in the latent space by iteratively feeding the intermediate warp into the noising process and denoising it to generate a translated image. In addition, to improve the reliability of the diffusion process, we design a confidence-aware process using cycle-consistency to consider only confident regions during translation. Experimental results show that our MIDMs generate more plausible images than state-of-the-art methods.

READ FULL TEXT

page 5

page 6

page 12

page 14

page 15

page 16

page 17

page 18

04/12/2020

Cross-domain Correspondence Learning for Exemplar-based Image Translation

We present a general framework for exemplar-based image translation, whi...
10/06/2021

DiffusionCLIP: Text-guided Image Manipulation Using Diffusion Models

Diffusion models are recent generative models that have shown great succ...
02/03/2023

CFFT-GAN: Cross-domain Feature Fusion Transformer for Exemplar-based Image Translation

Exemplar-based image translation refers to the task of generating images...
09/30/2022

Diffusion-based Image Translation using Disentangled Style and Content Representation

Diffusion-based image translation guided by semantic texts or a single t...
02/20/2023

Cross-domain Compositing with Pretrained Diffusion Models

Diffusion models have enabled high-quality, conditional image editing ca...
01/14/2019

XNet: GAN Latent Space Constraints

Recent GAN-based architectures have been able to deliver impressive perf...
05/22/2020

Image Translation by Latent Union of Subspaces for Cross-Domain Plaque Detection

Calcified plaque in the aorta and pelvic arteries is associated with cor...