Multimodal Image Synthesis and Editing: A Survey

by   Fangneng Zhan, et al.

As information exists in various modalities in real world, effective interaction and fusion among multimodal information plays a key role for the creation and perception of multimodal data in computer vision and deep learning research. With superb power in modelling the interaction among multimodal information, multimodal image synthesis and editing have become a hot research topic in recent years. Different from traditional visual guidance which provides explicit clues, multimodal guidance offers intuitive and flexible means in image synthesis and editing. On the other hand, this field is also facing several challenges in alignment of features with inherent modality gaps, synthesis of high-resolution images, faithful evaluation metrics, etc. In this survey, we comprehensively contextualize the advance of the recent multimodal image synthesis & editing and formulate taxonomies according to data modality and model architectures. We start with an introduction to different types of guidance modalities in image synthesis and editing. We then describe multimodal image synthesis and editing approaches extensively with detailed frameworks including Generative Adversarial Networks (GANs), GAN Inversion, Transformers, and other methods such as NeRF and Diffusion models. This is followed by a comprehensive description of benchmark datasets and corresponding evaluation metrics as widely adopted in multimodal image synthesis and editing, as well as detailed comparisons of different synthesis methods with analysis of respective advantages and limitations. Finally, we provide insights into the current research challenges and possible future research directions. A project associated with this survey is available at


page 3

page 6

page 7

page 8

page 9

page 10

page 11

page 12


An Introduction to Image Synthesis with Generative Adversarial Nets

There has been a drastic growth of research in Generative Adversarial Ne...

Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion

Digital art synthesis is receiving increasing attention in the multimedi...

A Survey on 3D-aware Image Synthesis

Recent years have seen remarkable progress in deep learning powered visu...

MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis

Existing multimodal conditional image synthesis (MCIS) methods generate ...

Adversarial Text-to-Image Synthesis: A Review

With the advent of generative adversarial networks, synthesizing images ...

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

Existing conditional image synthesis frameworks generate images based on...

Benchmarking and Analyzing 3D-aware Image Synthesis with a Modularized Codebase

Despite the rapid advance of 3D-aware image synthesis, existing studies ...

Please sign up or login with your details

Forgot password? Click here to reset