Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision

04/18/2020
by   Haitian Zheng, et al.
0

Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image. In the task, the additional exemplar image provides the style guidance that controls the appearance of the synthesized output. Despite the controllability advantage, the existing models are designed on datasets with specific and roughly aligned objects. In this paper, we tackle a more challenging and general task, where the exemplar is an arbitrary scene image that is semantically different from the given label map. To this end, we first propose a Masked Spatial-Channel Attention (MSCA) module which models the correspondence between two arbitrary scenes via efficient decoupled attention. Next, we propose an end-to-end network for joint global and local feature alignment and synthesis. Finally, we propose a novel self-supervision task to enable training. Experiments on the large-scale and more diverse COCO-stuff dataset show significant improvements over the existing methods. Moreover, our approach provides interpretability and can be readily extended to other content manipulation tasks including style and spatial interpolation or extrapolation.

READ FULL TEXT

page 2

page 7

page 11

page 13

page 14

page 19

page 22

page 23

research
11/27/2019

Example-Guided Scene Image Synthesis using Masked Spatial-Channel Attention and Patch-Based Self-Supervision

Example-guided image synthesis has been recently attempted to synthesize...
research
06/04/2019

Example-Guided Style Consistent Image Synthesis from Semantic Labeling

Example-guided image synthesis aims to synthesize an image from a semant...
research
08/29/2020

Dual Attention GANs for Semantic Image Synthesis

In this paper, we focus on the semantic image synthesis task that aims a...
research
08/20/2023

SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation

Despite significant progress in Text-to-Image (T2I) generative models, e...
research
12/02/2020

Curiosity-driven 3D Scene Structure from Single-image Self-supervision

Previous work has demonstrated learning isolated 3D objects (voxel grids...
research
06/17/2020

Sketch-Guided Scenery Image Outpainting

The outpainting results produced by existing approaches are often too ra...
research
12/29/2021

Learning Inception Attention for Image Synthesis and Image Recognition

Image synthesis and image recognition have witnessed remarkable progress...

Please sign up or login with your details

Forgot password? Click here to reset