Spatially Multi-conditional Image Generation

03/25/2022
by   Ritika Chakraborty, et al.
0

In most scenarios, conditional image generation can be thought of as an inversion of the image understanding process. Since generic image understanding involves the solving of multiple tasks, it is natural to aim at the generation of images via multi-conditioning. However, multi-conditional image generation is a very challenging problem due to the heterogeneity and the sparsity of the (in practice) available conditioning labels. In this work, we propose a novel neural architecture to address the problem of heterogeneity and sparsity of the spatially multi-conditional labels. Our choice of spatial conditioning, such as by semantics and depth, is driven by the promise it holds for better control of the image generation process. The proposed method uses a transformer-like architecture operating pixel-wise, which receives the available labels as input tokens to merge them in a learned homogeneous space of labels. The merged labels are then used for image generation via conditional generative adversarial training. In this process, the sparsity of the labels is handled by simply dropping the input tokens corresponding to the missing labels at the desired locations, thanks to the proposed pixel-wise operating architecture. Our experiments on three benchmark datasets demonstrate the clear superiority of our method over the state-of-the-art and the compared baselines.

READ FULL TEXT

page 8

page 13

page 14

page 17

page 18

page 19

page 20

page 21

research
12/04/2020

MPG: A Multi-ingredient Pizza Image Generator with Conditional StyleGANs

Multilabel conditional image generation is a challenging problem in comp...
research
04/02/2022

PixelFolder: An Efficient Progressive Pixel Synthesis Network for Image Generation

Pixel synthesis is a promising research paradigm for image generation, w...
research
12/16/2020

CompositeTasking: Understanding Images by Spatial Composition of Tasks

We define the concept of CompositeTasking as the fusion of multiple, spa...
research
02/10/2023

MaskSketch: Unpaired Structure-guided Masked Image Generation

Recent conditional image generation methods produce images of remarkable...
research
06/16/2018

The Neural Painter: Multi-Turn Image Generation

In this work we combine two research threads from Vision/ Graphics and N...
research
07/04/2019

Guided Image Generation with Conditional Invertible Neural Networks

In this work, we address the task of natural image generation guided by ...
research
07/11/2019

On the Evaluation of Conditional GANs

Conditional Generative Adversarial Networks (cGANs) are finding increasi...

Please sign up or login with your details

Forgot password? Click here to reset