Context-Aware Layout to Image Generation with Enhanced Object Appearance

03/22/2021
by   Sen He, et al.
0

A layout to image (L2I) generation model aims to generate a complicated image containing multiple objects (things) against natural background (stuff), conditioned on a given layout. Built upon the recent advances in generative adversarial networks (GANs), existing L2I models have made great progress. However, a close inspection of their generated images reveals two major limitations: (1) the object-to-object as well as object-to-stuff relations are often broken and (2) each object's appearance is typically distorted lacking the key defining characteristics associated with the object class. We argue that these are caused by the lack of context-aware object and stuff feature encoding in their generators, and location-sensitive appearance representation in their discriminators. To address these limitations, two new modules are proposed in this work. First, a context-aware feature transformation module is introduced in the generator to ensure that the generated feature encoding of either object or stuff is aware of other co-existing objects/stuff in the scene. Second, instead of feeding location-insensitive image features to the discriminator, we use the Gram matrix computed from the feature maps of the generated object images to preserve location-sensitive information, resulting in much enhanced object appearance. Extensive experiments show that the proposed method achieves state-of-the-art performance on the COCO-Thing-Stuff and Visual Genome benchmarks.

READ FULL TEXT

page 1

page 6

page 7

page 8

research
04/01/2021

Exploiting Relationship for Complex-scene Image Generation

The significant progress on Generative Adversarial Networks (GANs) has f...
research
04/01/2021

Text to Image Generation with Semantic-Spatial Aware GAN

A text to image generation (T2I) model aims to generate photo-realistic ...
research
12/09/2019

Learning a Layout Transfer Network for Context Aware Object Detection

We present a context aware object detection method based on a retrieve-a...
research
12/06/2018

Context-Aware Synthesis and Placement of Object Instances

Learning to insert an object instance into an image in a semantically co...
research
12/06/2018

Context-Aware Synthesis and Placement ofObject Instances

Learning to insert an object instance into an image in a semantically co...
research
04/16/2018

IterGANs: Iterative GANs to Learn and Control 3D Object Transformation

We are interested in learning visual representations which allow for 3D ...
research
10/09/2018

Context-Aware Text-Based Binary Image Stylization and Synthesis

In this work, we present a new framework for the stylization of text-bas...

Please sign up or login with your details

Forgot password? Click here to reset