DeepAI AI Chat
Log In Sign Up

Person-in-Context Synthesiswith Compositional Structural Space

by   Weidong Yin, et al.

Despite significant progress, controlled generation of complex images with interacting people remains difficult. Existing layout generation methods fall short of synthesizing realistic person instances; while pose-guided generation approaches focus on a single person and assume simple or known backgrounds. To tackle these limitations, we propose a new problem, Persons in Context Synthesis, which aims to synthesize diverse person instance(s) in consistent contexts, with user control over both. The context is specified by the bounding box object layout which lacks shape information, while pose of the person(s) by keypoints which are sparsely annotated. To handle the stark difference in input structures, we proposed two separate neural branches to attentively composite the respective (context/person) inputs into shared “compositional structural space”, which encodes shape, location and appearance information for both context and person structures in a disentangled manner. This structural space is then decoded to the image space using multi-level feature modulation strategy, and learned in a self supervised manner from image collections and their corresponding inputs. Extensive experiments on two large-scale datasets (COCO-Stuff <cit.> and Visual Genome <cit.>) demonstrate that our framework outperforms state-of-the-art methods w.r.t. synthesis quality.


page 2

page 4

page 6

page 8

page 10

page 11

page 12


Attribute-guided image generation from layout

Recent approaches have achieved great success in image generation from s...

Image Generation from Layout

Despite significant recent progress on generative models, controlled gen...

MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generation

Pose-guided person image generation usually involves using paired source...

Disentangled Cycle Consistency for Highly-realistic Virtual Try-On

Image virtual try-on replaces the clothes on a person image with a desir...

Pose-Guided Human Animation from a Single Image in the Wild

We present a new pose transfer method for synthesizing a human animation...

Interactive Image Synthesis with Panoptic Layout Generation

Interactive image synthesis from user-guided input is a challenging task...