Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning

08/19/2019
by   Boren Li, et al.
0

Generating semantic layout from scene graph is a crucial intermediate task connecting text to image. We present a conceptually simple, flexible and general framework using sequence to sequence (seq-to-seq) learning for this task. The framework, called Seq-SG2SL, derives sequence proxies for the two modality and a Transformer-based seq-to-seq model learns to transduce one into the other. A scene graph is decomposed into a sequence of semantic fragments (SF), one for each relationship. A semantic layout is represented as the consequence from a series of brick-action code segments (BACS), dictating the position and scale of each object bounding box in the layout. Viewing the two building blocks, SF and BACS, as corresponding terms in two different vocabularies, a seq-to-seq model is fittingly used to translate. A new metric, semantic layout evaluation understudy (SLEU), is devised to evaluate the task of semantic layout prediction inspired by BLEU. SLEU defines relationships within a layout as unigrams and looks at the spatial distribution for n-grams. Unlike the binary precision of BLEU, SLEU allows for some tolerances spatially through thresholding the Jaccard Index and is consequently more adapted to the task. Experimental results on the challenging Visual Genome dataset show improvement over a non-sequential approach based on graph convolution.

READ FULL TEXT

page 6

page 7

research
08/12/2022

Layout-Bridging Text-to-Image Synthesis

The crux of text-to-image synthesis stems from the difficulty of preserv...
research
01/16/2018

Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

We propose a novel hierarchical approach for text-to-image synthesis by ...
research
09/02/2019

Relationship-Aware Spatial Perception Fusion for Realistic Scene Layout Generation

The significant progress on Generative Adversarial Networks (GANs) have ...
research
09/02/2020

Intrinsic Relationship Reasoning for Small Object Detection

The small objects in images and videos are usually not independent indiv...
research
05/05/2022

Scene Graph Expansion for Semantics-Guided Image Outpainting

In this paper, we address the task of semantics-guided image outpainting...
research
09/07/2019

Scene Recognition with Prototype-agnostic Scene Layout

Abstract--- Exploiting the spatial structure in scene images is a key re...
research
07/24/2019

LayoutVAE: Stochastic Scene Layout Generation from a Label Set

Recently there is an increasing interest in scene generation within the ...

Please sign up or login with your details

Forgot password? Click here to reset