Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch

Scene Designer is a novel method for searching and generating images using free-hand sketches of scene compositions; i.e. drawings that describe both the appearance and relative positions of objects. Our core contribution is a single unified model to learn both a cross-modal search embedding for matching sketched compositions to images, and an object embedding for layout synthesis. We show that a graph neural network (GNN) followed by Transformer under our novel contrastive learning setting is required to allow learning correlations between object type, appearance and arrangement, driving a mask generation module that synthesises coherent scene layouts, whilst also delivering state of the art sketch based visual search of scenes.

READ FULL TEXT

page 1

page 2

page 5

page 6

page 8

research
09/11/2019

Specifying Object Attributes and Relations in Interactive Scene Generation

We introduce a method for the generation of images from an input scene g...
research
03/17/2022

CoGS: Controllable Generation and Search from Sketch and Style

We present CoGS, a novel method for the style-conditioned, sketch-driven...
research
06/15/2021

Compositional Sketch Search

We present an algorithm for searching image collections using free-hand ...
research
10/11/2019

Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval

Image-text retrieval of natural scenes has been a popular research topic...
research
03/28/2022

Partially Does It: Towards Scene-Level FG-SBIR with Partial Input

We scrutinise an important observation plaguing scene-level sketch resea...
research
10/09/2015

Free-hand Sketch Synthesis with Deformable Stroke Models

We present a generative model which can automatically summarize the stro...
research
09/02/2022

A Novel Approach for Pill-Prescription Matching with GNN Assistance and Contrastive Learning

Medication mistaking is one of the risks that can result in unpredictabl...

Please sign up or login with your details

Forgot password? Click here to reset