DeepAI
Log In Sign Up

Generating Holistic 3D Scene Abstractions for Text-based Image Retrieval

11/28/2016
by   Ang Li, et al.
0

Spatial relationships between objects provide important information for text-based image retrieval. As users are more likely to describe a scene from a real world perspective, using 3D spatial relationships rather than 2D relationships that assume a particular viewing direction, one of the main challenges is to infer the 3D structure that bridges images with users' text descriptions. However, direct inference of 3D structure from images requires learning from large scale annotated data. Since interactions between objects can be reduced to a limited set of atomic spatial relations in 3D, we study the possibility of inferring 3D structure from a text description rather than an image, applying physical relation models to synthesize holistic 3D abstract object layouts satisfying the spatial constraints present in a textual description. We present a generic framework for retrieving images from a textual description of a scene by matching images with these generated abstract object layouts. Images are ranked by matching object detection outputs (bounding boxes) to 2D layout candidates (also represented by bounding boxes) which are obtained by projecting the 3D scenes with sampled camera directions. We validate our approach using public indoor scene datasets and show that our method outperforms baselines built upon object occurrence histograms and learned 2D pairwise relations.

READ FULL TEXT

page 8

page 10

page 11

page 12

08/05/2019

Visual-Relation Conscious Image Generation from Structured-Text

Generating realistic images from text descriptions is a challenging prob...
09/26/2017

Region-Based Image Retrieval Revisited

Region-based image retrieval (RBIR) technique is revisited. In early att...
08/27/2018

Single Shot Scene Text Retrieval

Textual information found in scene images provides high level semantic i...
02/01/2021

Inferring spatial relations from textual descriptions of images

Generating an image from its textual description requires both a certain...
07/28/2021

Similarity and symmetry measures based on fuzzy descriptors of image objects` composition

The paper describes a method for measuring the similarity and symmetry o...
07/22/2022

Panoptic Scene Graph Generation

Existing research addresses scene graph generation (SGG) – a critical te...
03/14/2018

Approximate Query Matching for Image Retrieval

Traditional image recognition involves identifying the key object in a p...