DeepAI AI Chat
Log In Sign Up

Generating Triples with Adversarial Networks for Scene Graph Construction

02/07/2018
by   Matthew Klawonn, et al.
Rensselaer Polytechnic Institute
United States Air Force
0

Driven by successes in deep learning, computer vision research has begun to move beyond object detection and image classification to more sophisticated tasks like image captioning or visual question answering. Motivating such endeavors is the desire for models to capture not only objects present in an image, but more fine-grained aspects of a scene such as relationships between objects and their attributes. Scene graphs provide a formal construct for capturing these aspects of an image. Despite this, there have been only a few recent efforts to generate scene graphs from imagery. Previous works limit themselves to settings where bounding box information is available at train time and do not attempt to generate scene graphs with attributes. In this paper we propose a method, based on recent advancements in Generative Adversarial Networks, to overcome these deficiencies. We take the approach of first generating small subgraphs, each describing a single statement about a scene from a specific region of the input image chosen using an attention mechanism. By doing so, our method is able to produce portions of the scene graphs with attribute information without the need for bounding box labels. Then, the complete scene graph is constructed from these subgraphs. We show that our model improves upon prior work in scene graph generation on state-of-the-art data sets and accepted metrics. Further, we demonstrate that our model is capable of handling a larger vocabulary size than prior work has attempted.

READ FULL TEXT

page 1

page 5

page 7

09/25/2020

Are scene graphs good enough to improve Image Captioning?

Many top-performing image captioning models rely solely on object featur...
03/20/2023

Location-Free Scene Graph Generation

Scene Graph Generation (SGG) is a challenging visual understanding task....
08/08/2020

Assisting Scene Graph Generation with Self-Supervision

Research in scene graph generation has quickly gained traction in the pa...
11/30/2022

SGDraw: Scene Graph Drawing Interface Using Object-Oriented Representation

Scene understanding is an essential and challenging task in computer vis...
05/09/2019

Interactive Image Generation Using Scene Graphs

Recent years have witnessed some exciting developments in the domain of ...
07/31/2021

Chest ImaGenome Dataset for Clinical Reasoning

Despite the progress in automatic detection of radiologic findings from ...
11/25/2021

Scene Graph Generation with Geometric Context

Scene Graph Generation has gained much attention in computer vision rese...