DeepAI AI Chat
Log In Sign Up

Generating Triples with Adversarial Networks for Scene Graph Construction

by   Matthew Klawonn, et al.
Rensselaer Polytechnic Institute
United States Air Force

Driven by successes in deep learning, computer vision research has begun to move beyond object detection and image classification to more sophisticated tasks like image captioning or visual question answering. Motivating such endeavors is the desire for models to capture not only objects present in an image, but more fine-grained aspects of a scene such as relationships between objects and their attributes. Scene graphs provide a formal construct for capturing these aspects of an image. Despite this, there have been only a few recent efforts to generate scene graphs from imagery. Previous works limit themselves to settings where bounding box information is available at train time and do not attempt to generate scene graphs with attributes. In this paper we propose a method, based on recent advancements in Generative Adversarial Networks, to overcome these deficiencies. We take the approach of first generating small subgraphs, each describing a single statement about a scene from a specific region of the input image chosen using an attention mechanism. By doing so, our method is able to produce portions of the scene graphs with attribute information without the need for bounding box labels. Then, the complete scene graph is constructed from these subgraphs. We show that our model improves upon prior work in scene graph generation on state-of-the-art data sets and accepted metrics. Further, we demonstrate that our model is capable of handling a larger vocabulary size than prior work has attempted.


page 1

page 5

page 7


Are scene graphs good enough to improve Image Captioning?

Many top-performing image captioning models rely solely on object featur...

Location-Free Scene Graph Generation

Scene Graph Generation (SGG) is a challenging visual understanding task....

Assisting Scene Graph Generation with Self-Supervision

Research in scene graph generation has quickly gained traction in the pa...

SGDraw: Scene Graph Drawing Interface Using Object-Oriented Representation

Scene understanding is an essential and challenging task in computer vis...

Interactive Image Generation Using Scene Graphs

Recent years have witnessed some exciting developments in the domain of ...

Chest ImaGenome Dataset for Clinical Reasoning

Despite the progress in automatic detection of radiologic findings from ...

Scene Graph Generation with Geometric Context

Scene Graph Generation has gained much attention in computer vision rese...