Scene Graph Generation for Better Image Captioning?

09/23/2021
by   Maximilian Mozes, et al.
0

We investigate the incorporation of visual relationships into the task of supervised image caption generation by proposing a model that leverages detected objects and auto-generated visual relationships to describe images in natural language. To do so, we first generate a scene graph from raw image pixels by identifying individual objects and visual relationships between them. This scene graph then serves as input to our graph-to-text model, which generates the final caption. In contrast to previous approaches, our model thus explicitly models the detection of objects and visual relationships in the image. For our experiments we construct a new dataset from the intersection of Visual Genome and MS COCO, consisting of images with both a corresponding gold scene graph and human-authored caption. Our results show that our methods outperform existing state-of-the-art end-to-end models that generate image descriptions directly from raw input pixels when compared in terms of the BLEU and METEOR evaluation metrics.

READ FULL TEXT

page 3

page 5

research
04/04/2018

Image Generation from Scene Graphs

To truly understand the visual world our models should be able not only ...
research
11/21/2018

An Interpretable Model for Scene Graph Generation

We propose an efficient and interpretable scene graph generator. We cons...
research
02/01/2019

Rethinking Visual Relationships for High-level Image Understanding

Relationships, as the bond of isolated entities in images, reflect the i...
research
10/12/2021

Topic Scene Graph Generation by Attention Distillation from Caption

If an image tells a story, the image caption is the briefest narrator. G...
research
06/22/2017

Pixels to Graphs by Associative Embedding

Graphs are a useful abstraction of image content. Not only can graphs re...
research
09/04/2018

Text2Scene: Generating Abstract Scenes from Textual Descriptions

In this paper, we propose an end-to-end model that learns to interpret n...
research
06/12/2019

Visual Relationships as Functions: Enabling Few-Shot Scene Graph Prediction

Scene graph prediction --- classifying the set of objects and predicates...

Please sign up or login with your details

Forgot password? Click here to reset