DeepAI AI Chat
Log In Sign Up

Comprehensive Image Captioning via Scene Graph Decomposition

by   Yiwu Zhong, et al.

We address the challenging problem of image captioning by revisiting the representation of image scene graph. At the core of our method lies the decomposition of a scene graph into a set of sub-graphs, with each sub-graph capturing a semantic component of the input image. We design a deep model to select important sub-graphs, and to decode each selected sub-graph into a single target sentence. By using sub-graphs, our model is able to attend to different components of the image. Our method thus accounts for accurate, diverse, grounded and controllable captioning at the same time. We present extensive experiments to demonstrate the benefits of our comprehensive captioning model. Our method establishes new state-of-the-art results in caption diversity, grounding, and controllability, and compares favourably to latest methods in caption quality. Our project website can be found at yiwuzhong/Sub-GC.html.


page 13

page 21

page 22

page 23

page 24


Are scene graphs good enough to improve Image Captioning?

Many top-performing image captioning models rely solely on object featur...

Consensus Graph Representation Learning for Better Grounded Image Captioning

The contemporary visual captioning models frequently hallucinate objects...

SG2Caps: Revisiting Scene Graphs for Image Captioning

The mainstream image captioning models rely on Convolutional Neural Netw...

Unpaired Image Captioning via Scene Graph Alignments

Deep neural networks have achieved great success on the image captioning...

ReFormer: The Relational Transformer for Image Captioning

Image captioning is shown to be able to achieve a better performance by ...

Controllable Image Captioning via Prompting

Despite the remarkable progress of image captioning, existing captioners...

Decomposed Generation Networks with Structure Prediction for Recipe Generation from Food Images

Recipe generation from food images and ingredients is a challenging task...