DeepAI AI Chat
Log In Sign Up

Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation

10/20/2022
by   Yu Zhao, et al.
Harbin Institute of Technology
Tianjin University
0

Image-to-text tasks, such as open-ended image captioning and controllable image description, have received extensive attention for decades. Here, we further advance this line of work by presenting Visual Spatial Description (VSD), a new perspective for image-to-text toward spatial semantics. Given an image and two objects inside it, VSD aims to produce one description focusing on the spatial perspective between the two objects. Accordingly, we manually annotate a dataset to facilitate the investigation of the newly-introduced task and build several benchmark encoder-decoder models by using VL-BART and VL-T5 as backbones. In addition, we investigate pipeline and joint end-to-end architectures for incorporating visual spatial relationship classification (VSRC) information into our model. Finally, we conduct experiments on our benchmark dataset to evaluate all our models. Results show that our models are impressive, providing accurate and human-like spatial-oriented text descriptions. Meanwhile, VSRC has great potential for VSD, and the joint end-to-end architecture is the better choice for their integration. We make the dataset and codes public for research purposes.

READ FULL TEXT

page 4

page 13

09/10/2019

Select and Attend: Towards Controllable Content Selection in Text Generation

Many text generation tasks naturally contain two steps: content selectio...
06/14/2019

Image Captioning: Transforming Objects into Words

Image captioning models typically follow an encoder-decoder architecture...
01/28/2020

Incorporating Joint Embeddings into Goal-Oriented Dialogues with Multi-Task Learning

Attention-based encoder-decoder neural network models have recently show...
08/16/2021

AutoChart: A Dataset for Chart-to-Text Generation Task

The analytical description of charts is an exciting and important resear...
04/11/2020

End to End Chinese Lexical Fusion Recognition with Sememe Knowledge

In this paper, we present Chinese lexical fusion recognition, a new task...
04/15/2022

Image Captioning In the Transformer Age

Image Captioning (IC) has achieved astonishing developments by incorpora...