iParaphrasing: Extracting Visually Grounded Paraphrases via an Image

by   Chenhui Chu, et al.
CyberAgent, Inc.
Osaka University

A paraphrase is a restatement of the meaning of a text in other words. Paraphrases have been studied to enhance the performance of many natural language processing tasks. In this paper, we propose a novel task iParaphrasing to extract visually grounded paraphrases (VGPs), which are different phrasal expressions describing the same visual concept in an image. These extracted VGPs have the potential to improve language and image multimodal tasks such as visual question answering and image captioning. How to model the similarity between VGPs is the key of iParaphrasing. We apply various existing methods as well as propose a novel neural network-based method with image attention, and report the results of the first attempt toward iParaphrasing.


page 2

page 6

page 10


VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks

Neural module networks (NMN) have achieved success in image-grounded tas...

Learning the meanings of function words from grounded language using a visual question answering model

Interpreting a seemingly-simple function word like "or", "behind", or "m...

Enabling Robots to Draw and Tell: Towards Visually Grounded Multimodal Description Generation

Socially competent robots should be equipped with the ability to perceiv...

Resolving References to Objects in Photographs using the Words-As-Classifiers Model

A common use of language is to refer to visually present objects. Modell...

A Novel Attention-based Aggregation Function to Combine Vision and Language

The joint understanding of vision and language has been recently gaining...

Understanding Grounded Language Learning Agents

Neural network-based systems can now learn to locate the referents of wo...

Please sign up or login with your details

Forgot password? Click here to reset