Defoiling Foiled Image Captions

05/16/2018
by   Pranava Madhyastha, et al.
0

We address the task of detecting foiled image captions, i.e. identifying whether a caption contains a word that has been deliberately replaced by a semantically similar word, thus rendering it inaccurate with respect to the image being described. Solving this problem should in principle require a fine-grained understanding of images to detect linguistically valid perturbations in captions. In such contexts, encoding sufficiently descriptive image information becomes a key challenge. In this paper, we demonstrate that it is possible to solve this task using simple, interpretable yet powerful representations based on explicit object information. Our models achieve state-of-the-art performance on a standard dataset, with scores exceeding those achieved by humans on the task. We also measure the upper-bound performance of our models using gold standard annotations. Our analysis reveals that the simpler model performs well even without image information, suggesting that the dataset contains strong linguistic bias.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/19/2015

Exploring Metaphorical Senses and Word Representations for Identifying Metonyms

A metonym is a word with a figurative meaning, similar to a metaphor. Be...
research
10/25/2018

Engaging Image Captioning Via Personality

Standard image captioning tasks such as COCO and Flickr30k are factual, ...
research
05/03/2017

FOIL it! Find One mismatch between Image and Language caption

In this paper, we aim to understand whether current language and vision ...
research
03/30/2017

Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training

While strong progress has been made in image captioning over the last ye...
research
12/22/2016

Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task

We introduce a new multi-modal task for computer systems, posed as a com...
research
06/20/2019

Informative Image Captioning with External Sources of Information

An image caption should fluently present the essential information in a ...
research
05/22/2023

Evaluating Pragmatic Abilities of Image Captioners on A3DS

Evaluating grounded neural language model performance with respect to pr...

Please sign up or login with your details

Forgot password? Click here to reset