Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning

09/30/2020
by   Xiangxi Shi, et al.
0

Change Captioning is a task that aims to describe the difference between images with natural language. Most existing methods treat this problem as a difference judgment without the existence of distractors, such as viewpoint changes. However, in practice, viewpoint changes happen often and can overwhelm the semantic difference to be described. In this paper, we propose a novel visual encoder to explicitly distinguish viewpoint changes from semantic changes in the change captioning task. Moreover, we further simulate the attention preference of humans and propose a novel reinforcement learning process to fine-tune the attention directly with language evaluation rewards. Extensive experimental results show that our method outperforms the state-of-the-art approaches by a large margin in both Spot-the-Diff and CLEVR-Change datasets.

READ FULL TEXT
01/08/2019

Viewpoint Invariant Change Captioning

The ability to detect that something has changed in an environment is va...
10/20/2021

R^3Net:Relation-embedded Representation Reconstruction Network for Change Captioning

Change captioning is to use a natural language sentence to describe the ...
03/25/2021

Describing and Localizing Multiple Changes with Transformers

Change captioning tasks aim to detect changes in image pairs observed be...
10/14/2019

Tell-the-difference: Fine-grained Visual Descriptor via a Discriminating Referee

In this paper, we investigate a novel problem of telling the difference ...
06/01/2022

CLIP4IDC: CLIP for Image Difference Captioning

Image Difference Captioning (IDC) aims at generating sentences to descri...
03/12/2016

Image Captioning with Semantic Attention

Automatically generating a natural language description of an image has ...
08/19/2021

Intercept Graph: An Interactive Radial Visualization for Comparison of State Changes

State change comparison of multiple data items is often necessary in mul...