Compositional Generalization in Image Captioning

09/10/2019
by   Mitja Nikolaus, et al.
0

Image captioning models are usually evaluated on their ability to describe a held-out set of images, not on their ability to generalize to unseen concepts. We study the problem of compositional generalization, which measures how well a model composes unseen combinations of concepts when describing images. State-of-the-art image captioning models show poor generalization performance on this task. We propose a multi-task model to address the poor performance, that combines caption generation and image--sentence ranking, and uses a decoding mechanism that re-ranks the captions according their similarity to the image. This model is substantially better at generalizing to unseen combinations of concepts compared to state-of-the-art captioning models.

READ FULL TEXT
research
01/28/2021

The Role of Syntactic Planning in Compositional Image Captioning

Image captioning has focused on generalizing to images drawn from the sa...
research
08/27/2016

Learning to generalize to new compositions in image understanding

Recurrent neural networks have recently been used for learning to descri...
research
09/10/2020

Weakly Supervised Content Selection for Improved Image Captioning

Image captioning involves identifying semantic concepts in the scene and...
research
06/09/2019

Learning to Predict Novel Noun-Noun Compounds

We introduce temporally and contextually-aware models for the novel task...
research
10/23/2018

A Neural Compositional Paradigm for Image Captioning

Mainstream captioning models often follow a sequential structure to gene...
research
10/17/2017

Describing Natural Images Containing Novel Objects with Knowledge Guided Assitance

Images in the wild encapsulate rich knowledge about varied abstract conc...
research
12/04/2020

Understanding Guided Image Captioning Performance across Domains

Image captioning models generally lack the capability to take into accou...

Please sign up or login with your details

Forgot password? Click here to reset