Senti-Attend: Image Captioning using Sentiment and Attention

11/24/2018
by   Omid Mohamad Nezami, et al.
0

There has been much recent work on image captioning models that describe the factual aspects of an image. Recently, some models have incorporated non-factual aspects into the captions, such as sentiment or style. However, such models typically have difficulty in balancing the semantic aspects of the image and the non-factual dimensions of the caption; in addition, it can be observed that humans may focus on different aspects of an image depending on the chosen sentiment or style of the caption. To address this, we design an attention-based model to better add sentiment to image captions. The model embeds and learns sentiment with respect to image-caption data, and uses both high-level and word-level sentiment information during the learning process. The model outperforms the state-of-the-art work in image captioning with sentiment using standard evaluation metrics. An analysis of generated captions also shows that our model does this by a better selection of the sentiment-bearing adjectives and adjective-noun pairs.

READ FULL TEXT

page 1

page 8

page 10

page 11

page 12

page 13

page 14

page 15

research
08/08/2019

Towards Generating Stylized Image Captions via Adversarial Training

While most image captioning aims to generate objective descriptions of i...
research
07/10/2018

"Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention

Generating stylized captions for an image is an emerging topic in image ...
research
11/21/2015

Mapping Images to Sentiment Adjective Noun Pairs with Factorized Neural Nets

We consider the visual sentiment task of mapping an image to an adjectiv...
research
10/31/2019

Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder

Most RNN-based image captioning models receive supervision on the output...
research
09/07/2019

Look and Modify: Modification Networks for Image Captioning

Attention-based neural encoder-decoder frameworks have been widely used ...
research
09/18/2020

Image Captioning with Attention for Smart Local Tourism using EfficientNet

Smart systems have been massively developed to help humans in various ta...
research
05/28/2022

BAN-Cap: A Multi-Purpose English-Bangla Image Descriptions Dataset

As computers have become efficient at understanding visual information a...

Please sign up or login with your details

Forgot password? Click here to reset