ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer

02/15/2022
by   Kohei Uehara, et al.
6

Image narrative generation describes the creation of stories regarding the content of image data from a subjective viewpoint. Given the importance of the subjective feelings of writers, characters, and readers in storytelling, image narrative generation methods must consider human emotion, which is their major difference from descriptive caption generation tasks. The development of automated methods to generate story-like text associated with images may be considered to be of considerable social significance, because stories serve essential functions both as entertainment and also for many practical purposes such as education and advertising. In this study, we propose a model called ViNTER (Visual Narrative Transformer with Emotion arc Representation) to generate image narratives that focus on time series representing varying emotions as "emotion arcs," to take advantage of recent advances in multimodal Transformer-based pre-trained models. We present experimental results of both manual and automatic evaluations, which demonstrate the effectiveness of the proposed emotion-aware approach to image narrative generation.

READ FULL TEXT

page 1

page 3

page 5

page 8

research
10/14/2020

Modeling Protagonist Emotions for Emotion-Aware Storytelling

Emotions and their evolution play a central role in creating a captivati...
research
05/25/2021

Multi-Task Learning of Generation and Classification for Emotion-Aware Dialogue Response Generation

For a computer to naturally interact with a human, it needs to be human-...
research
01/07/2023

Visual Story Generation Based on Emotion and Keywords

Automated visual story generation aims to produce stories with correspon...
research
08/31/2023

Socratis: Are large multimodal models emotionally aware?

Existing emotion prediction benchmarks contain coarse emotion labels whi...
research
10/11/2022

CHAE: Fine-Grained Controllable Story Generation with Characters, Actions and Emotions

Story generation has emerged as an interesting yet challenging NLP task ...
research
03/19/2021

Computational Emotion Analysis From Images: Recent Advances and Future Directions

Emotions are usually evoked in humans by images. Recently, extensive res...
research
01/26/2021

Automatic Comic Generation with Stylistic Multi-page Layouts and Emotion-driven Text Balloon Generation

In this paper, we propose a fully automatic system for generating comic ...

Please sign up or login with your details

Forgot password? Click here to reset