Multimodal Event Transformer for Image-guided Story Ending Generation

01/26/2023
by   Yucheng Zhou, et al.
0

Image-guided story ending generation (IgSEG) is to generate a story ending based on given story plots and ending image. Existing methods focus on cross-modal feature fusion but overlook reasoning and mining implicit information from story plots and ending image. To tackle this drawback, we propose a multimodal event transformer, an event-based reasoning framework for IgSEG. Specifically, we construct visual and semantic event graphs from story plots and ending image, and leverage event-based reasoning to reason and mine implicit information in a single modality. Next, we connect visual and semantic event graphs and utilize cross-modal fusion to integrate different-modality features. In addition, we propose a multimodal injector to adaptive pass essential information to decoder. Besides, we present an incoherence detection to enhance the understanding context of a story plot and the robustness of graph modeling for our model. Experimental results show that our method achieves state-of-the-art performance for the image-guided story ending generation.

READ FULL TEXT
research
01/26/2023

Improving Cross-modal Alignment for Text-Guided Image Inpainting

Text-guided image inpainting (TGII) aims to restore missing regions base...
research
03/10/2022

Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling

As a technically challenging topic, visual storytelling aims at generati...
research
08/17/2023

Text-Only Training for Visual Storytelling

Visual storytelling aims to generate a narrative based on a sequence of ...
research
05/15/2021

Premise-based Multimodal Reasoning: A Human-like Cognitive Process

Reasoning is one of the major challenges of Human-like AI and has recent...
research
10/22/2022

EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention

One of the key challenges of automatic story generation is how to genera...
research
02/27/2023

TOT: Topology-Aware Optimal Transport For Multimodal Hate Detection

Multimodal hate detection, which aims to identify harmful content online...
research
07/07/2019

Informative Visual Storytelling with Cross-modal Rules

Existing methods in the Visual Storytelling field often suffer from the ...

Please sign up or login with your details

Forgot password? Click here to reset