Knowledge-Enriched Visual Storytelling

by   Chao-Chun Hsu, et al.
University of Colorado Boulder
Penn State University
Academia Sinica

Stories are diverse and highly personalized, resulting in a large possible output space for story generation. Existing end-to-end approaches produce monotonous stories because they are limited to the vocabulary and knowledge in a single training dataset. This paper introduces KG-Story, a three-stage framework that allows the story generation model to take advantage of external Knowledge Graphs to produce interesting stories. KG-Story distills a set of representative words from the input prompts, enriches the word set by using external knowledge graphs, and finally generates stories based on the enriched word set. This distill-enrich-generate framework allows the use of external resources not only for the enrichment phase, but also for the distillation and generation phases. In this paper, we show the superiority of KG-Story for visual storytelling, where the input prompt is a sequence of five photos and the output is a short story. Per the human ranking evaluation, stories generated by KG-Story are on average ranked better than that of the state-of-the-art systems. Our code and output stories are available at


page 1

page 4

page 6

page 8


Plot and Rework: Modeling Storylines for Visual Storytelling

Writing a coherent and engaging story is not easy. Creative writers use ...

WriterForcing: Generating more interesting story endings

We study the problem of generating interesting endings for stories. Neur...

Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization

While much research has been done in text-to-image synthesis, little wor...

StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning

Existing automatic story evaluation methods place a premium on story lex...

Hierarchically-Attentive RNN for Album Summarization and Storytelling

We address the problem of end-to-end visual storytelling. Given a photo ...

Semantic Frame Forecast

This paper introduces semantic frame forecast, a task that predicts the ...

Let's Talk! Striking Up Conversations via Conversational Visual Question Generation

An engaging and provocative question can open up a great conversation. I...

Please sign up or login with your details

Forgot password? Click here to reset