Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences

11/24/2019
by   Shizhe Chen, et al.
0

A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products. In this paper, we tackle a new multimedia task of automatic storyboard creation to facilitate this process and inspire human artists. Inspired by the fact that our understanding of languages is based on our past experience, we propose a novel inspire-and-create framework with a story-to-image retriever that selects relevant cinematic images for inspiration and a storyboard creator that further refines and renders images to improve the relevancy and visual consistency. The proposed retriever dynamically employs contextual information in the story with hierarchical attentions and applies dense visual-semantic matching to accurately retrieve and ground images. The creator then employs three rendering steps to increase the flexibility of retrieved images, which include erasing irrelevant regions, unifying styles of images and substituting consistent characters. We carry out extensive experiments on both in-domain and out-of-domain visual story datasets. The proposed model achieves better quantitative performance than the state-of-the-art baselines for storyboard creation. Qualitative visualizations and user studies further verify that our approach can create high-quality storyboards even for stories in the wild.

READ FULL TEXT

page 2

page 8

research
12/06/2018

StoryGAN: A Sequential Conditional GAN for Story Visualization

In this work we propose a new task called Story Visualization. Given a m...
research
09/11/2019

What Makes A Good Story? Designing Composite Rewards for Visual Storytelling

Previous storytelling approaches mostly focused on optimizing traditiona...
research
10/06/2022

Vision Transformer Based Model for Describing a Set of Images as a Story

Visual Story-Telling is the process of forming a multi-sentence story fr...
research
08/03/2022

Word-Level Fine-Grained Story Visualization

Story visualization aims to generate a sequence of images to narrate eac...
research
08/04/2021

Towards Coherent Visual Storytelling with Ordered Image Attention

We address the problem of visual storytelling, i.e., generating a story ...
research
10/13/2021

Assisting News Media Editors with Cohesive Visual Storylines

Creating a cohesive, high-quality, relevant, media story is a challenge ...
research
09/13/2022

Exemplar-Based Image Colorization with A Learning Framework

Image learning and colorization are hot spots in multimedia domain. Insp...

Please sign up or login with your details

Forgot password? Click here to reset