Generating Descriptions with Grounded and Co-Referenced People

04/05/2017
by   Anna Rohrbach, et al.
0

Learning how to generate descriptions of images or videos received major interest both in the Computer Vision and Natural Language Processing communities. While a few works have proposed to learn a grounding during the generation process in an unsupervised way (via an attention mechanism), it remains unclear how good the quality of the grounding is and whether it benefits the description quality. In this work we propose a movie description model which learns to generate description and jointly ground (localize) the mentioned characters as well as do visual co-reference resolution between pairs of consecutive sentences/clips. We also propose to use weak localization supervision through character mentions provided in movie descriptions to learn the character grounding. At training time, we first learn how to localize characters by relating their visual appearance to mentions in the descriptions via a semi-supervised approach. We then provide this (noisy) supervision into our description model which greatly improves its performance. Our proposed description model improves over prior work w.r.t. generated description quality and additionally provides grounding and local co-reference resolution. We evaluate it on the MPII Movie Description dataset using automatic and human evaluation measures and using our newly collected grounding and co-reference data for characters.

READ FULL TEXT

page 1

page 6

page 8

page 9

research
01/12/2015

A Dataset for Movie Description

Descriptive video service (DVS) provides linguistic descriptions of movi...
research
06/01/2019

Learning to Generate Grounded Image Captions without Localization Supervision

When generating a sentence description for an image, it frequently remai...
research
10/23/2020

Identifying Similar Movie Characters Quickly but Effectively Using Non-exhaustive Pair-wise Attention

Identifying similar movie characters is a captivating task that can be o...
research
06/04/2015

The Long-Short Story of Movie Description

Generating descriptions for videos has many applications including assis...
research
01/15/2016

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

Automatic description generation from natural images is a challenging pr...
research
10/20/2022

MBTI Personality Prediction for Fictional Characters Using Movie Scripts

An NLP model that understands stories should be able to understand the c...
research
03/07/2023

Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions

Human language users can generate descriptions of perceptual concepts be...

Please sign up or login with your details

Forgot password? Click here to reset