Video Description: A Survey of Methods, Datasets and Evaluation Metrics

06/01/2018
by   Nayyer Aafaq, et al.
0

Automatic video description is useful for assisting the visually impaired, human computer interaction, robotics and video indexing. The past few years have seen a surge of research interest in this area due to the unprecedented success of deep learning in computer vision and natural language processing. Numerous methods, datasets and evaluation measures have been proposed in the literature calling the need for a comprehensive survey to better focus research efforts in this flourishing direction. This paper answers exactly to this need by surveying state of the art approaches including deep learning models; comparing benchmark datasets in terms of their domain, number of classes, and repository size; and identifying the pros and cons of various evaluation metrics such as BLEU, ROUGE, METEOR, CIDEr, SPICE and WMD. Our survey shows that video description research has a long way to go before it can match human performance and that the main reasons for this shortfall are twofold. Firstly, existing datasets do not adequately represent the diversity in open domain videos and complex linguistic structures. Secondly, current measures of evaluation are not aligned with human judgement. For example, the same video can have very different, yet correct descriptions. We conclude that there is a need for improvement in evaluation measures as well as datasets in terms of size, diversity and annotation accuracy because they directly influence the development of better video description models. From an algorithmic point of view, diagnosis of the description quality is challenging because of the difficultly to assess the level of contribution from visual features compared to the bias that comes naturally from the language model adopted.

READ FULL TEXT

page 2

page 6

page 12

page 14

page 15

page 24

research
11/30/2020

A Comprehensive Review on Recent Methods and Challenges of Video Description

Video description involves the generation of the natural language descri...
research
01/15/2016

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

Automatic description generation from natural images is a challenging pr...
research
08/02/2020

A Survey on Text Classification: From Shallow to Deep Learning

Text classification is the most fundamental and essential task in natura...
research
07/22/2019

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods

Integration of vision and language tasks has seen a significant growth i...
research
07/18/2022

Towards the Human Global Context: Does the Vision-Language Model Really Judge Like a Human Being?

As computer vision and NLP make progress, Vision-Language(VL) is becomin...
research
04/27/2021

TRECVID 2020: A comprehensive campaign for evaluating video retrieval tasks across multiple application domains

The TREC Video Retrieval Evaluation (TRECVID) is a TREC-style video anal...
research
02/13/2013

Object Recognition with Imperfect Perception and Redundant Description

This paper deals with a scene recognition system in a robotics contex. T...

Please sign up or login with your details

Forgot password? Click here to reset