Use of Affective Visual Information for Summarization of Human-Centric Videos

by   Berkay Köprü, et al.

Increasing volume of user-generated human-centric video content and their applications, such as video retrieval and browsing, require compact representations that are addressed by the video summarization literature. Current supervised studies formulate video summarization as a sequence-to-sequence learning problem and the existing solutions often neglect the surge of human-centric view, which inherently contains affective content. In this study, we investigate the affective-information enriched supervised video summarization task for human-centric videos. First, we train a visual input-driven state-of-the-art continuous emotion recognition model (CER-NET) on the RECOLA dataset to estimate emotional attributes. Then, we integrate the estimated emotional attributes and the high-level representations from the CER-NET with the visual information to define the proposed affective video summarization architectures (AVSUM). In addition, we investigate the use of attention to improve the AVSUM architectures and propose two new architectures based on temporal attention (TA-AVSUM) and spatial attention (SA-AVSUM). We conduct video summarization experiments on the TvSum database. The proposed AVSUM-GRU architecture with an early fusion of high level GRU embeddings and the temporal attention based TA-AVSUM architecture attain competitive video summarization performances by bringing strong performance improvements for the human-centric videos compared to the state-of-the-art in terms of F-score and self-defined face recall metrics.



There are no comments yet.


page 1

page 2

page 3

page 4


Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization

Emotional content is a key element in user-generated videos. However, it...

Efficient Video Summarization Framework using EEG and Eye-tracking Signals

This paper proposes an efficient video summarization framework that will...

Show and Recall: Learning What Makes Videos Memorable

With the explosion of video content on the Internet, there is a need for...

Video Summarization Based on Video-text Modelling

Modern video summarization methods are based on deep neural networks whi...

A Multi-task Neural Approach for Emotion Attribution, Classification and Summarization

Emotional content is a crucial ingredient in user-generated videos. Howe...

Audiovisual Highlight Detection in Videos

In this paper, we test the hypothesis that interesting events in unstruc...

Query Twice: Dual Mixture Attention Meta Learning for Video Summarization

Video summarization aims to select representative frames to retain high-...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.