Visual-Texual Emotion Analysis with Deep Coupled Video and Danmu Neural Networks

11/19/2018
by   Chenchen Li, et al.
0

User emotion analysis toward videos is to automatically recognize the general emotional status of viewers from the multimedia content embedded in the online video stream. Existing works fall in two categories: 1) visual-based methods, which focus on visual content and extract a specific set of features of videos. However, it is generally hard to learn a mapping function from low-level video pixels to high-level emotion space due to great intra-class variance. 2) textual-based methods, which focus on the investigation of user-generated comments associated with videos. The learned word representations by traditional linguistic approaches typically lack emotion information and the global comments usually reflect viewers' high-level understandings rather than instantaneous emotions. To address these limitations, in this paper, we propose to jointly utilize video content and user-generated texts simultaneously for emotion analysis. In particular, we introduce exploiting a new type of user-generated texts, i.e., "danmu", which are real-time comments floating on the video and contain rich information to convey viewers' emotional opinions. To enhance the emotion discriminativeness of words in textual feature extraction, we propose Emotional Word Embedding (EWE) to learn text representations by jointly considering their semantics and emotions. Afterwards, we propose a novel visual-textual emotion analysis model with Deep Coupled Video and Danmu Neural networks (DCVDN), in which visual and textual features are synchronously extracted and fused to form a comprehensive representation by deep-canonically-correlated-autoencoder-based multi-view learning. Through extensive experiments on a self-crawled real-world video-danmu dataset, we prove that DCVDN significantly outperforms the state-of-the-art baselines.

READ FULL TEXT

page 4

page 16

page 17

research
05/11/2022

User Guide for KOTE: Korean Online Comments Emotions Dataset

Sentiment analysis that classifies data into positive or negative has be...
research
02/24/2020

Emosaic: Visualizing Affective Content of Text at Varying Granularity

This paper presents Emosaic, a tool for visualizing the emotional tone o...
research
11/16/2015

Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization

Emotional content is a key element in user-generated videos. However, it...
research
12/21/2018

A Multi-task Neural Approach for Emotion Attribution, Classification and Summarization

Emotional content is a crucial ingredient in user-generated videos. Howe...
research
09/06/2017

Affect Recognition in Ads with Application to Computational Advertising

Advertisements (ads) often include strongly emotional content to leave a...
research
07/29/2019

EmoCo: Visual Analysis of Emotion Coherence in Presentation Videos

Emotions play a key role in human communication and public presentations...
research
08/09/2019

Interactive Variance Attention based Online Spoiler Detection for Time-Sync Comments

Nowadays, time-sync comment (TSC), a new form of interactive comments, h...

Please sign up or login with your details

Forgot password? Click here to reset