DeepAI AI Chat
Log In Sign Up

Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction

by   Shizhe Chen, et al.
Renmin University of China

Continuous dimensional emotion prediction is a challenging task where the fusion of various modalities usually achieves state-of-the-art performance such as early fusion or late fusion. In this paper, we propose a novel multi-modal fusion strategy named conditional attention fusion, which can dynamically pay attention to different modalities at each time step. Long-short term memory recurrent neural networks (LSTM-RNN) is applied as the basic uni-modality model to capture long time dependencies. The weights assigned to different modalities are automatically decided by the current input features and recent history information rather than being fixed at any kinds of situation. Our experimental results on a benchmark dataset AVEC2015 show the effectiveness of our method which outperforms several common fusion strategies for valence prediction.


page 1

page 2

page 3

page 4


Video Affective Effects Prediction with Multi-modal Fusion and Shot-Long Temporal Context

Predicting the emotional impact of videos using machine learning is a ch...

Memory Based Attentive Fusion

The use of multi-modal data for deep machine learning has shown promise ...

Multi-modal Automated Speech Scoring using Attention Fusion

In this study, we propose a novel multi-modal end-to-end neural approach...

Multi-Modality Multi-Loss Fusion Network

In this work we investigate the optimal selection and fusion of features...

Hierachical Delta-Attention Method for Multimodal Fusion

In vision and linguistics; the main input modalities are facial expressi...

Evidence-aware multi-modal data fusion and its application to total knee replacement prediction

Deep neural networks have been widely studied for predicting a medical c...

NUAA-QMUL-AIIT at Memotion 3: Multi-modal Fusion with Squeeze-and-Excitation for Internet Meme Emotion Analysis

This paper describes the participation of our NUAA-QMUL-AIIT team in the...