DeepAI AI Chat
Log In Sign Up

Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts

by   Harsh Agarwal, et al.
Indian Institute of Technology Kanpur

Emotion Recognition in Conversations (ERC) is an important and active research problem. Recent work has shown the benefits of using multiple modalities (e.g., text, audio, and video) for the ERC task. In a conversation, participants tend to maintain a particular emotional state unless some external stimuli evokes a change. There is a continuous ebb and flow of emotions in a conversation. Inspired by this observation, we propose a multimodal ERC model and augment it with an emotion-shift component. The proposed emotion-shift component is modular and can be added to any existing multimodal ERC model (with a few modifications), to improve emotion recognition. We experiment with different variants of the model, and results show that the inclusion of emotion shift signal helps the model to outperform existing multimodal models for ERC and hence showing the state-of-the-art performance on MOSEI and IEMOCAP datasets.


page 1

page 2

page 3

page 4


ICANet: A Method of Short Video Emotion Recognition Driven by Multimodal Data

With the fast development of artificial intelligence and short videos, e...

A Multimodal Corpus for Emotion Recognition in Sarcasm

While sentiment and emotion analysis have been studied extensively, the ...

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

Recognizing emotions during social interactions has many potential appli...

Noise-Resistant Multimodal Transformer for Emotion Recognition

Multimodal emotion recognition identifies human emotions from various da...

Bias and Fairness on Multimodal Emotion Detection Algorithms

Numerous studies have shown that machine learning algorithms can latch o...

Few-Shot Emotion Recognition in Conversation with Sequential Prototypical Networks

Several recent studies on dyadic human-human interactions have been done...

The Contextual Dynamics of Multimodal Emotion Recognition in Videos

Emotional expressions form a key part of user behavior on today's digita...