Using Scene and Semantic Features for Multi-modal Emotion Recognition

by   Zhifeng Wang, et al.

Automatic emotion recognition is a hot topic with a wide range of applications. Much work has been done in the area of automatic emotion recognition in recent years. The focus has been mainly on using the characteristics of a person such as speech, facial expression and pose for this purpose. However, the processing of scene and semantic features for emotion recognition has had limited exploration. In this paper, we propose to use combined scene and semantic features, along with personal features, for multi-modal emotion recognition. Scene features will describe the environment or context in which the target person is operating. The semantic feature can include objects that are present in the environment, as well as their attributes and relationships with the target person. In addition, we use a modified EmbraceNet to extract features from the images, which is trained to learn both the body and pose features simultaneously. By fusing both body and pose features, the EmbraceNet can improve the accuracy and robustness of the model, particularly when dealing with partially missing data. This is because having both body and pose features provides a more complete representation of the subject in the images, which can help the model to make more accurate predictions even when some parts of body are missing. We demonstrate the efficiency of our method on the benchmark EMOTIC dataset. We report an average precision of 40.39% across the 26 emotion categories, which is a 5% improvement over previous approaches.


page 1

page 3

page 5

page 7

page 8


PERI: Part Aware Emotion Recognition In The Wild

Emotion recognition aims to interpret the emotional states of a person b...

Context Based Emotion Recognition usingEMOTIC Dataset

n our everyday lives and social interactions we often try to perceive th...

Survey on Emotional Body Gesture Recognition

Automatic emotion recognition has become a trending research topic in th...

Context Based Emotion Recognition using EMOTIC Dataset

In our everyday lives and social interactions we often try to perceive t...

Emotion Understanding in Videos Through Body, Context, and Visual-Semantic Embedding Loss

We present our winning submission to the First International Workshop on...

Automatic Group Cohesiveness Detection With Multi-modal Features

Group cohesiveness is a compelling and often studied composition in grou...

Please sign up or login with your details

Forgot password? Click here to reset