Analyzing the Affect of a Group of People Using Multi-modal Framework

by   Xiaohua Huang, et al.

Millions of images on the web enable us to explore images from social events such as a family party, thus it is of interest to understand and model the affect exhibited by a group of people in images. But analysis of the affect expressed by multiple people is challenging due to varied indoor and outdoor settings, and interactions taking place between various numbers of people. A few existing works on Group-level Emotion Recognition (GER) have investigated on face-level information. Due to the challenging environments, face may not provide enough information to GER. Relatively few studies have investigated multi-modal GER. Therefore, we propose a novel multi-modal approach based on a new feature description for understanding emotional state of a group of people in an image. In this paper, we firstly exploit three kinds of rich information containing face, upperbody and scene in a group-level image. Furthermore, in order to integrate multiple person's information in a group-level image, we propose an information aggregation method to generate three features for face, upperbody and scene, respectively. We fuse face, upperbody and scene information for robustness of GER against the challenging environments. Intensive experiments are performed on two challenging group-level emotion databases to investigate the role of face, upperbody and scene as well as multi-modal framework. Experimental results demonstrate that our framework achieves very promising performance for GER.


page 1

page 3

page 4

page 5

page 9


Automatic Group Cohesiveness Detection With Multi-modal Features

Group cohesiveness is a compelling and often studied composition in grou...

A Multi-Modal Approach to Infer Image Affect

The group affect or emotion in an image of people can be inferred by ext...

A Global Alignment Kernel based Approach for Group-level Happiness Intensity Estimation

With the progress in automatic human behavior understanding, analysing t...

Predicting Group Cohesiveness in Images

Cohesiveness of a group is an essential indicator of emotional state, st...

On Safe and Usable Chatbots for Promoting Voter Participation

Chatbots, or bots for short, are multi-modal collaborative assistants th...

Finding Dory in the Crowd: Detecting Social Interactions using Multi-Modal Mobile Sensing

Remembering our day-to-day social interactions is challenging even if yo...

A Quantum Probability Driven Framework for Joint Multi-Modal Sarcasm, Sentiment and Emotion Analysis

Sarcasm, sentiment, and emotion are three typical kinds of spontaneous a...

Please sign up or login with your details

Forgot password? Click here to reset