Towards Multimodal Emotion Recognition in German Speech Events in Cars using Transfer Learning

09/06/2019
by   Deniz Cevher, et al.
0

The recognition of emotions by humans is a complex process which considers multiple interacting signals such as facial expressions and both prosody and semantic content of utterances. Commonly, research on automatic recognition of emotions is, with few exceptions, limited to one modality. We describe an in-car experiment for emotion recognition from speech interactions for three modalities: the audio signal of a spoken interaction, the visual signal of the driver's face, and the manually transcribed content of utterances of the driver. We use off-the-shelf tools for emotion detection in audio and face and compare that to a neural transfer learning approach for emotion recognition from text which utilizes existing resources from other domains. We see that transfer learning enables models based on out-of-domain corpora to perform well. This method contributes up to 10 percentage points in F1, with up to 76 micro-average F1 across the emotions joy, annoyance and insecurity. Our findings also indicate that off-the-shelf-tools analyzing face and audio are not ready yet for emotion detection in in-car speech interactions without further adjustments.

READ FULL TEXT
research
04/16/2018

Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning

Emotion recognition has become an important field of research in Human C...
research
11/03/2021

Multi-Cue Adaptive Emotion Recognition Network

Expressing and identifying emotions through facial and physical expressi...
research
02/16/2022

Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models

Automatic emotion recognition plays a key role in computer-human interac...
research
11/01/2019

Clinical Depression and Affect Recognition with EmoAudioNet

Automatic analysis of emotions and affects from speech is an inherently ...
research
09/30/2020

Embedded Emotions – A Data Driven Approach to Learn Transferable Feature Representations from Raw Speech Input for Emotion Recognition

Traditional approaches to automatic emotion recognition are relying on t...
research
12/12/2022

An Approach for Improving Automatic Mouth Emotion Recognition

The study proposes and tests a technique for automated emotion recogniti...
research
04/05/2021

Acted vs. Improvised: Domain Adaptation for Elicitation Approaches in Audio-Visual Emotion Recognition

Key challenges in developing generalized automatic emotion recognition s...

Please sign up or login with your details

Forgot password? Click here to reset