Investigations on Audiovisual Emotion Recognition in Noisy Conditions

by   Michael Neumann, et al.

In this paper we explore audiovisual emotion recognition under noisy acoustic conditions with a focus on speech features. We attempt to answer the following research questions: (i) How does speech emotion recognition perform on noisy data? and (ii) To what extend does a multimodal approach improve the accuracy and compensate for potential performance degradation at different noise levels? We present an analytical investigation on two emotion datasets with superimposed noise at different signal-to-noise ratios, comparing three types of acoustic features. Visual features are incorporated with a hybrid fusion approach: The first neural network layers are separate modality-specific ones, followed by at least one shared layer before the final prediction. The results show a significant performance decrease when a model trained on clean audio is applied to noisy data and that the addition of visual features alleviates this effect.



There are no comments yet.


page 4

page 5


Emotion Recognition in Audio and Video Using Deep Neural Networks

Humans are able to comprehend information from multiple domains for e.g....

The Effect of Silence Feature in Dimensional Speech Emotion Recognition

Silence is a part of human-to-human communication, which can be a clue f...

Multimodal Local-Global Ranking Fusion for Emotion Recognition

Emotion recognition is a core research area at the intersection of artif...

Dynamic Layer Customization for Noise Robust Speech Emotion Recognition in Heterogeneous Condition Training

Robustness to environmental noise is important to creating automatic spe...

Learning Fine-Grained Multimodal Alignment for Speech Emotion Recognition

Speech emotion recognition is a challenging task because the emotion exp...

Is Everything Fine, Grandma? Acoustic and Linguistic Modeling for Robust Elderly Speech Emotion Recognition

Acoustic and linguistic analysis for elderly emotion recognition is an u...

On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks

Speech emotion recognition (SER) is an important aspect of effective hum...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.