Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning

11/11/2020
by   Jonathan Boigne, et al.
0

We propose a novel transfer learning method for speech emotion recognition allowing us to obtain promising results when only few training data is available. With as low as 125 examples per emotion class, we were able to reach a higher accuracy than a strong baseline trained on 8 times more data. Our method leverages knowledge contained in pre-trained speech representations extracted from models trained on a more general self-supervised task which doesn't require human annotations, such as the wav2vec model. We provide detailed insights on the benefits of our approach by varying the training data size, which can help labeling teams to work more efficiently. We compare performance with other popular methods on the IEMOCAP dataset, a well-benchmarked dataset among the Speech Emotion Recognition (SER) research community. Furthermore, we demonstrate that results can be greatly improved by combining acoustic and linguistic knowledge from transfer learning. We align acoustic pre-trained representations with semantic representations from the BERT model through an attention-based recurrent neural network. Performance improves significantly when combining both modalities and scales with the amount of data. When trained on the full IEMOCAP dataset, we reach a new state-of-the-art of 73.9

READ FULL TEXT
research
03/30/2018

Reusing Neural Speech Representations for Auditory Emotion Recognition

Acoustic emotion recognition aims to categorize the affective state of t...
research
11/18/2020

On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition

Pre-training for feature extraction is an increasingly studied approach ...
research
04/08/2021

Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

Emotion recognition datasets are relatively small, making the use of the...
research
04/06/2022

Emotional Speech Recognition with Pre-trained Deep Visual Models

In this paper, we propose a new methodology for emotional speech recogni...
research
11/03/2022

Speech-based emotion recognition with self-supervised models using attentive channel-wise correlations and label smoothing

When recognizing emotions from speech, we encounter two common problems:...
research
10/11/2019

Emotion Recognition in Conversations with Transfer Learning from Generative Conversation Modeling

Recognizing emotions in conversations is a challenging task due to the p...
research
06/24/2019

Multimodal and Multi-view Models for Emotion Recognition

Studies on emotion recognition (ER) show that combining lexical and acou...

Please sign up or login with your details

Forgot password? Click here to reset