Using Crowdsourcing to Train Facial Emotion Machine Learning Models with Ambiguous Labels

by   Peter Washington, et al.

Current emotion detection classifiers predict discrete emotions. However, literature in psychology has documented that compound and ambiguous facial expressions are often evoked by humans. As a stride towards development of machine learning models that more accurately reflect compound and ambiguous emotions, we replace traditional one-hot encoded label representations with a crowd's distribution of labels. We center our study on the Child Affective Facial Expression (CAFE) dataset, a gold standard dataset of pediatric facial expressions which includes 100 human labels per image. We first acquire crowdsourced labels for 207 emotions from CAFE and demonstrate that the consensus labels from the crowd tend to match the consensus from the original CAFE raters, validating the utility of crowdsourcing. We then train two versions of a ResNet-152 classifier on CAFE images with two types of labels (1) traditional one-hot encoding and (2) vector labels representing the crowd distribution of responses. We compare the resulting output distributions of the two classifiers. While the traditional F1-score for the one-hot encoding classifier is much higher (94.33 the crowd-trained classifier much more closely resembles the distribution of human labels (t=3.2827, p=0.0014). For many applications of affective computing, reporting an emotion probability distribution that more closely resembles human interpretation can be more important than traditional machine learning metrics. This work is a first step for engineers of interactive systems to account for machine learning cases with ambiguous classes and we hope it will generate a discussion about machine learning with ambiguous labels and leveraging crowdsourcing as a potential solution.


Facial Expression Editing with Continuous Emotion Labels

Recently deep generative models have achieved impressive results in the ...

Multi-Classifier Interactive Learning for Ambiguous Speech Emotion Recognition

In recent years, speech emotion recognition technology is of great signi...

Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition

By utilizing label distribution learning, a probability distribution is ...

Performance Evaluation of Supervised Machine Learning Techniques for Efficient Detection of Emotions from Online Content

Emotion detection from the text is an important and challenging problem ...

Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution

Crowd sourcing has become a widely adopted scheme to collect ground trut...

Addressing Ambiguity of Emotion Labels Through Meta-learning

Emotion labels in emotion recognition corpora are highly noisy and ambig...

Evaluating Classifiers Without Expert Labels

This paper considers the challenge of evaluating a set of classifiers, a...