Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

08/31/2020
by   Leena Mathur, et al.
0

Automated deception detection systems can enhance health, justice, and security in society by helping humans detect deceivers in high-stakes situations across medical and legal domains, among others. This paper presents a novel analysis of the discriminative power of dimensional representations of facial affect for automated deception detection, along with interpretable features from visual, vocal, and verbal modalities. We used a video dataset of people communicating truthfully or deceptively in real-world, high-stakes courtroom situations. We leveraged recent advances in automated emotion recognition in-the-wild by implementing a state-of-the-art deep neural network trained on the Aff-Wild database to extract continuous representations of facial valence and facial arousal from speakers. We experimented with unimodal Support Vector Machines (SVM) and SVM-based multimodal fusion methods to identify effective features, modalities, and modeling approaches for detecting deception. Unimodal models trained on facial affect achieved an AUC of 80 facial affect contributed towards the highest-performing multimodal approach (adaptive boosting) that achieved an AUC of 91 were not part of training sets. This approach achieved a higher AUC than existing automated machine learning approaches that used interpretable visual, vocal, and verbal features to detect deception in this dataset, but did not use facial affect. Across all videos, deceptive and truthful speakers exhibited significant differences in facial valence and facial arousal, contributing computational support to existing psychological theories on affect and deception. The demonstrated importance of facial affect in our models informs and motivates the future development of automated, affect-aware machine learning approaches for modeling and detecting deception and other social behaviors in-the-wild.

READ FULL TEXT
research
08/17/2021

Affect-Aware Deep Belief Network Representations for Multimodal Unsupervised Deception Detection

Automated systems that detect the social behavior of deception can enhan...
research
07/13/2023

Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches

Deception detection is an interdisciplinary field attracting researchers...
research
07/22/2021

Privileged Information for Modeling Affect In The Wild

A key challenge of affective computing research is discovering ways to r...
research
02/06/2021

Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection

Automated systems that detect deception in high-stakes situations can en...
research
07/05/2016

Aggressive actions and anger detection from multiple modalities using Kinect

Prison facilities, mental correctional institutions, sports bars and pla...
research
09/07/2022

Evaluating Temporal Patterns in Applied Infant Affect Recognition

Agents must monitor their partners' affective states continuously in ord...
research
08/25/2022

Supervised Contrastive Learning for Affect Modelling

Affect modeling is viewed, traditionally, as the process of mapping meas...

Please sign up or login with your details

Forgot password? Click here to reset