DeepAI AI Chat
Log In Sign Up

Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers

by   Xiang Zhang, et al.

Multi-modal learning has been intensified in recent years, especially for applications in facial analysis and action unit detection whilst there still exist two main challenges in terms of 1) relevant feature learning for representation and 2) efficient fusion for multi-modalities. Recently, there are a number of works have shown the effectiveness in utilizing the attention mechanism for AU detection, however, most of them are binding the region of interest (ROI) with features but rarely apply attention between features of each AU. On the other hand, the transformer, which utilizes a more efficient self-attention mechanism, has been widely used in natural language processing and computer vision tasks but is not fully explored in AU detection tasks. In this paper, we propose a novel end-to-end Multi-Head Fused Transformer (MFT) method for AU detection, which learns AU encoding features representation from different modalities by transformer encoder and fuses modalities by another fusion transformer module. Multi-head fusion attention is designed in the fusion transformer module for the effective fusion of multiple modalities. Our approach is evaluated on two public multi-modal AU databases, BP4D, and BP4D+, and the results are superior to the state-of-the-art algorithms and baseline models. We further analyze the performance of AU detection from different modalities.


page 3

page 4

page 7


Multimodal Learning with Channel-Mixing and Masked Autoencoder on Facial Action Unit Detection

Recent studies utilizing multi-modal data aimed at building a robust mod...

Multi-Modal Sarcasm Detection Based on Contrastive Attention Mechanism

In the past decade, sarcasm detection has been intensively conducted in ...

Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

Sarcasm is a linguistic phenomenon indicating a discrepancy between lite...

Improved Xception with Dual Attention Mechanism and Feature Fusion for Face Forgery Detection

With the rapid development of deep learning technology, more and more fa...

Deep Collaborative Multi-Modal Learning for Unsupervised Kinship Estimation

Kinship verification is a long-standing research challenge in computer v...

TAMFormer: Multi-Modal Transformer with Learned Attention Mask for Early Intent Prediction

Human intention prediction is a growing area of research where an activi...

Multimodal Token Fusion for Vision Transformers

Many adaptations of transformers have emerged to address the single-moda...