Modality-based Factorization for Multimodal Fusion

11/30/2018
by   Elham J. Barezi, et al.
0

We propose a multimodal data fusion method by obtaining a M+1 dimensional tensor to consider the high-order relationship between M modalities and the output layer of a neural network model. Applying a modality-based tensor factorization method, which adopts different factors for different modalities, results in removing the redundant information with respect to model outputs and leads to fewer model parameters with minimal loss of performance. This factorization method works as a regularizer which leads to a less complicated model and avoids overfitting. In addition, a modality-based factorization approach helps to understand the amount of useful information in each modality. We have applied this method to three different multimodal datasets in sentiment analysis, personality trait recognition, and emotion recognition. The results demonstrate that the approach yields a 1% to 4% improvement on several evaluation measures compared to the state-of-the-art for all three tasks.

READ FULL TEXT
research
08/25/2022

Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis

Multimodal sentiment analysis is an important research task to predict t...
research
04/22/2013

Bayesian crack detection in ultra high resolution multimodal images of paintings

The preservation of our cultural heritage is of paramount importance. Th...
research
12/31/2022

Depression Diagnosis and Analysis via Multimodal Multi-order Factor Fusion

Depression is a leading cause of death worldwide, and the diagnosis of d...
research
07/04/2020

Low Rank Fusion based Transformers for Multimodal Sequences

Our senses individually work in a coordinated fashion to express our emo...
research
05/23/2023

Cross-Attention is Not Enough: Incongruity-Aware Hierarchical Multimodal Sentiment Analysis and Emotion Recognition

Fusing multiple modalities for affective computing tasks has proven effe...
research
08/01/2023

Multi-Modality Multi-Loss Fusion Network

In this work we investigate the optimal selection and fusion of features...
research
11/11/2020

Improving Multimodal Accuracy Through Modality Pre-training and Attention

Training a multimodal network is challenging and it requires complex arc...

Please sign up or login with your details

Forgot password? Click here to reset