Deep Equilibrium Multimodal Fusion

by   Jinhong Ni, et al.

Multimodal fusion integrates the complementary information present in multiple modalities and has gained much attention recently. Most existing fusion approaches either learn a fixed fusion strategy during training and inference, or are only capable of fusing the information to a certain extent. Such solutions may fail to fully capture the dynamics of interactions across modalities especially when there are complex intra- and inter-modality correlations to be considered for informative multimodal fusion. In this paper, we propose a novel deep equilibrium (DEQ) method towards multimodal fusion via seeking a fixed point of the dynamic multimodal fusion process and modeling the feature correlations in an adaptive and recursive manner. This new way encodes the rich information within and across modalities thoroughly from low level to high level for efficacious downstream multimodal learning and is readily pluggable to various multimodal frameworks. Extensive experiments on BRCA, MM-IMDB, CMU-MOSI, SUN RGB-D, and VQA-v2 demonstrate the superiority of our DEQ fusion. More remarkably, DEQ fusion consistently achieves state-of-the-art performance on multiple multimodal benchmarks. The code will be released.


page 1

page 2

page 3

page 4


Alternative Telescopic Displacement: An Efficient Multimodal Alignment Method

Feature alignment is the primary means of fusing multimodal data. We pro...

Sparse Fusion for Multimodal Transformers

Multimodal classification is a core task in human-centric machine learni...

Multimodal Fusion Interactions: A Study of Human and Automatic Quantification

Multimodal fusion of multiple heterogeneous and interconnected signals i...

Clarifying the Half Full or Half Empty Question: Multimodal Container Classification

Multimodal integration is a key component of allowing robots to perceive...

DM^2S^2: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention

There is increasing interest in the use of multimodal data in various we...

AutoFraudNet: A Multimodal Network to Detect Fraud in the Auto Insurance Industry

In the insurance industry detecting fraudulent claims is a critical task...

Multimodal Hyperspectral Image Classification via Interconnected Fusion

Existing multiple modality fusion methods, such as concatenation, summat...

Please sign up or login with your details

Forgot password? Click here to reset