DeepAI AI Chat
Log In Sign Up

Are Multimodal Transformers Robust to Missing Modality?

by   Mengmeng Ma, et al.
University of Delaware
Snap Inc.

Multimodal data collected from the real world are often imperfect due to missing modalities. Therefore multimodal models that are robust against modal-incomplete data are highly preferred. Recently, Transformer models have shown great success in processing multimodal data. However, existing work has been limited to either architecture designs or pre-training strategies; whether Transformer models are naturally robust against missing-modal data has rarely been investigated. In this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of Transformers in the presence of modal-incomplete data. Unsurprising, we find Transformer models are sensitive to missing modalities while different modal fusion strategies will significantly affect the robustness. What surprised us is that the optimal fusion strategy is dataset dependent even for the same Transformer model; there does not exist a universal strategy that works in general cases. Based on these findings, we propose a principle method to improve the robustness of Transformer models by automatically searching for an optimal fusion strategy regarding input data. Experimental validations on three benchmarks support the superior performance of the proposed method.


page 1

page 2

page 3

page 4


Multimodal Prompting with Missing Modalities for Visual Recognition

In this paper, we tackle two challenges in multimodal learning for visua...

Fusion of Satellite Images and Weather Data with Transformer Networks for Downy Mildew Disease Detection

Crop diseases significantly affect the quantity and quality of agricultu...

TFusion: Transformer based N-to-One Multimodal Fusion Block

People perceive the world with different senses, such as sight, hearing,...

Multimodal Token Fusion for Vision Transformers

Many adaptations of transformers have emerged to address the single-moda...

Real Estate Attribute Prediction from Multiple Visual Modalities with Missing Data

The assessment and valuation of real estate requires large datasets with...

Multimodal Learning with Transformers: A Survey

Transformer is a promising neural network learner, and has achieved grea...

Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis

With the proliferation of user-generated online videos, Multimodal Senti...