Defending Multimodal Fusion Models against Single-Source Adversaries

06/25/2022
by   Karren Yang, et al.
0

Beyond achieving high performance across many vision tasks, multimodal models are expected to be robust to single-source faults due to the availability of redundant information between modalities. In this paper, we investigate the robustness of multimodal neural networks against worst-case (i.e., adversarial) perturbations on a single modality. We first show that standard multimodal fusion models are vulnerable to single-source adversaries: an attack on any single modality can overcome the correct information from multiple unperturbed modalities and cause the model to fail. This surprising vulnerability holds across diverse multimodal tasks and necessitates a solution. Motivated by this finding, we propose an adversarially robust fusion strategy that trains the model to compare information coming from all the input sources, detect inconsistencies in the perturbed modality compared to the other modalities, and only allow information from the unperturbed modalities to pass through. Our approach significantly improves on state-of-the-art methods in single-source robustness, achieving gains of 7.8-25.2 object detection, and 1.6-6.7 performance on unperturbed (i.e., clean) data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/22/2021

Understanding and Measuring Robustness of Multimodal Learning

The modern digital world is increasingly becoming multimodal. Although m...
research
06/11/2019

On Single Source Robustness in Deep Fusion Models

Algorithms that fuse multiple input sources benefit from both complement...
research
08/22/2018

CentralNet: a Multilayer Approach for Multimodal Fusion

This paper proposes a novel multimodal fusion approach, aiming to produc...
research
06/30/2021

Attention Bottlenecks for Multimodal Fusion

Humans perceive the world by concurrently processing and fusing high-dim...
research
04/19/2023

MMDR: A Result Feature Fusion Object Detection Approach for Autonomous System

Object detection has been extensively utilized in autonomous systems in ...
research
04/07/2021

Multimodal Object Detection via Bayesian Fusion

Object detection with multimodal inputs can improve many safety-critical...
research
06/07/2023

Multimodal Fusion Interactions: A Study of Human and Automatic Quantification

Multimodal fusion of multiple heterogeneous and interconnected signals i...

Please sign up or login with your details

Forgot password? Click here to reset