PMR: Prototypical Modal Rebalance for Multimodal Learning

11/14/2022
by   Yunfeng Fan, et al.
0

Multimodal learning (MML) aims to jointly exploit the common priors of different modalities to compensate for their inherent limitations. However, existing MML methods often optimize a uniform objective for different modalities, leading to the notorious "modality imbalance" problem and counterproductive MML performance. To address the problem, some existing methods modulate the learning pace based on the fused modality, which is dominated by the better modality and eventually results in a limited improvement on the worse modal. To better exploit the features of multimodal, we propose Prototypical Modality Rebalance (PMR) to perform stimulation on the particular slow-learning modality without interference from other modalities. Specifically, we introduce the prototypes that represent general features for each class, to build the non-parametric classifiers for uni-modal performance evaluation. Then, we try to accelerate the slow-learning modality by enhancing its clustering toward prototypes. Furthermore, to alleviate the suppression from the dominant modality, we introduce a prototype-based entropy regularization term during the early training stage to prevent premature convergence. Besides, our method only relies on the representations of each modality and without restrictions from model structures and fusion methods, making it with great application potential for various scenarios.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/29/2022

Balanced Multimodal Learning via On-the-fly Gradient Modulation

Multimodal learning helps to comprehensively understand the world, by in...
research
02/14/2023

Balanced Audiovisual Dataset for Imbalance Analysis

The imbalance problem is widespread in the field of machine learning, wh...
research
09/07/2023

Multi-Modality Guidance Network For Missing Modality Inference

Multimodal models have gained significant success in recent years. Stand...
research
10/17/2022

MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion

Multimodal knowledge graph completion (MKGC) aims to predict missing ent...
research
01/12/2023

Multimodal Deep Learning

This book is the result of a seminar in which we reviewed multimodal app...
research
04/05/2023

Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping

Jointly harnessing complementary features of multi-modal input data in a...
research
02/24/2023

Revisiting Modality Imbalance In Multimodal Pedestrian Detection

Multimodal learning, particularly for pedestrian detection, has recently...

Please sign up or login with your details

Forgot password? Click here to reset