BM-NAS: Bilevel Multimodal Neural Architecture Search

04/19/2021
by   Yihang Yin, et al.
0

Deep neural networks (DNNs) have shown superior performances on various multimodal learning problems. However, it often requires huge efforts to adapt DNNs to individual multimodal tasks by manually engineering unimodal features and designing multimodal feature fusion strategies. This paper proposes Bilevel Multimodal Neural Architecture Search (BM-NAS) framework, which makes the architecture of multimodal fusion models fully searchable via a bilevel searching scheme. At the upper level, BM-NAS selects the inter/intra-modal feature pairs from the pretrained unimodal backbones. At the lower level, BM-NAS learns the fusion strategy for each feature pair, which is a combination of predefined primitive operations. The primitive operations are elaborately designed and they can be flexibly combined to accommodate various effective feature fusion modules such as multi-head attention (Transformer) and Attention on Attention (AoA). Experimental results on three multimodal tasks demonstrate the effectiveness and efficiency of the proposed BM-NAS framework. BM-NAS achieves competitive performances with much less search time and fewer model parameters in comparison with the existing generalized multimodal NAS methods.

READ FULL TEXT

page 3

page 6

page 12

page 13

research
01/22/2022

NAS-VAD: Neural Architecture Search for Voice Activity Detection

The need for automatic design of deep neural networks has led to the eme...
research
04/25/2020

Deep Multimodal Neural Architecture Search

Designing effective neural networks is fundamentally important in deep m...
research
02/03/2021

MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records

One important challenge of applying deep learning to electronic health r...
research
02/12/2023

Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia

Alzheimer's dementia (AD) affects memory, thinking, and language, deteri...
research
09/12/2023

Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices

The recent surge of interest surrounding Multimodal Neural Networks (MM-...
research
04/26/2022

Multi stain graph fusion for multimodal integration in pathology

In pathology, tissue samples are assessed using multiple staining techni...
research
02/12/2021

Neural Architecture Search as Program Transformation Exploration

Improving the performance of deep neural networks (DNNs) is important to...

Please sign up or login with your details

Forgot password? Click here to reset