Log In Sign Up

MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records

by   Zhen Xu, et al.

One important challenge of applying deep learning to electronic health records (EHR) is the complexity of their multimodal structure. EHR usually contains a mixture of structured (codes) and unstructured (free-text) data with sparse and irregular longitudinal features – all of which doctors utilize when making decisions. In the deep learning regime, determining how different modality representations should be fused together is a difficult problem, which is often addressed by handcrafted modeling and intuition. In this work, we extend state-of-the-art neural architecture search (NAS) methods and propose MUltimodal Fusion Architecture SeArch (MUFASA) to simultaneously search across multimodal fusion strategies and modality-specific architectures for the first time. We demonstrate empirically that our MUFASA method outperforms established unimodal NAS on public EHR data with comparable computation costs. In addition, MUFASA produces architectures that outperform Transformer and Evolved Transformer. Compared with these baselines on CCS diagnosis code prediction, our discovered models improve top-5 recall from 0.88 to 0.91 and demonstrate the ability to generalize to other EHR tasks. Studying our top architecture in depth, we provide empirical evidence that MUFASA's improvements are derived from its ability to both customize modeling for each data modality and find effective fusion strategies.


page 1

page 2

page 3

page 4


BM-NAS: Bilevel Multimodal Neural Architecture Search

Deep neural networks (DNNs) have shown superior performances on various ...

Using Neural Architecture Search for Improving Software Flaw Detection in Multimodal Deep Learning Models

Software flaw detection using multimodal deep learning models has been d...

MFAS: Multimodal Fusion Architecture Search

We tackle the problem of finding good architectures for multimodal class...

Deep Multimodal Neural Architecture Search

Designing effective neural networks is fundamentally important in deep m...

Structure Optimization for Deep Multimodal Fusion Networks using Graph-Induced Kernels

A popular testbed for deep learning has been multimodal recognition of h...

An Approach for Combining Multimodal Fusion and Neural Architecture Search Applied to Knowledge Tracing

Knowledge Tracing is the process of tracking mastery level of different ...

Graph Convolutional Transformer: Learning the Graphical Structure of Electronic Health Records

Effective modeling of electronic health records (EHR) is rapidly becomin...