Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis

12/18/2022
by   Firas Khader, et al.
0

Multimodal deep learning has been used to predict clinical endpoints and diagnoses from clinical routine data. However, these models suffer from scaling issues: they have to learn pairwise interactions between each piece of information in each data type, thereby escalating model complexity beyond manageable scales. This has so far precluded a widespread use of multimodal deep learning. Here, we present a new technical approach of "learnable synergies", in which the model only selects relevant interactions between data modalities and keeps an "internal memory" of relevant data. Our approach is easily scalable and naturally adapts to multimodal data inputs from clinical routine. We demonstrate this approach on three large multimodal datasets from radiology and ophthalmology and show that it outperforms state-of-the-art models in clinically relevant diagnosis tasks. Our new approach is transferable and will allow the application of multimodal deep learning to a broad set of clinically relevant problems.

READ FULL TEXT

page 19

page 24

page 32

page 33

page 34

research
05/25/2023

Incomplete Multimodal Learning for Complex Brain Disorders Prediction

Recent advancements in the acquisition of various brain data sources hav...
research
03/11/2023

Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review

Cancer has relational information residing at varying scales, modalities...
research
06/17/2022

Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis

Alzheimer's Disease (AD) is the most common neurodegenerative disorder w...
research
02/23/2023

Quantifying Modeling Feature Interactions: An Information Decomposition Framework

The recent explosion of interest in multimodal applications has resulted...
research
07/11/2023

One-Versus-Others Attention: Scalable Multimodal Integration

Multimodal learning models have become increasingly important as they su...
research
02/09/2022

MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis

Recent studies show that deep learning models achieve good performance o...
research
12/31/2022

Depression Diagnosis and Analysis via Multimodal Multi-order Factor Fusion

Depression is a leading cause of death worldwide, and the diagnosis of d...

Please sign up or login with your details

Forgot password? Click here to reset