Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data

by   Nathaniel Braman, et al.

Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H E slide images and/or DNA sequencing), and clinical variables into a comprehensive multimodal risk score. Prognostic embeddings from each modality are learned and combined via attention-gated tensor fusion. To maximize the information gleaned from each modality, we introduce a multimodal orthogonalization (MMO) loss term that increases model performance by incentivizing constituent embeddings to be more complementary. DOF predicts OS in glioma patients with a median C-index of 0.788 +/- 0.067, significantly outperforming (p=0.023) the best performing unimodal model with a median C-index of 0.718 +/- 0.064. The prognostic model significantly stratifies glioma patients by OS within clinical subsets, adding further granularity to prognostic clinical grading and molecular subtyping.



There are no comments yet.


page 4


Predicting Brain Degeneration with a Multimodal Siamese Neural Network

To study neurodegenerative diseases, longitudinal studies are carried on...

Gated Multimodal Units for Information Fusion

This paper presents a novel model for multimodal learning based on gated...

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

Cancer diagnosis, prognosis and therapeutic response predictions are bas...

Modality-based Factorization for Multimodal Fusion

We propose a multimodal data fusion method by obtaining a M+1 dimensiona...

Prediction of overall survival and molecular markers in gliomas via analysis of digital pathology images using deep learning

Cancer histology reveals disease progression and associated molecular pr...

Prediction of Thrombectomy FunctionalOutcomes using Multimodal Data

Recent randomised clinical trials have shown that patients with ischaemi...

M2P2: Multimodal Persuasion Prediction using Adaptive Fusion

Identifying persuasive speakers in an adversarial environment is a criti...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.