Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision

04/01/2023
by   Gregory Holste, et al.
6

Prostate cancer is one of the leading causes of cancer-related death in men worldwide. Like many cancers, diagnosis involves expert integration of heterogeneous patient information such as imaging, clinical risk factors, and more. For this reason, there have been many recent efforts toward deep multimodal fusion of image and non-image data for clinical decision tasks. Many of these studies propose methods to fuse learned features from each patient modality, providing significant downstream improvements with techniques like cross-modal attention gating, Kronecker product fusion, orthogonality regularization, and more. While these enhanced fusion operations can improve upon feature concatenation, they often come with an extremely high learning capacity, meaning they are likely to overfit when applied even to small or low-dimensional datasets. Rather than designing a highly expressive fusion operation, we propose three simple methods for improved multimodal fusion with small datasets that aid optimization by generating auxiliary sources of supervision during training: extra supervision, clinical prediction, and dense fusion. We validate the proposed approaches on prostate cancer diagnosis from paired histopathology imaging and tabular clinical features. The proposed methods are straightforward to implement and can be applied to any classification task with paired image and non-image data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/18/2023

Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction

Cancer prognosis and survival outcome predictions are crucial for therap...
research
01/18/2021

Feature Fusion of Raman Chemical Imaging and Digital Histopathology using Machine Learning for Prostate Cancer Detection

The diagnosis of prostate cancer is challenging due to the heterogeneity...
research
04/26/2022

RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy

The current cancer treatment practice collects multimodal data, such as ...
research
03/25/2022

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

The rapid development of diagnostic technologies in healthcare is leadin...
research
12/18/2019

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

Cancer diagnosis, prognosis and therapeutic response predictions are bas...
research
02/05/2020

Multimodal fusion of imaging and genomics for lung cancer recurrence prediction

Lung cancer has a high rate of recurrence in early-stage patients. Predi...
research
06/16/2020

Interpretable multimodal fusion networks reveal mechanisms of brain cognition

Multimodal fusion benefits disease diagnosis by providing a more compreh...

Please sign up or login with your details

Forgot password? Click here to reset