Cross-Modal Information Maximization for Medical Imaging: CMIM

10/20/2020
by   Tristan Sylvain, et al.
0

In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound, etc.) and their associated radiology reports. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time. In this paper, we propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time, using recent advances in mutual information maximization. By maximizing cross-modal information at train time, we are able to outperform several state-of-the-art baselines in two different settings, medical image classification, and segmentation. In particular, our method is shown to have a strong impact on the inference-time performance of weaker modalities.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/03/2020

Robust Latent Representations via Cross-Modal Translation and Alignment

Multi-modal learning relates information across observation modalities o...
research
07/14/2021

Self-Supervised Multi-Modal Alignment for Whole Body Medical Imaging

This paper explores the use of self-supervised deep learning in medical ...
research
01/12/2019

ChronoMID - Cross-Modal Neural Networks for 3-D Temporal Medical Imaging Data

ChronoMID builds on the success of cross-modal convolutional neural netw...
research
03/22/2021

Brain Image Synthesis with Unsupervised Multivariate Canonical CSCℓ_4Net

Recent advances in neuroscience have highlighted the effectiveness of mu...
research
11/08/2020

Cross-Modal Self-Attention Distillation for Prostate Cancer Segmentation

Automatic segmentation of the prostate cancer from the multi-modal magne...
research
05/27/2022

Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed

Failure detection in automated image classification is a critical safegu...
research
08/26/2023

ReFuSeg: Regularized Multi-Modal Fusion for Precise Brain Tumour Segmentation

Semantic segmentation of brain tumours is a fundamental task in medical ...

Please sign up or login with your details

Forgot password? Click here to reset