Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

02/22/2020
by   Cheng Chen, et al.
10

Accurate medical image segmentation commonly requires effective learning of the complementary information from multimodal data. However, in clinical practice, we often encounter the problem of missing imaging modalities. We tackle this challenge and propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code, which uniquely sticks to each modality, and the modality-invariant content code, which absorbs multimodal information for the segmentation task. With enhanced modality-invariance, the disentangled content code from each modality is fused into a shared representation which gains robustness to missing data. The fusion is achieved via a learning-based strategy to gate the contribution of different modalities at different locations. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset. With competitive performance to the state-of-the-art approaches for full modality, our method achieves outstanding robustness under various missing modality(ies) situations, significantly exceeding the state-of-the-art method by over 16 average for Dice on whole tumor segmentation.

READ FULL TEXT

page 7

page 8

research
11/08/2021

Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities

Using multimodal Magnetic Resonance Imaging (MRI) is necessary for accur...
research
11/02/2020

Predicting Brain Degeneration with a Multimodal Siamese Neural Network

To study neurodegenerative diseases, longitudinal studies are carried on...
research
08/26/2022

Learning Multi-Modal Brain Tumor Segmentation from Privileged Semi-Paired MRI Images with Curriculum Disentanglement Learning

Due to the difficulties of obtaining multimodal paired images in clinica...
research
08/19/2019

A unified representation network for segmentation with missing modalities

Over the last few years machine learning has demonstrated groundbreaking...
research
12/14/2022

M-GenSeg: Domain Adaptation For Target Modality Tumor Segmentation With Annotation-Efficient Supervision

Automated medical image segmentation using deep neural networks typicall...
research
08/28/2020

Soft Tissue Sarcoma Co-Segmentation in Combined MRI and PET/CT Data

Tumor segmentation in multimodal medical images has seen a growing trend...
research
03/13/2023

Mirror U-Net: Marrying Multimodal Fission with Multi-task Learning for Semantic Segmentation in Medical Imaging

Positron Emission Tomography (PET) and Computer Tomography (CT) are rout...

Please sign up or login with your details

Forgot password? Click here to reset