TranSiam: Fusing Multimodal Visual Features Using Transformer for Medical Image Segmentation

04/26/2022
by   Xuejian Li, et al.
0

Automatic segmentation of medical images based on multi-modality is an important topic for disease diagnosis. Although the convolutional neural network (CNN) has been proven to have excellent performance in image segmentation tasks, it is difficult to obtain global information. The lack of global information will seriously affect the accuracy of the segmentation results of the lesion area. In addition, there are visual representation differences between multimodal data of the same patient. These differences will affect the results of the automatic segmentation methods. To solve these problems, we propose a segmentation method suitable for multimodal medical images that can capture global information, named TranSiam. TranSiam is a 2D dual path network that extracts features of different modalities. In each path, we utilize convolution to extract detailed information in low level stage, and design a ICMT block to extract global information in high level stage. ICMT block embeds convolution in the transformer, which can extract global information while retaining spatial and detailed information. Furthermore, we design a novel fusion mechanism based on cross attention and selfattention, called TMM block, which can effectively fuse features between different modalities. On the BraTS 2019 and BraTS 2020 multimodal datasets, we have a significant improvement in accuracy over other popular methods.

READ FULL TEXT

page 2

page 3

page 6

research
04/10/2023

HST-MRF: Heterogeneous Swin Transformer with Multi-Receptive Field for Medical Image Segmentation

The Transformer has been successfully used in medical image segmentation...
research
07/04/2023

H-DenseFormer: An Efficient Hybrid Densely Connected Transformer for Multimodal Tumor Segmentation

Recently, deep learning methods have been widely used for tumor segmenta...
research
01/24/2022

Mutual Attention-based Hybrid Dimensional Network for Multimodal Imaging Computer-aided Diagnosis

Recent works on Multimodal 3D Computer-aided diagnosis have demonstrated...
research
10/22/2022

MS-DC-UNeXt: An MLP-based Multi-Scale Feature Learning Framework For X-ray Images

The advancement of deep learning theory and infrastructure is crucial in...
research
09/07/2023

Multimodal Transformer for Material Segmentation

Leveraging information across diverse modalities is known to enhance per...
research
01/05/2021

Deep Class-Specific Affinity-Guided Convolutional Network for Multimodal Unpaired Image Segmentation

Multi-modal medical image segmentation plays an essential role in clinic...
research
09/03/2019

Hyper-Pairing Network for Multi-Phase Pancreatic Ductal Adenocarcinoma Segmentation

Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal cancer...

Please sign up or login with your details

Forgot password? Click here to reset