CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

by   Yutong Xie, et al.

Convolutional neural networks (CNNs) have been the de facto standard for nowadays 3D medical image segmentation. The convolutional operations used in these networks, however, inevitably have limitations in modeling the long-range dependency due to their inductive bias of locality and weight sharing. Although Transformer was born to address this issue, it suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. In this paper, we propose a novel framework that efficiently bridges a Convolutional neural network and a Transformer (CoTr) for accurate 3D medical image segmentation. Under this framework, the CNN is constructed to extract feature representations and an efficient deformable Transformer (DeTrans) is built to model the long-range dependency on the extracted feature maps. Different from the vanilla Transformer which treats all image positions equally, our DeTrans pays attention only to a small set of key positions by introducing the deformable self-attention mechanism. Thus, the computational and spatial complexities of DeTrans have been greatly reduced, making it possible to process the multi-scale and high-resolution feature maps, which are usually of paramount importance for image segmentation. We conduct an extensive evaluation on the Multi-Atlas Labeling Beyond the Cranial Vault (BCV) dataset that covers 11 major human organs. The results indicate that our CoTr leads to a substantial performance improvement over other CNN-based, transformer-based, and hybrid methods on the 3D multi-organ segmentation task. Code is available at <>


MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet

U-Nets have achieved tremendous success in medical image segmentation. N...

High-Resolution Swin Transformer for Automatic Medical Image Segmentation

The Resolution of feature maps is critical for medical image segmentatio...

C2FTrans: Coarse-to-Fine Transformers for Medical Image Segmentation

Convolutional neural networks (CNN), the most prevailing architecture fo...

Learning from partially labeled data for multi-organ and tumor segmentation

Medical image benchmarks for the segmentation of organs and tumors suffe...

MEW-UNet: Multi-axis representation learning in frequency domain for medical image segmentation

Recently, Visual Transformer (ViT) has been widely used in various field...

CMUNeXt: An Efficient Medical Image Segmentation Network based on Large Kernel and Skip Fusion

The U-shaped architecture has emerged as a crucial paradigm in the desig...

Sar Ship Detection based on Swin Transformer and Feature Enhancement Feature Pyramid Network

With the booming of Convolutional Neural Networks (CNNs), CNNs such as V...

Please sign up or login with your details

Forgot password? Click here to reset