A Cross-Stitch Architecture for Joint Registration and Segmentation in Adaptive Radiotherapy

by   Laurens Beljaards, et al.

Recently, joint registration and segmentation has been formulated in a deep learning setting, by the definition of joint loss functions. In this work, we investigate joining these tasks at the architectural level. We propose a registration network that integrates segmentation propagation between images, and a segmentation network to predict the segmentation directly. These networks are connected into a single joint architecture via so-called cross-stitch units, allowing information to be exchanged between the tasks in a learnable manner. The proposed method is evaluated in the context of adaptive image-guided radiotherapy, using daily prostate CT imaging. Two datasets from different institutes and manufacturers were involved in the study. The first dataset was used for training (12 patients) and validation (6 patients), while the second dataset was used as an independent test set (14 patients). In terms of mean surface distance, our approach achieved 1.06 ± 0.3 mm, 0.91 ± 0.4 mm, 1.27 ± 0.4 mm, and 1.76 ± 0.8 mm on the validation set and 1.82 ± 2.4 mm, 2.45 ± 2.4 mm, 2.45 ± 5.0 mm, and 2.57 ± 2.3 mm on the test set for the prostate, bladder, seminal vesicles, and rectum, respectively. The proposed multi-task network outperformed single-task networks, as well as a network only joined through the loss function, thus demonstrating the capability to leverage the individual strengths of the segmentation and registration tasks. The obtained performance as well as the inference speed make this a promising candidate for daily re-contouring in adaptive radiotherapy, potentially reducing treatment-related side effects and improving quality-of-life after treatment.


page 4

page 7

page 11


Joint Registration and Segmentation via Multi-Task Learning for Adaptive Radiotherapy of Prostate Cancer

Medical image registration and segmentation are two of the most frequent...

Patient-Specific Finetuning of Deep Learning Models for Adaptive Radiotherapy in Prostate CT

Contouring of the target volume and Organs-At-Risk (OARs) is a crucial s...

KidneyRegNet: A Deep Learning Method for 3DCT-2DUS Kidney Registration during Breathing

This work proposed a novel deep registration pipeline for 3D CT and 2D U...

3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations

We propose a supervised nonrigid image registration method, trained usin...

Segmentation of Arterial Walls in Intravascular Ultrasound Cross-Sectional Images Using Extremal Region Selection

Intravascular Ultrasound (IVUS) is an intra-operative imaging modality t...

Elastic registration based on compliance analysis and biomechanical graph matching

An automatic elastic registration method suited for vascularized organs ...

Three-dimensional reconstruction and characterization of bladder deformations

Background and Objective: Pelvic floor disorders are prevalent diseases ...

Please sign up or login with your details

Forgot password? Click here to reset