Matching in the Wild: Learning Anatomical Embeddings for Multi-Modality Images

by   Xiaoyu Bai, et al.

Radiotherapists require accurate registration of MR/CT images to effectively use information from both modalities. In a typical registration pipeline, rigid or affine transformations are applied to roughly align the fixed and moving images before proceeding with the deformation step. While recent learning-based methods have shown promising results in the rigid/affine step, these methods often require images with similar field-of-view (FOV) for successful alignment. As a result, aligning images with different FOVs remains a challenging task. Self-supervised landmark detection methods like self-supervised Anatomical eMbedding (SAM) have emerged as a useful tool for mapping and cropping images to similar FOVs. However, these methods are currently limited to intra-modality use only. To address this limitation and enable cross-modality matching, we propose a new approach called Cross-SAM. Our approach utilizes a novel iterative process that alternates between embedding learning and CT-MRI registration. We start by applying aggressive contrast augmentation on both CT and MRI images to train a SAM model. We then use this SAM to identify corresponding regions on paired images using robust grid-points matching, followed by a point-set based affine/rigid registration, and a deformable fine-tuning step to produce registered paired images. We use these registered pairs to enhance the matching ability of SAM, which is then processed iteratively. We use the final model for cross-modality matching tasks. We evaluated our approach on two CT-MRI affine registration datasets and found that Cross-SAM achieved robust affine registration on both datasets, significantly outperforming other methods and achieving state-of-the-art performance.


page 2

page 4

page 7

page 13


Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity

Non-rigid inter-modality registration can facilitate accurate informatio...

SAM++: Enhancing Anatomic Matching using Semantic Information and Structural Inference

Medical images like CT and MRI provide detailed information about the in...

SAMConvex: Fast Discrete Optimization for CT Registration using Self-supervised Anatomical Embedding and Correlation Pyramid

Estimating displacement vector field via a cost volume computed in the f...

Accurate and Robust Alignment of Variable-stained Histologic Images Using a General-purpose Greedy Diffeomorphic Registration Tool

Variously stained histology slices are routinely used by pathologists to...

Non-Rigid Image Registration Using Self-Supervised Fully Convolutional Networks without Training Data

A novel non-rigid image registration algorithm is built upon fully convo...

A Hierarchical Descriptor Framework for On-the-Fly Anatomical Location Matching between Longitudinal Studies

We propose a method to match anatomical locations between pairs of medic...

ContraReg: Contrastive Learning of Multi-modality Unsupervised Deformable Image Registration

Establishing voxelwise semantic correspondence across distinct imaging m...

Please sign up or login with your details

Forgot password? Click here to reset