Deformable Image Registration (DIR) is a challenging task and active field of research in medical image analysis . Its main application is fusion of the information acquired by the different modalities to facilitate diagnosis and treatment planning . For example, in radiotherapy treatment planning Magnet Resonance (MR) images are used to segment the tumor and organs at risk, while the tissue density information provided by the corresponding Computer Tomography (CT) image is used for dose planning 
. CT and MR images are acquired using separate devices and often on different days. Therefore the patient will not be in exactly the same posture and the position of inner organs might change, due to respiration, peristalsis, bladder filling, gravity, etc. Thus, DIR is needed. The main difficulty of MR-CT DIR is the definition of an image similarity measure, which reliably quantifies the local image alignment for optimizing the many free parameters of the spatial transformation. This is an inherent problem as multi-modal images are acquired because they provide complementary information.
Multi-modal similarity measures. The main voxel-wise multi-modal image (dis)similarity measures are (i) statistical measures that use intensity information directly and try to maximize (non-linear) statistical dependencies between the intensities of the images (e.g. Normalized Mutual Information (NMI) , MI ), and (ii) structural measures based on structural representations that try to be invariant to different modalities (e.g. normalized gradient fields , entropy images , Modality Independent Neighborhood Descriptor (MIND) ).
Intensity remapping. The drawback of structural representations is that all unstructured (e.g. homogenous) regions are mapped to the same representation regardless of their original intensity. To avoid this information reduction, methods to directly re-map intensities have been proposed [8, 6, 9]
. The joint histogram of the coarsely registered images was employed to remap the intensities of both images to a common modality based on the least conditional variance to remove structures not visible in both images. Assuming that the global self-similarity of the images (i.e. the similarities between all image patches) is preserved across modalities, intensities were mapped into a 1D Laplacien Eigenmap based on patch intensity similarity 
. A k-means clustering based binning scheme, to remap spatially unconnected components with similar intensities to distinct intensities, was proposed in for retina images. While these intensity-remappings provide some improvements, they are simplifications to the underlying complex relationship between the intensity of the two modalities.
Learning from paired data.
Given aligned multi-modal training data, attempts have been made to learn this complex relationship. The last layer of a deep neural network (DNN) classifier, which discriminates between matching and not matching patches, was used to directly learn the similarity measure
. The DNN was initialized by a stacked denoised autoencoder, where the lower layers were separately trained per modality to get modality-dependent features. It was observed that the learned CT filters look mostly like edge-detectors, while the MR filters detect more complex texture features. In
the expected joint intensity distribution was learned from co-registered images. The dissimilarity measure was then based on the Bhattacharyya distance between the expected and observed distribution. Machine learning has been used to learn how to map one modality to the other. synthesized CT from MR brain images by matching MR patches to an atlas (created from co-registered MR and CT images) and augmented these by considering all convex patch combinations. 
proposed a bi-directional image synthesis approach for non-rigid registration of the pelvic area, where random forests are trained on Haar-like features extracted from pairs of pre-aligned CT and MR patches. An auto-context model was used to incorporate neighboring prediction results. All these learning-based approaches depend on co-registered multi-modal images for training. This is very difficult for deforming structures as dense (voxel-wise) spatial correspondences are required and CT and MR images cannot be acquired simultaneously yet.
Learning without paired data. A cross-modality synthesis method which does not require paired data was proposed in . It is based on generating multiple target modality candidate values for each source voxel independently using cross-modal nearest neighbor search. A global solution is then found by simultaneously maximizing global MI and local spatial consistency. Finally, a coupled sparse representation was used to further refine the synthesized images. When applied to T1/T2 brain MRIs, T1 images were better synthesized than T2 images (0.93 vs. 0.85 correlation to ground truth). Extending the method to a supervised setting outperformed state-of-the-art supervised methods slightly.
Recently cyclic-consistent Generative Adversarial Networks (cycle-GANs) were proposed for learning an image-to-image mapping between two domains (&) from unpaired datasets . The method is based on two generator networks ( to synthesize image from , ) and two discriminator networks (, ). Besides the usual discriminator loss to differentiate synthesized and real images (e.g. ), a cycle loss was introduced which measures the difference between the real image and its twice synthesized image, e.g. . Good performances were shown for various domain translation tasks like labels to photos and arial photos to maps. Very recently, this promising approach was employed for slice-wise synthesizing CT from MR head images from unpaired data . It achieved lower mean squared errors (74 vs. 89 HU) than when training the same generator network on rigidly aligned MR and CT data . It was reasoned that this could be due to misalignments, as the images contained also deforming structures (e.g. neck, mouth). Cycle-GANs were used for synthesis of MR from unpaired CT images for enriching a cardiac dataset for training thereafter a segmentation network . A view alignment step using the segmentations was incorporated to make the layout (e.g. position, size of anatomy) of the CT and MR images similar, such that the discriminator cannot use the layout to differentiate between them. Furthermore the myocardium mask for both modalities was provided during training, as the cycle-GAN not only changed the intensities but also anatomical locations such that the mask was no longer in correspondence with the image. Hence this is not a completely unsupervised approach. Similarly, a shape-consistency loss from segmentations was incorporated in  to avoid geometric changes between the real and synthesized images. It was argued that ”from the discriminator perspective, geometric transformations do not change the realness of synthesized images since the shape of training data is arbitrary”. However this does not hold if there is a geometric bias between the two datasets.
Synthesized MR PD/T1 brain images via patch matching were shown to be useful for segmentation and inter-modality cross-subject registration . If this also holds for MR-CT synthesis via cycle-GANs for thoracic and abdominal regions has not yet been studied. Our contributions include (i) combining two state-of-the-art multi-modal DIR similarity measures (NMI, MIND), (ii) studying the effect of the image region size on the consistency of the synthesized 3D images, and (iii) evaluating the usefulness of synthesized images for deformable registration of CT and MR images from the thorax and abdomen against a strong baseline.
We used 17 unpaired and 3 paired 3D MR-CT images from the VISCERAL Anatomy3 benchmark training set (unenhanced, whole body, MR-T1) and their gold standard segmentations for evaluation . The 3 subjects with paired data had IDs 10000021, 10000067 and 10000080. All MRIs were bias field corrected using the N4ITK method . All images were resampled to an isotropic resolution of 1.25 mm. This was motivated by the image resolution of the original MRIs being 1.2561.25 mm in left-right, posterior-anterior and superior-inferior direction. The CT images had a resolution between 0.80.81.5 mm and 1.01.01.5 mm.
To reduce memory requirements, we automatically extracted from each image two regions such that the included gold standard segmentations were at least 5 mm away from the inferior and superior region boundary. The thorax region covered the segmentations of the liver, spleen, gallbladder, and right and left lung. The abdominal region contained the bladder, lumbar vertebra 1, right and left kidney, and right and left psoas major muscle, see Figs. 2, 1, left column.
Closer investigation of poor performing registration results showed that for case 10000067 the segmentation labels of the right and left kidney were swapped in the MRI. Additionally, for 10000080 the segmentation of the lumbar vertebra 1 in the MRI seems to be that of lumbar vertebra 2. We corrected these kidney annotations and excluded this lumbar vertebra 1 segmentations from the results.
3.1 Image Synthesis
Cycle-GAN. For image synthesis, we followed the cycle-GAN network architecture as described in [16, 2], starting from an existing implementation111https://github.com/xhujoy/CycleGAN-tensorflow. In short, the two generators (,
) are 2D fully convolutional networks with 9 residual blocks and two fractionally strided convolution layers (res-net). The discriminators (, ) are fully convolutional architectures to classify overlapping image patches as real or fake (PatchGAN) 222The discriminators consist of 5 convolutions layers (I256-C128-C64-C32-C32, stride length 2-2-2-1-1, 44 kernels) for and 4 layers (I256-C128-C64-C64, stride length 2-2-1-1) for .
The networks take input images of size 256256 pixels and channels. Larger-sized test images were synthesized from the average result of 256256 regions extracted with a stride length of . The cycle-GAN was optimized to reduce the overall loss , which is a weighted sum of the discriminator losses , and the generator cyclic loss :
3.2 Image Registration
Rigid Registration. The CT and MR images were first rigidly registered using the function imregister from the MATLAB Image Processing Toolbox , set to multi-modal configuration (Mattes Mutual Information, one-plus-one evolutionary optimization). These rigid registration results were then used as starting points for all subsequent deformable image registrations.
Deformable Registration - MIND. The so-called modality independent neighborhood descriptor (MIND) was proposed as dissimilarity measure for multi-modal DIR . MIND is based on a multi-dimensional descriptor per voxel , which captures the self-similarity of the image patch around (denoted as ) with the patches in a local neighborhood of . The single entries are calculated by a Gaussian function
where is a normalization constant such that the maximum value in is 1, defines the patch dissimilarity with Gaussian kernel of the same size as patch and the half-size of the patch being equal to . is the variance of a six-neighborhood search region. is calculated in a dense fashion for each image independently. The dissimilarity of images and is finally defined by with
In the MIND registration framework, the images are downsampled via Gaussian pyramids and the deformation field is regularized via the squared L2-norm. Additionally after each Gauss-Newton update step during optimization, each deformation field is replaced by combining half of its own transformation with half of the inverse transformation of the other deformation field (see [7, 25]) to obtain diffeomorphic transformations. We used the provided code  and compared results after integrating the MIND measure in our DIR method .
Deformable Registration - ourDIR.
We extended our DIR method, based a linearly interpolated grid of control points and various displacement regularization measures, to incorporate the multi-modal (dis)similarity measures normalized mutual information (NMI) and MIND, and their combination NMI+MIND.
Given fixed image , moving image , displacements at the control points, and interpolation function to get dense displacements, the NMI dissimilarity is defined by , with marginal entropies , and joint entropy computed from intensity histograms with 100 equally-spaced bins between the 0.5 and 99.5 percentiles of the image intensity. The gradients of with respect to are calculated as described in 
. To avoid infinity gradients, we replace zero probabilities with, where is the number of image voxels. We combined the dissimilarities NMI and MIND by
where is a scaling parameter to get in the same range  and is a weighting term. The choice of is not trivial, as the magnitude of change per dissimilarity measure from initial to ideal knot displacements ) is unknown. We tested 3 strategies, namely (i) using a fixed parameter , (ii) using the initial gradient magnitude via
or (iii) basing it on the change in dissimilarity during registration:
The final cost function is where regularizes the displacements at the control points by their TV or L2 norm.
4 Experiments and Results
4.1 Image Synthesis
Images intensities were linearly scaled to [0,255]. Suitable image regions were cropped to fit the network size instead of resizing the image in-plane , as resizing lead to distortions in the synthesized images due to systematic differences in image size between the two modalities. Image regions (ROIs) of size 286286, for were extracted randomly from the training data. Dark ROIs, with a mean intensity of less than 1/4 of that of the ROI in the center of the 3D image, were not selected to avoid including a lot of background. ROIs were further randomly cropped to 256256
during network training. A Cycle-GANs was trained for 200 epochs per image region (thorax or abdomen). We used a training regime as previously reported, namely Adam optimizer, learning rate fixed to 0.0002 for 1-100 epochs and linearly reduced to 0 for 101-200 epochs, . 3D test images were created using an in-plane stride length of and a channel stride length of for and for .
4.2 Image Registration
MR-CT deformable image registration based on image multi-modal similarity measure MIND, NMI, or NMI+MIND was compared with image synthesis and then deformable registration using local NCC as image similarity. All registrations used ourDIR framework with total variation regularization. Registration parameters were optimized via grid search. These are the weighting of regularization term , the control point spacing pixels, and the number of multi-resolution levels . The best strategy for combining NMI and MIND was using the initial gradient magnitude, i.e. Eq. (8), and .
Example of synthesized images are shown in Figs. 1-2. Inconsistency across slices can be seen when ROIs with few slices (=) are used. Synthesized image structures do mostly not adhere to the contours of the lung segmentations from the real image. To be realistic, the generators had to learned the substantial bias in lung volume between the two modalities, see Table 1. CT images were generally acquired in end-inhale state, while MRIs in end-exhale. Changing the patchGAN discriminator to a shallower architecture, such that each output node has a receptor field of 3434 instead of 7070 could sometimes reduced the misalignment of the lung segmentations (e.g. Figs. 1d), but was less powerful in image synthesis (e.g. region between the lungs in Fig.2d).
The performance of the deformable image registration for the original images (CT, MR) and the cycle-GAN synthesized images are listed in Table 2. While synthesized images can achieve a similar overlap than multi-modal NMI registration for the abdomen (77.4 vs. 78.8%), they are substantially worse for the thorax due to the bias in lung volume. Performance of synthesized CT images, which were more affected by synthesized volume lung volume changes, was generally lower than for synthesized MRIs. Results are shown in Fig. 3.
|CT-MR||Synthesized MR||Synthesized CT|
We combined two multi-modal image similarity measure (NMI, MIND) and observed a similar performance as when using only NMI, and in contrast to  no improvement of MIND over NMI.
We investigated the usefulness of a fully unsupervised MR-CT image modality synthesis method for deformable image registration of MR and CT images. Against the established multi-modal deformable registration methods, synthesizing images via cycle-GAN and then using a robust mono-modal image similarity measure achieved at best a similar performance. In particular one has to be careful to have collections of the two image modalities which are balanced, i.e. not biased for a modality, as such differences are readily synthesized by the cycle-GAN framework. Ensuring that synthesized images are truly in spatial correspondence with the source image would require incorporating a deformable image registration into the cycle-GAN.
Acknowledgments: We thank the EU’s 7th Framework Program (Agreement No. 611889, TRANS-FUSIMO) for funding and acknowledge NVIDIA for GPU support.
-  Sotiras, A., Davatzikos, C., Paragios, N.: Deformable medical image registration: A survey. IEEE Trans Med Imag 32(7) (2013) 1153–1190
-  Wolterink, J., Dinkla, A., Savenije, M., Seevinck, P., van den Berg, C., Išgum, I.: Deep mr to ct synthesis using unpaired data. In: Int Workshop on Simulation and Synthesis in Medical Imaging, Springer (2017) 14–23
-  Studholme, C., Hill, D., Hawkes, D.: An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognition 32(1) (1999(32)) 71–86
-  Pluim, J., Maintz, J., Viergever, M.: Image registration by maximization of combined mutual information and gradient information. In: Medical Image Computing and Computer-Assisted Intervention, Springer (2000) 452–461
-  Haber, E., Modersitzki, J.: Intensity gradient based registration and fusion of multi-modal images. Medical Image Computing and Computer-Assisted Intervention (2006) 726–733
-  Wachinger, C., Navab, N.: Entropy and Laplacian images: Structural representations for multi-modal registration. Med Image Anal 16(1) (2012) 1–17
-  Heinrich, M., Jenkinson, M., Bhushan, M., Matin, T., Gleeson, F., Brady, M., Schnabel, J.: MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration. Med Image Anal 16(7) (2012) 1423–1435
-  Andronache, A., von Siebenthal, M., Székely, G., Cattin, P.: Non-rigid registration of multi-modal images using both mutual information and cross-correlation. Med Image Anal 12(1) (2008) 3–15
-  Knops, Z., Maintz, J., Viergever, M., Pluim, J.: Registration using segment intensity remapping and mutual information. In: Medical Image Computing and Computer-Assisted Intervention, Springer (2004) 805–812
-  Cheng, X., Zhang, L., Zheng, Y.: Deep similarity learning for multimodal medical images. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization (2016) 1–5
-  So, R., Chung, A.: A novel learning-based dissimilarity metric for rigid and non-rigid medical image registration by using Bhattacharyya distances. Pattern Recognition 62 (2017) 161–174
-  Roy, S., Carass, A., Jog, A., Prince, J., Lee, J.: MR to CT registration of brains using image synthesis. In: Proc SPIE. Volume 9034., NIH Public Access (2014)
-  Cao, X., Yang, J., Gao, Y., Guo, Y., Wu, G., Shen, D.: Dual-core steered non-rigid registration for multi-modal images via bi-directional image synthesis. Med Image Anal (2017)
-  Liu, Y.H., Sinusas, A.: Hybrid Imaging in Cardiovascular Medicine. CRC Press (2017)
-  Vemulapalli, R., Van Nguyen, H., Kevin Zhou, S.: Unsupervised cross-modal synthesis of subject-specific scans. In: ICCV. (2015) 630–638
-  Zhu, J.Y., Park, T., Isola, P., Efros, A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv:1703.10593 (2017)
-  Nie, D., Trullo, R., Lian, J., Petitjean, C., Ruan, S., Wang, Q., Shen, D.: Medical image synthesis with context-aware generative adversarial networks. In: Medical Image Computing and Computer-Assisted Intervention, Springer (2017) 417–425
-  Chartsias, A., Joyce, T., Dharmakumar, R., Tsaftaris, S.: Adversarial image synthesis for unpaired multi-modal cardiac data. In: Int Workshop on Simulation and Synthesis in Medical Imaging, Springer (2017) 3–13
-  Zhang, Z., Yang, L., Zheng, Y.: Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2018) 9242–9251
-  Iglesias, J., Konukoglu, E., Zikic, D., Glocker, B., Van Leemput, K., Fischl, B.: Is synthesizing MRI contrast useful for inter-modality analysis? In: Medical Image Computing and Computer-Assisted Intervention, Springer (2013) 631–638
-  Jimenez-del Toro, O., Müller, H., Krenn, M., Gruenberg, K., Taha, A., Winterstein, M., Eggel, I., Foncubierta-Rodríguez, A., Goksel, O., Jakab, A., et al.: Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: VISCERAL anatomy benchmarks. IEEE Trans Med Imag 35(11) (2016) 2459–2475
-  Tustison, N., Avants, B., Cook, P., Zheng, Y., Egan, A., Yushkevich, P., Gee, J.: N4ITK: improved N3 bias correction. IEEE Trans Med Imag 29(6) (2010) 1310–1320
-  Isola, P., Zhu, J.Y., Zhou, T., Efros, A.: Image-to-image translation with conditional adversarial networks. arXiv:1611.07004 (2016)
-  MathWorks: Matlab imregister documentation. https://ch.mathworks.com/help/images/ref/imregister.html (2017) Online. Accessed: 04-December-2017.
-  Avants, B., Epstein, C., Grossman, M., Gee, J.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal 12(1) (2008) 26–41
-  Heinrich, M.: Symmetric Gauss-Newton deformable registration code. http://www.ibme.ox.ac.uk/research/biomedia/julia-schnabel/files/symgn.zip (2012) note = ”[Online; accessed 26-November-2017]”.
-  Vishnevskiy, V., Gass, T., Szekely, G., Tanner, C., Goksel, O.: Isotropic total variation regularization of displacements in parametric image registration. IEEE Trans Med Imag 36(2) (2017) 385–395
-  Crum, W., Hill, D., Hawkes, D.: Information theoretic similarity measures in non-rigid registration. In: Information Processing in Medical Imaging, Springer (2003) 378–387
-  Lundqvist, R., Bengtsson, E., Thurfjell, L.: A combined intensity and gradient-based similarity criterion for interindividual SPECT brain scan registration. EURASIP Journal on Advances in Signal Processing 2003(5) (2003) 967364