Radiologists often diagnose the progress of disease by comparing medical images at different temporal phases. In case of diagnosis of liver tumor such as hepatocellular carcinoma (HCC), the contrast of normal liver tissue and tumor region in contrast enhanced CT (CECT) distinctly varies before and after the infection of contrast agent. This provides radiologists an important clue to diagnose cancers and plan surgery or radiation therapy . However, liver images taken at different phases are usually different in their shape due to disease progress, breathing, patient motion, etc, so image registration is important to improve accuracy of dynamic studies.
Classical image registration methods [11, 4] are usually implemented in a variational framework that solves an energy minimization problem over the space of deformations. Since the diffeomorphic image registration ensures the preservation of topology and one-to-one mapping between the source and target images, the algorithmic extensions to large deformation such as LDDMM  and SyN  have been applied to various image registration studies. However, these approaches usually require substantial time and extensive computation.
To address this issue, recent image registration techniques are often based on deep neural networks that instantaneously generate deformation fields. In supervised learning approaches[12, 14], the ground-truths of deformation fields are required for training neural networks, which are typically generated by the traditional registration method. However, the performance of these existing supervised methods depends on the quality of the ground-truth registration fields, or they do not explicitly enforce the consistency criterion to uniquely describe the correspondences between two images.
In order to overcome the aforementioned limitations and provide topology-preserving guarantee, many unsupervised learning methods have been developed in these days. Balakrishnan et al. 
propose 3D medical image registration algorithm using a spatial transform network. Zhang presents a CNN framework that enforces an inverse-consistent constraint for the deformation fields. However, for the registration of large deformable volumes such as livers, the existing unsupervised learning methods often result in inaccurate registration due to the potential for the degeneracy of the mapping. Although Dalca et el.  tried to address this problem by incorporating a diffeomorphic integration layer, we found that its application of the liver registration is still limited.
In this paper, we present a novel unsupervised registration method using convolutional neural networks (CNN) with cycle-consistency. We show that the cyclic constraint can be adopted for the image registration case naturally, and this cycle consistency improves topology preservation in generating fewer folding problems. Also, our network is trained with diverse source and target images in multiphase CECT acquisition so that a single neural network of our method provides deformable registration between every pairs once the network is trained. Experimental results demonstrate that the proposed method performs accurate 3D image registration for any pair of images within a few seconds in the challenging problem of 3D liver registration in multiphase CECT.
2 Proposed Method
The overall framework of our method is illustrated in Figure 1. For the input images, and , in different phases, we define two registration networks as and , where (resp. ) denotes the 3-D deformation field from to (resp. to ). We use a 3D spatial transformation layer in the networks to warp the moving image by the estimated deformation field, so that the registration networks are trained to minimize the dissimilarity between the deformed moving source image and fixed target image. Accordingly, once a pair of images are given to the registration networks, the moving images are deformed into the fixed images.
To guarantee the topology preservation between the deformed and fixed images, we here adopt the cycle consistency constraint between the original moving image and its re-deformed image. That is, the deformed volumes are given as the inputs to the networks again by switching their order to impose the cycle consistency. This constraint ensures that the shape of deformed images successively returns to the original shape.
2.1 Loss Function
We train the networks by solving the following loss function:
where , , and are registration loss, cycle loss, and identity loss, respectively (see Fig. 2), and and are hyper-parameters. Based on this loss function, our method is trained in an unsupervised manner without ground-truth deformation fields.
Registration Loss. The registration loss function is based on the energy function of classical variational image registration. For example, the energy function for the registration of floating image to the target volume is composed of two terms:
where is the moving image, and is the fixed image. computes image dissimilarity between the deformed image by the estimated deformation field and the fixed image, and evaluates the smoothness of the deformation field. Here, denotes the 3D spatial transformation function. In particular, we employ the cross-correlation as the similarity function to deal with the contrast change during CECT exam, and the L2-loss for regularization function. Accordingly, our registration loss function can be written as:
where denotes the cross-correlation defined by
where and denote the mean value of and , respectively.
Cycle Loss. The cycle consistency condition is implemented by minimizing the loss function as shown in Fig. 2(a). Since an image is first deformed to an image , which is then deformed again by the other network to generate image , the cyclic consistency imposes . Similarly, an image should be successively deformed by the two networks to generate image . Then, the cyclic consistency is to impose that .
As shown in Fig. 2
(a), since the network in our registration receives both the moving image and the fixed image, the implementation of the cycle consistency loss should be given by as the vector-form of the cycle consistency condition:
where Thus, the cycle loss is computed by:
where denotes the -norm.
Identity Loss. Another important consideration for the design of loss function is that the network should not change the stationary regions of the body, i.e. the stationary regions should be the fixed points of the network. As shown in Fig. 2(b), this constraint can be implemented by imposing that the input image should not be changed when the identical images are used as the floating and reference volumes. More specifically, we use the following identity loss:
By minimizing this identity loss (7), the cross-correlation between the deformed image and the fixed image can be maximized. Thus, the identity loss guides the stability of the deformable field estimation in stationary regions.
2.2 Network Architecture and 3D Spatial Transformation Layer
To generate a displacement vector field in width-, height-, and depth direction, we adopt VoxelMorph-1  as our baseline network. Note that our model without both the cycle and identity loss is equivalent to VoxelMorph-1. This 3D network consists of encoder, decoder and their skip connections similar to U-Net .
The 3D spatial transformation layer  is to deform the moving volume with the deformation field . We use the spatial transformation function
with tri-linear interpolation for warping the imageby , which can be written as:
where indicates the voxel index, denotes the 8-voxel cubic neighborhood around , and is three directions in 3D image space.
To verify the performance of our method, we conducted liver registration from multiphase CT images. The dataset was collected from liver cancer (HCC) patients at Asan Medical Center, Seoul, South Korea. Each scans has pathologically proven hepatic nodules and four-phase liver CT (unenhanced, arterial, portal, and 180-s delayed phases). The slice thickness was 5mm. We did not perform pre-processing such as affine transformation except for matching the number of slices for the moving and fixed images. Here, we extracted slices only including liver by a pre-trained liver segmentation network and performed zero-padding to the above and below volumes based on the center of mass of liver.
We used 555 scans for training and 50 scans for testing. For the network training, we stacked two volumes with different phases as the input. We normalized the input intensity with the maximum value of each volume. Also, we randomly down-sampled the training data from to to fit in the GPU memory, while we evaluated the test data with original size of , where
is different for each pair of input. For data augmentation, we adopted random horizontal/vertical flipping and rotation with 90 degree for each pair of training volume. Our proposed method was implemented with pyTorch library. We applied Adam with momentum optimization algorithm to train the models with a learning rate of
, and set the batch size 1. The model was trained for 50 epochs using a NVIDIA GeForce GTX 1080 Ti GPU.
|Method||Arterial Portal||Delayed Portal|
|tumor size||TRE||time||tumor size||TRE||time|
3.1 Registration Results
We evaluated the registration performance using the target registration error (TRE) based on the 20 anatomical points in the liver and adjacent organs at the portal phases, which are marked by radiologists. Also, we measured the tumor size that verifies the registration performance in the view point of tumor diagnosis. We compared our method to Elastix  that is known to have its state-of-the-art performance among the classical approaches. We also compared with VoxelMorph-1 . Additionally, we performed ablation studies by excluding the cycle loss or identity loss. Apart from the loss, different ablated networks were subjected to the same training procedure for fair comparison.
Fig. 3 shows the registration performance. We visualize the TRE values of the deformed arterial and delayed images with respect to each test data, and also show the average TRE values of all subjects with respect to the deformed arterial and delayed images into the fixed portal image. We can observe that the proposed method achieves significant improvement compared to VoxelMorph-1, while the error of the proposed method is slightly higher than Elastix. Also, in Table 1, we can confirm that the tumor size of deformed images from our proposed method is the most accurate for the case of delay to portal registration, and comparable in arterial to portal registration.
To demonstrate the effect of the cycle consistency, we also computed the percentage of voxels with a non-positive Jacobian determinant on the deformation fields and the normalized mean square error (NMSE) between the original moving image and re-deformed image. As shown in Table 2, we confirm that the proposed method is less prone to folding problem and improves topological preservation for liver registration.
Fig. 4 illustrates an example of registration results that deforms the multiphase 3D images with the four distinct phases. Moreover, we calculated the time required for the proposed method to deform one image into the fixed image (see Table 1). Specifically, the conventional Elastix takes approximately 19.6 minutes for the image registration, while the proposed method takes only 10 seconds.
|Method||Arterial Portal||Delayed Portal|
|% of det||NMSE||% of det||NMSE|
We presented an unsupervised image registration method using a cycle consistent convolutional neural network. Using two registration networks, our proposed method is trained to satisfy the cycle consistency that imposes inverse consistency between a pair of images. However, once the networks are trained, a single network can provide accurate 3D image registration with any pair of new data, so the computational complexity is same as VoxelMorph-1. Our liver registration results demonstrated that the proposed method works well for any image pairs with different contrast.
-  Avants, B.B., Epstein, C.L., Grossman, M., Gee, J.C.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis 12(1), 26–41 (2008)
Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: An unsupervised learning model for deformable medical image registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9252–9260 (2018)
-  Beg, M.F., Miller, M.I., Trouvé, A., Younes, L.: Computing large deformation metric mappings via geodesic flows of diffeomorphisms. International journal of computer vision 61(2), 139–157 (2005)
-  Christensen, G.E., Johnson, H.J.: Consistent image registration. IEEE transactions on medical imaging 20(7), 568–582 (2001)
-  Dalca, A.V., Balakrishnan, G., Guttag, J., Sabuncu, M.R.: Unsupervised learning for fast probabilistic diffeomorphic registration. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 729–738. Springer (2018)
-  Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in neural information processing systems. pp. 2017–2025 (2015)
-  Kim, K.W., Lee, J.M., Choi, B.I.: Assessment of the treatment response of hcc. Abdominal imaging 36(3), 300–314 (2011)
-  Klein, S., Staring, M., Murphy, K., Viergever, M.A., Pluim, J.P.: Elastix: a toolbox for intensity-based medical image registration. IEEE transactions on medical imaging 29(1), 196–205 (2010)
-  Mahapatra, D., Antony, B., Sedai, S., Garnavi, R.: Deformable medical image registration using generative adversarial networks. In: Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on. pp. 1449–1453. IEEE (2018)
-  Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
-  Thirion, J.P.: Image matching as a diffusion process: an analogy with maxwell’s demons. Medical image analysis 2(3), 243–260 (1998)
-  Yang, X., Kwitt, R., Styner, M., Niethammer, M.: Quicksilver: Fast predictive image registration–a deep learning approach. NeuroImage 158, 378–396 (2017)
-  Zhang, J.: Inverse-consistent deep networks for unsupervised deformable image registration. arXiv preprint arXiv:1809.03443 (2018)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)