Deformable registration is a major challenge in medical image processing. The result is a dense mapping showing pixel-wise non-linear correspondences between a pair of images that best aligns the input image onto the target image by means of some similarity definition
. Deformable registration is applied in the analysis of patient-specific temporal or anatomical changes, e.g. from pre-operative to post-operative state, or to show inter-patient variances[Sotiras et al.(2013)Sotiras, Davatzikos, and Paragios]. Deformable registration is also performed in atlas-based segmentation, where an input image is matched onto a target image with known segmentation [Cabezas et al.(2011)Cabezas, Oliver, Lladó, Freixenet, and Cuadra].
Existing registration methods can be separated into two categories. The first category is based on non-learning methods which estimate a registrationby optimizing a cost function of the form
where denotes warped by . A common assumption of
is a displacement or velocity vector field. The final deformation results in which maps every pixel coordinate to other pixel coordinates. The first term in (1) is referred to as data term, which is typically chosen to be a pixel intensity error measure. Optimization of the data term alone is considered ill-posed. The second term , weighted by trade-off factor , is a regularizer that shapes the registration by any chosen prior, which helps solving the ill-posed problem. Common regularization is done by enforcing smoothness onto the displacement vector field by penalizing first or higher order spatial derivatives of [Werlberger et al.(2010)Werlberger, Pock, and Bischof]. The result of the registration algorithm heavily depends on the cost function and therefore on the chosen prior of .
The second category implicitly learns the regularization prior by training a convolutional network on a large database of domain-specific images. Early approaches rely on ground truth registrations [Sokooti et al.(2017)Sokooti, de Vos, Berendsen, Lelieveldt, Išgum, and Staring], which are hard to obtain especially in medical imaging. More recent methods [Balakrishnan et al.(2019)Balakrishnan, Zhao, Sabuncu, Guttag, and Dalca] propose unsupervised registration using the spatial transformer function [Jaderberg et al.(2015)Jaderberg, Simonyan, Zisserman, and Kavukcuoglu]. However, these methods either only support small displacements or require segmentation maps of the image pairs during training to assist the convergence [Hu et al.(2018)Hu, Modat, Gibson, Li, Ghavami, Bonmati, Wang, Bandula, Moore, Emberton, Ourselin, Noble, Barratt, and Vercauteren]. Additionally, the trained networks are limited to register images from the training domain (e.g. CT or MRI).
Inspired by the idea of deep image priors [Lempitsky et al.(2018)Lempitsky, Vedaldi, and Ulyanov], we subsequently propose our learning-free method for deformable medical image registration using the structure of an untrained convolutional network as regularization prior.
Lempitsky et al. have recently shown that excellent performance of CNNs for inverse image problems, such as denoising, is not only based on their ability to learn image priors from data, but is also based on the structure of a convolutional image generator itself [Lempitsky et al.(2018)Lempitsky, Vedaldi, and Ulyanov]. They gave evidence that the structure of a network alone is sufficient to capture enough image statistics to provide state-of-the-art performance in inverse image tasks.
Leveraged by this idea, we reformulate the task of deformable image registration by using the structure of a convolutional network as regularizer (see Fig. 1). An image generator network with randomly-initialized parameters is interpreted as parameterization of the dense displacement field from which the deformation between an input image and a target image can be obtained by adding to the identity warp . The input has the same spatial dimensions as
and is sampled from a random normal distribution in every iteration. This leads to the following optimization problem
where denotes the differentiable spatial transformer function [Jaderberg et al.(2015)Jaderberg, Simonyan, Zisserman, and Kavukcuoglu]. Eq. (2) is optimized for every image pair using the Adam gradient descent optimizer [Kingma and Ba(2014)]. As data term, we chose pixel-wise mean absolute error . The architecture of the image generator network is chosen according to [Lempitsky et al.(2018)Lempitsky, Vedaldi, and Ulyanov]. It has an encoder-decoder structure with skip connections between the encoding and decoding part. To begin the optimization from close to an identity warp, we initialize the parameters with .
3 Results & Conclusion
We demonstrate our approach on the task of 2D brain magnetic resonance imaging (MRI) registration. The data used in this work contain 109 pairs of MRI scans from The Cancer Genome Atlas [NCI(2019)] showing lower-grade gliomas. We use the structural similarity index (SSIM) [Wang et al.(2004)Wang, Bovik, Sheikh, Simoncelli, et al.] between and and the mean of the determinants of Jacobians [Ashburner(2007)]
of the deformation as evaluation metrics. The latter metric shows regularity of. We compare our method to state-of-the-art methods from the Insight ToolKit (ITK) registration framework by combining an initial affine registration and a subsequent deformable displacement field registration [Avants et al.(2012)Avants, Tustison, Song, Wu, Stauffer, McCormick, Johnson, and Gee]. Results for exemplary image pairs and boxplots of results for all image pairs are shown in Fig. 2. Additional results including registration fields are shown in appendix A.
The results reveal that the structure of a convolutional network can act as regularization in deformable medical image registration with state-of-the-art performance. This connects traditional non-learning methods and learning-based methods by using randomly-initialized convolutional networks as prior.
This research has received funding from the European Union as being part of the EFRE OPhonLas project.
- [NCI(2019)] The Cancer Genome Atlas Program. https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga, 2019. Accessed: 2019-04-01.
- [Ashburner(2007)] John Ashburner. A fast diffeomorphic image registration algorithm. NeuroImage, 38(1):95–113, 2007. doi: 10.1016/j.neuroimage.2007.07.007.
- [Avants et al.(2012)Avants, Tustison, Song, Wu, Stauffer, McCormick, Johnson, and Gee] Brian B Avants, Nicholas J Tustison, Gang Song, Baohua Wu, Michael Stauffer, Matthew M McCormick, Hans J Johnson, and James C Gee. A unified image registration framework for ITK. In International Workshop on Biomedical Image Registration, pages 266–275. Springer, 2012.
- [Balakrishnan et al.(2019)Balakrishnan, Zhao, Sabuncu, Guttag, and Dalca] Guha Balakrishnan, Amy Zhao, Mert R. Sabuncu, John Guttag, and Adrian V. Dalca. VoxelMorph: A Learning Framework for Deformable Medical Image Registration. IEEE Transactions on Medical Imaging, 2019. doi: 10.1109/TMI.2019.2897538.
- [Cabezas et al.(2011)Cabezas, Oliver, Lladó, Freixenet, and Cuadra] Mariano Cabezas, Arnau Oliver, Xavier Lladó, Jordi Freixenet, and Meritxell Bach Cuadra. A review of atlas-based segmentation for magnetic resonance brain images. Computer Methods and Programs in Biomedicine, 104(3):e158–e177, 2011. doi: 10.1016/j.cmpb.2011.07.015.
- [Hu et al.(2018)Hu, Modat, Gibson, Li, Ghavami, Bonmati, Wang, Bandula, Moore, Emberton, Ourselin, Noble, Barratt, and Vercauteren] Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, Nooshin Ghavami, Ester Bonmati, Guotai Wang, Steven Bandula, Caroline M. Moore, Mark Emberton, Sébastien Ourselin, J. Alison Noble, Dean C. Barratt, and Tom Vercauteren. Weakly-supervised convolutional neural networks for multimodal image registration. Medical Image Analysis, 49:1–13, 2018. doi: https://doi.org/10.1016/j.media.2018.07.002.
- [Jaderberg et al.(2015)Jaderberg, Simonyan, Zisserman, and Kavukcuoglu] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial Transformer Networks. In Advances in Neural Information Processing Systems 28, pages 2017–2025, 2015.
- [Kingma and Ba(2014)] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv e-prints, 2014. URL https://arxiv.org/abs/1412.6980.
- [Lempitsky et al.(2018)Lempitsky, Vedaldi, and Ulyanov] Victor Lempitsky, Andrea Vedaldi, and Dmitry Ulyanov. Deep Image Prior. In doi: 10.1109/CVPR.2018.00984.
- [Sokooti et al.(2017)Sokooti, de Vos, Berendsen, Lelieveldt, Išgum, and Staring] Hessam Sokooti, Bob de Vos, Floris Berendsen, Boudewijn P. F. Lelieveldt, Ivana Išgum, and Marius Staring. Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks. In Medical Image Computing and Computer Assisted Intervention, pages 232–239, 2017.
- [Sotiras et al.(2013)Sotiras, Davatzikos, and Paragios] Aristeidis Sotiras, Christos Davatzikos, and Nikos Paragios. Deformable Medical Image Registration: A Survey. IEEE Transactions on Medical Imaging, 32(7):1153–1190, July 2013. doi: 10.1109/TMI.2013.2265603.
- [Wang et al.(2004)Wang, Bovik, Sheikh, Simoncelli, et al.] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
- [Werlberger et al.(2010)Werlberger, Pock, and Bischof] Manuel Werlberger, Thomas Pock, and Horst Bischof. Motion estimation with non-local total variation regularization. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2464–2471, June 2010. doi: 10.1109/CVPR.2010.5539945.
Appendix A Results