Deform-GAN:An Unsupervised Learning Model for Deformable Registration

1 Introduction

Due to the complex relationship between intensity distributions, multi-modal registration [4] remains a challenging topic. Optimization based registration, which optimizes similarity across phases or modals by aligning the voxel pairs, has been a dominant solution for a long time. However, along with the high complexity of solving 3D images optimization, it is very hard to define a descriptor which is robust enough to cope with the most considerable differences between the image pairs.

Nowadays, a lot of methods leveraging deep learning has been proposed to solve the problems mentioned above. These approaches usually require registration fields of ground truth or landmarks which need to be annotated by experts. Some methods [1][2]

explored unsupervised strategies built on the spatial transformer network.

There are two main challenges in unsupervised-learning based registration. The first one is to define a loss which can efficiently provide similarity measurement across modalities or sequences. For example, mutual information (MI) has been widely and successfully used in registration tasks. But it requires binning or quantizing, which can cause gradient vanishing problem[7]

. The second challenge is no ground-truth. The intuition to solve multi-modal problem is image-to-image translation

[5][3]. But without pixel-wise aligned data pairs, it is difficult to train a GAN to generate synthesized images in which the all texture is mapping to the source exactly. For example, Cycle-GAN can generate the images from MR which just look like CT, but the accuracy in the details cannot meet the requirements of registration.

In this paper, we propose a novel unsupervised method which can easily achieve deformable registration between different sequences or modalities. Local gradient loss, an efficient and robust metric, is the first time to be used in deep-learning-based registration method. We combine adversarial learning approach with spatial transformation to simplify multi-modal similarity to mono-modal similarity. Experiment results show that our approach is competitive to state-of-the-art image registration solutions in terms of accuracy and speed.

2 Method

Figure 1: Overview architecture of the proposed model and training steps. Our model mainly consists of three components: a transformation network $T$ , a generator $G$ and a discriminator $D$ . While training, we take one gradient descent step on $T$ , one step on $G$ and one step on $D$ by turns.

Our model mainly consists of three parts: an image transformation network $T$ which outputs the registration warp field for spatial transformation, an image generator $G$ which does multi-modal translation and a discriminator $D$ which can distinguish real images and synthesized images. The architecture of our model and the training details is illustrated in Fig.1.

2.1 Architecture

The transformation network $T$ takes the reference image $R$ and the floating image $F$ as input, and then outputs the registration warp field $ϕ$ . The mapping can be written as $T : (R, F) \Rightarrow ϕ$ . The floating image $F$ is warped into $F (ϕ)$ using a spatial transformation function. Then $F (ϕ)$ is sent to the generator $G$ which synthesizes images $F^{'} (ϕ)$ of the reference image domain. That, in turn, provides an easier registration task of single-modal between $R$ and $F^{'} (ϕ)$ .

In the proposed network, registration problem is divided into two parts: multi-modal registration $(R, F (ϕ))$ and mono-modal registration $(R, F^{'} (ϕ))$ , which share the same deformation warp field $ϕ$ . So every voxel $F^{'} (ϕ (p))$ in synthesized images should be mapped to $F (ϕ (p))$ precisely. However, in the early learning period, $T$ is poor and the registration result is not accurate. If we use the architecture like Pix2Pix[6] and send the unpaired $F (ϕ)$ and $R$ to the discriminator, the generator will be confused and generate a misaligned $F^{'} (ϕ)$ . To solve this problem, we present a gradient-constrained GAN method for unpaired data. This method is different in that the loss is learned, and can, not only penalize any possible structure that differs between output and target, but also penalize that between output and source. The task of generator consists of three parts: fooling the discriminator, minimizing $L 1$ distance between output and the target, and keeping the output texture similar to the source. The discriminator’s job remains unchanged: only to discriminate real and fake.

Both T and G are U-Net-like[8] network. For details, our code and model parameters are available online at https://github.com/Lauraxy/Multi_Modal_Registration. These three networks should be trained by turns: one step of optimizing $D$ , one step of optimizing $G$ and one step of optimizing $T$ . Note that when training one network, the weights of other two networks should be fixed. Please refer to Fig.1 for details. As $G$ is updated gradually, $F^{'} (ϕ)$ becomes more and more real which helps to update $T$ and $ϕ$ . Then $F (ϕ)$ can be aligned better to $R$ and in turn contributes to train $G$ . This results in that $T$ and $G$ are reaching mutual beneficial.

2.2 Loss

We had tried several loss functions for evaluating similarity between multi-sequence images, such as MI, NGF, and so on. However, each of them has their own weakness and cannot achieve satisfying registration results. For example, we try using Parzen Window estimation of MI to solve gradient-vanish problem, but huge memory consumption make it difficult to train a model in practice. NGF, in our experiment, cannot drive the warp field to convergence. Here we present a local gradient loss which can depict local structure information across modalities. It is similar to NGF, but more robust against noise and easy to converge fast.

Suppose that $p$ is a voxel position of volume $I$ , and we can get the local gradient by:

\nabla^I (p) = (\sum p \in n^{3} x^{'} (p), \sum p \in n^{3} y^{'} (p), \sum p \in n^{3} z^{'} (p))

(1)

Where $x^{'}$ , $y^{'}$ , $z^{'}$ are gradient filed and $p$ iterates over a $n^{3}$ volume around $p$ . Then the gradient can be normalized by:

n (I, p) = \frac{\nabla^I (p)}{∥ \nabla^I (p) ∥ + ε}

(2)

Where $∥ \cdot ∥$ means L2 distance. The local gradient loss between $R$ and $F$ can be defined as follow:

L_{L G} (R, F) = \sum p \in Ω | n (R, p) \cdot n (F, p) |

(3)

$Ω$ is the volume domain of $R$ and $F$ . In the experiment of local gradient, if $n$ in Eq.1 is too small, the network would be difficult to converge. Instead, if $n$ is too large, the edge of $R$ and $F$ cannot be aligned accurately. Finally we set $n = 7$ and get the best results.

Next we will talk about the loss of T, which can be expressed as:

L_{T} (R, F, ϕ) = L_{s i m} (R, F (ϕ)) + α L_{s m o o t h} (ϕ)

(4)

We set $L_{s i m}$ as two parts: the negative local cross-correlation of $R$ and $F^{'} (ϕ)$ , the negative local gradient distance between $R$ and $F (ϕ)$ :

L_{s i m} (R, F (ϕ)) = - L_{L C C} (R, F^{'} (ϕ)) - β L_{L G} (R, F (ϕ))

(5)

Smooth loss, which enforce spatially smooth deformation, can be set as follow [2]:

L_{s m o o t h} (ϕ) = \sum p \in Ω {∥ \nabla ϕ (p) ∥}^{2}

(6)

Then we talk about the generator $G$ and discriminator $D$ . First of all let us review Pix2Pix, a promising approach for many image-to-image translation tasks. The loss of Pix2Pix can be expressed as:

L_{G^{*}} = a r g min G max D L_{c^{G A N}} (G, D) + λ L_{L 1} (G)

(7)

Where $L_{c^{G A N}}$ is the objective of a conditional GAN[6], and $L_{L 1}$ is the L1 distance between the source and the ground truth target. Different from Pix2Pix, in multi-modal registration task, the source and the target are not pixel-wise mapping data. That means directly push the source to near the ground truth in an L1 sense may lead to false translation, which is harmful for registraion. Here we introduce the local gradient loss to constrain gradient distance between the synthesized images $F^{'} (ϕ)$ and the source images $F (ϕ)$ and keep the output texture similar to the source. We mix the GAN objective with local gradient loss to a complete loss:

	$L_{G^{'}} =$	$a r g min G max D L_{c^{G A N}} (G, D) - μ L_{L G} (F^{'} (ϕ), F (ϕ))$		(8)
		$+ λ L_{L 1} (F^{'} (ϕ), R)$		(8)

3 Experiments and Results

3.1 Dataset

We evaluated our method with Brain Tumor Segmentation (BraTS) 2018 dataset[12], which provides a large number of multi-sequence MRI scans, including T1, T2, FLAIR, and T1Gd. Different sequence in the dataset have been aligned very well. We evaluated the registration on T1 and T2 data pairs. We randomly chose 235 data for training and the rest 50 for testing. We cropped and downsized the images to the input size of $112 \times 128 \times 96$ . We added random shift, rotation, scaling and elastic deformation to the scans and generated data pairs for registration, while the synthetic deformation fields can be regarded as registration ground-truth. The range of deformations can be large enough to -40 +40 voxels and it is a challenge of registration.

3.2 Baseline Methods

We compare two well-established image registration methods with ours: A conventional MI-based approach[?] and VoxelMorph method[4]. In the former one, MI is implemented as driving forces within a fluid registration framework. The latter one introduces novel diffeomorphic integration layers combined with a transform layer to enable unsupervised end-to-end learning. But the original VoxelMorph set similarity as local cross-correlation which only function well in single-model registration.As described in chapter 2.2, we also tried several similarity metric with Voxel-morph framework, such as MI, NGF and LG. But only LG is capable of the regis-tration task. So we just use Voxelmorph with LG for comparison.

3.3 Evaluation

We set the loss weight as: $α = 1$ in Eq.4, $β = 2$ in Eq.5 and $μ = 5$ , $λ = 100$ in Eq.8. For CC and local gradient, window size is set as $7 \times 7 \times 7$ . We use ADAM optimizer with learning rate $1 e - 4$ . NVIDIA Geforce 1080ti GPU with 11GB memory is applied for training and testing. To evaluate the effect of gradient-constrained loss(Eq.8) in generator $G$ , we set the network with and without gradient-constrained, named Deform-GAN-2 and Deform-GAN-1, respectively.

Figure 2: Registration results for different methods.

The registration results are illustrated in Fig.2. MI is intrinsically a global measure and so its local estimation is difficult. VoxelMorph with local CC is based on gray value and cannot function well for cross-sequence. The results of VoxelMorph with gradient loss become much better and can handle large deformation between $R$ and $F$ . This demonstrates the effectiveness of the local gradient loss. Our methods, both Deform-GAN-1 and Deform-GAN-2, prove higher accuracy of registration. Even for blur and noisy image (please see the second row), Deform-GAN can get satisfying results, and obviously, Deform-GAN-2 is even better.

For further evaluation on the two setting of Deform-GAN, the warped floating images $F (ϕ)$ and synthesized images $F^{'} (ϕ)$

from two GANs at different stages of training are shown in Fig.3. We can see that gradient constraint brings the faster convergence during training. Even at first epoch, white matter can be seen clearly in

F^{'} (ϕ)

. What’s more, Deform-GAN-2 is more stable in the training process (As the yellow arrows point out, there is less noise in

F^{'} (ϕ)

of Deform-GAN-2 than that of Deform-GAN-1). Note that

F^{'} (ϕ)

is important for calculating

C C (R, F^{'} (ϕ))

, it should be aligned to

F (ϕ)

strictly. The red arrows point out that the alignment between

F (ϕ)

and

F^{'} (ϕ)

of Deform-GAN-2 is much better.

Figure 3: Warped floating images $F (ϕ)$ and synthesized images $F^{'} (ϕ)$ from two Deform-GANs at different stages of training. The yellow arrows point out that Deform-GAN-2 is more stable than Deform-GAN-1. The red arrows indicate the misaligned area between $F (ϕ)$ and $F^{'} (ϕ)$ in Deform-GAN-1. We can also see that Deform-GAN-2 learns more quickly in the early stages of training.

In order to quantify the registration results of our methods and the compared methods, we proposed additional evaluation experiments. For the BraTS dataset, we can warp the floating image by synthetic deformation fields as ground truth. Hence, the root-mean-square error (RMSE) of pixel-wise intensity can be calculated for the evaluation. Also, because the mask of tumor is given by BraTS challenge, we can calculate Dice score to evaluate registration result around tumor area. Table 1 shows the quantitative results. It can be seen that our method outperforms the others in terms of tumor Dice and RMSE. In terms of registration speed, deep-learning based methods are significantly faster than the traditional one. In particular, our method only need to run the transformation network in the inference process, so the runtime is still very fast, though a bit slower than VoxeMorph.

Method	RMSE(%)	Tumer Dice	Runtime(s)
MI	1.39 $\pm$ 0.40	0.55 $\pm$ 0.18	-/6.1
VoxelMorph-LG	1.42 $\pm$ 0.36	0.61 $\pm$ 0.12	0.09/3.9
Deform-GAN-1	1.33 $\pm$ 0.31	0.67 $\pm$ 0.13	0.11/4.4
Deform-GAN-2	1.18 $\pm$ 0.23	0.69 $\pm$ 0.10	0.11/4.4

Table 1: Evaluation of registration on the BraTS dataset in terms of RMSE, average Dice of whole tumor and average runtimes on GPU/CPU.

4 Conclusion

A fast multi-modal deformable registration method that makes use of unsupervised learning is proposed. Adversarial learning method combined with spatial transformation helps to reduce similarity calculation between multi-modal to that between mono-modal. We are able to improve the registration results by a weighted sum of local gradient and local $C C$ in a way that the gradient based loss takes global coarse alignment, while local $C C$ loss ensures registration accuracy. Compared to recent learning based methods, our approach can effectively cope with the multi-modal registration problems with large deformation, non-functional intensity relations, noise and blur, promising in state-of-the-art accuracy and fast runtimes.

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

References

[1] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. Dalca (2018) An unsupervised learning model for deformable medical image registration. In
Proceedings of the IEEE conference on computer vision and pattern recognition
, pp. 9252–9260. Cited by: §1.
[2] A. V. Dalca, G. Balakrishnan, J. Guttag, and M. R. Sabuncu (2018) Unsupervised learning for fast probabilistic diffeomorphic registration. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 729–738. Cited by: §1.
[3] J. Fan, X. Cao, Z. Xue, P. Yap, and D. Shen (2018) Adversarial similarity network for evaluating image alignment in deep learning based registration. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 739–746. Cited by: §1.
[4] M. P. Heinrich, M. Jenkinson, M. Bhushan, T. Matin, F. V. Gleeson, M. Brady, and J. A. Schnabel (2012) MIND: modality independent neighbourhood descriptor for multi-modal deformable registration. Medical image analysis 16 (7), pp. 1423–1435. Cited by: §1.
[5] Y. Hu, E. Gibson, N. Ghavami, E. Bonmati, C. M. Moore, M. Emberton, T. Vercauteren, J. A. Noble, and D. C. Barratt (2018)
Adversarial deformation regularization for training image registration neural networks
. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 774–782. Cited by: §1.
[6] P. Isola, J. Zhu, T. Zhou, and A. A. Efros (2017)
Image-to-image translation with conditional adversarial networks
. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125–1134. Cited by: §2.1, §2.2.
[7] T. Lau, J. Luo, S. Zhao, E. I. Chang, Y. Xu, et al. (2019) Unsupervised 3d end-to-end medical image registration with volume tweening network. arXiv preprint arXiv:1902.05020. Cited by: §1.
[8] O. Ronneberger, P. Fischer, and T. Brox Convolutional networks for biomedical image segmentation. In Paper presented at: International Conference on Medical Image Computing and Computer-Assisted Intervention2015, Cited by: §2.1.

Deform-GAN:An Unsupervised Learning Model for Deformable Registration

Authors

Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations

Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation

Edge Preserving Multi-Modal Registration Based On Gradient Intensity Self-Similarity

Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge

Multi-modal unsupervised brain image registration using edge maps

A Resolution Enhancement Plug-in for Deformable Registration of Medical Images

FlowReg: Fast Deformable Unsupervised Medical Image Registration using Optical Flow

Code Repositories

Multi_Modal_Registration

1 Introduction

2 Method

2.1 Architecture

2.2 Loss

3 Experiments and Results

3.1 Dataset

3.2 Baseline Methods

3.3 Evaluation

4 Conclusion

References

Deform-GAN:An Unsupervised Learning Model for Deformable Registration

Authors

Related Research

Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations

Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation

Edge Preserving Multi-Modal Registration Based On Gradient Intensity Self-Similarity

Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge

Multi-modal unsupervised brain image registration using edge maps

A Resolution Enhancement Plug-in for Deformable Registration of Medical Images

FlowReg: Fast Deformable Unsupervised Medical Image Registration using Optical Flow

Code Repositories

Multi_Modal_Registration

This week in AI

1 Introduction

2 Method

2.1 Architecture

2.2 Loss

3 Experiments and Results

3.1 Dataset

3.2 Baseline Methods

3.3 Evaluation

4 Conclusion

References