Anatomical structures are imaged with a variety of modalities depending on the clinical indication. For instance, Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) show cardiac structures with complementary information important for the assessment of heart diseases [1, 2]
. Despite differences between CT and MRI, often a similar analysis is required, such as quantitative assessment via segmentation. Common practice is to develop a segmentation method, e.g., a convolutional neural network (CNN), separately for CT and MRI data. Such separate training leaves potentially valuable cross-modality information unused. By leveraging multi-modal learning, we can exploit shared cross-modality information, possibly making better use of limited datasets and improving overall performance on each modality.
Previous works on multi-modal image segmentation mostly use multi-parametric MRI (e.g., T1, T2, FLAIR). The inputs to a CNN are paired images, i.e., multi-modal data are acquired from the same patient and co-registered across the sequences. To learn representations for multi-modal segmentation, early fusion and late fusion strategies are typically utilized. Specifically, early fusion means concatenating multi-modal images as different channels at the input layer of a network. This strategy has demonstrated effectiveness on segmenting brain tissue [3, 4, 5] and brain lesions [6, 7, 8]
in multiple sequences of MRI. For late fusion, each modality has modality-specific layers at an early stage of a CNN. The features extracted from different modalities are fused at a certain middle layer of the CNN. The intuition is to initially learn independent features from each modality, and then fuse them at a semantic level. Briefly speaking, late fusion forms a “Y”-shaped architecture, as shown in Fig.1 (a). It has been widely applied for analyzing brain imaging , spinal structures , prostate cancer , and others. Recently, more complex multi-modal CNNs have been designed, by leveraging dense connections 13] or multi-scale feature fusion . These more complicated models still follow the idea of combining modality-specific and shared layers.
We identify two main limitations in the current multi-modal segmentation literature. Firstly, input images are typically paired, which requires multiple images from the same patient as well as a registration step at pre-processing. How to leverage unpaired multi-modal images, e.g, data acquired from different cohorts, still remains unclear. Secondly, multi-modality is often limited to different sequences of MRI. An arguably more challenging situation of multi-modal learning combining CT and MRI is less well explored. Due to distinct physical principles of the underlying image acquisition, the very different visual appearance may require new ways to exchange cross-modality information compared to multi-sequence MRI.
To tackle above limitations, studying unpaired multi-modal learning from non-registered CT and MRI has gained some recent interest [15, 16, 17, 18]. The very different statistical distributions of CT and MRI makes this a challenging problem in terms of learning shared representations. As the unpaired images have little pixel-to-pixel coherence, cross-modality relationship only exists in a semantic space. Valindria et al.  is the closest work to this paper, working on CT/MRI multi-organ segmentation by investigating several dual-stream CNNs, demonstrating a benefit of cross-modality learning of CT and MRI. The state-of-the-art performance is achieved by a “X”-shaped network, as shown in Fig. 1 (b). A recent work  also demonstrate that such an “X”-shaped model is effective for unsupervised multi-modal learning. The modality-specific encoder and decoder layers are designed to tackle the distribution shift between two modalities, while the shared middle layers fuse multi-modal representations.
Our paper proposes a novel compact model for unpaired CT and MRI multi-modal segmentation, by explicitly addressing distribution shift and distilling cross-modality knowledge. We use modality-specific internal feature normalization parameters (e.g., batch normalization layers), while sharing all the convolutional kernels. Importantly, we further propose to distill semantic knowledge from pre-softmax features. A new loss term is derived by minimizing the KL-divergence of a semantic confusion matrix, to explicitly leverage the shared information across modalities. We extensively evaluate our method on two CT and MRI multi-class segmentation tasks, including cardiac segmentation with a 2D dilated CNN and abdominal multi-organ segmentation with a 3D U-Net. Our method consistently outperforms single-model training and state-of-the-art multi-modal learning schemes on both segmentation tasks. The contributions of this work are summarized as follows:
We present a novel, flexible, compact multi-modal learning scheme for accurate segmentation of anatomical structures from unpaired CT and MRI.
We propose a new mechanism to distill semantic knowledge from high-level CNN representations. Based on this, we further derive an effective loss function to guide multi-modal learning.
We conduct extensive validations on two different multi-class segmentation tasks with 2D and 3D CNN architectures, demonstrating general effectiveness of our method.
Code for our proposed approach is publicly available at https://github.com/carrenD/ummkd.
Ii Related Work
Before presenting the proposed approach, we review the literature that inspired the design of our multi-modal learning scheme. The two key aspects are: 1) separating internal feature normalizations for each modality, given the very different statistical distributions of CT and MRI; 2) knowledge distillation from pre-softmax activations, in order to leverage information shared across modalities to guide the multi-modal learning.
Ii-a Independent normalization of CT and MRI
Representation learning between CT and MRI has attracted increasing research interest in recent years. Zhang et al. 
learn image-to-image translation using unpaired CT and MRI cardiac images. Dou et al. present unsupervised domain adaptation of CNNs between CT and MRI for the task of cardiac segmentation using adversarial learning. Huo et al.  learns a CycleGAN based segmentation model from unpaired CT and MRI, only using segmentation labels from one modality. In terms of supervised multi-modal segmentation, image style transfer techniques may not be necessary because we have precise annotations to fully supervise the learning process. To the best of our knowledge, Valindria et al.  is the only paper working on supervised unpaired CT and MRI segmentation so far. They extensively investigate four different types of dual-stream architectures, showing that a “X”-shaped architecture obtains the best performance. This indicates that the distribution shift between CT and MRI heavily affects feature-sharing, requiring modality-specific encoders/decoders. We hypothesis that if the features from different modalities are better normalized, learning cross-modality representations may become easier. In literature, independently normalizing features from different domains has demonstrated efficacy for image classification  and life-long learning on multi-modal MRI brain segmentation .
Ii-B Knowledge distillation
The concept of knowledge distillation (KD) originates from Hinton et al. 
for model compression, i.e., transferring what has been learned by a large model to a smaller-scale model using soft-label supervision. A key aspect that enables KD is to leverage soft labels instead of hard one-hot labels. Temperature scaling is an essential component to allow this by obtaining softer probability distributions across classes, in order to amplify the inter-class relationships. Previous work has adopted knowledge distillation to address various tasks not limited to original model compression, but also a wider scope of scenarios such as domain adaptation , life-long learning , adversarial attacks 
and self-supervised learning. In medical imaging, the potential of the knowledge distillation technique is promising yet relatively under-explored as far as we know. Wang et al. 
employ KD for efficient neuronal structure segmentation from 3D optical microscope images with a teacher-student network. Kats et al. borrow the concept of KD to perform brain lesion segmentation with soft labels by dilating mask boundaries. Christodoulidis et al. 
utilize KD for multi-source transfer learning on the task of lung pattern analysis. With promising results in prior work, we expect an increase of interest in KD. In this paper, we absorb the spirit of knowledge distillation, and explore how to incorporate effective soft probability alignment with temperature scaling within a novel KD loss derived from processing high-level feature-maps for the task of unpaired multi-modal learning.
An overview of our proposed multi-modal segmentation method is shown in Fig. 1 (d). In this section, we first present a compact model design with modality-specific parameters for feature normalization. Next, we show how to distill knowledge from semantic feature-maps and derive the loss term for multi-modal learning. The training procedures are finally described.
Iii-a Separate internal feature normalization
The central architecture of our proposed learning scheme is a separate normalization for internal activations, to mitigate the discrepancy from different data sources. Our design is very different from previous multi-modal learning methods. Rather than using modality-specific encoders/decoders with early/late fusions, we employ the same set of CNN kernels to extract features for both modalities yielding higher parameter efficiency. Using these modality-agnostic kernels gives raise to the hope of extracting universal representations that are more expressive and robust. To achieve this, calibration of the features extracted by the model is important. Normalizing internal activations into Gaussian distribution is common practice for improving convergence speed and generalization of a network. Letdenote the activation in the -th layer, is the
-th group of activations in the layer for which the mean and variance are computed, the normalization layer is:
where and denote the trainable scale and shift. There are different ways to define the activation set of , e.g, Batch Normalization , Instance Normalization , Layer Normalization , and Group Normalization .
We employ separate internal normalization for each modality, as the statistics of CT and MRI data are very different and should not be normalized in the same way which otherwise may yield defective features. For instance, if we have in a certain layer, then the mean over both would be zero which is meaningless. Using separate normalization layer for each modality can effectively avoid such problems during multi-modal learning. Our modality-specific normalization gives raise to a compact multi-modal architecture with minimal extra parametrization forming a “Chilopod” shape, as illustrated in Fig. 1
(c). More specifically, the normalization layers for different modalities (i.e., CT and MRI) are implemented under separate variable scopes, while the convolution layers are constructed under shared variable scope. In every training iteration, the samples from each modality are loaded separately with sub-groups, and forwarded into shared convolution layers and independent normalization layers to obtain the logits which are employed to calculate the knowledge distillation loss.
Iii-B Knowledge distillation loss
As we share all CNN kernels across modalities, the encoders are expected to extract universal representations, capturing common patterns such as shape, which may be more robust and discriminative across modalities. Training, however, becomes more difficult and we find that ordinary objectives, e.g., cross-entropy or Dice loss, are inadequate. We propose to explicitly enhance the alignment of distilled knowledge from semantic features of both modalities.
The assumption of KD is that the probabilities from softmax contain richer information than one-hot outputs. An additional temperature scaling for the pre-softmax activations gives softer probability distributions over classes, which can further amplify such knowledge. In our approach, we distill semantic knowledge from high-resolution feature maps before softmax, as illustrated in Fig. 1 (d). We average the activations over all locations of each class and compute the soft predictions across all classes. We describe this process for 2D below, while it can be easily extended to 3D.
We denote the activation tensor before softmax by, where is the batch size, and denote width and height, is number of channels which also equals to the number of classes as is pre-softmax tensor. Let denote one neuron activation in with index of , and denote the set of locations in the -th sample where the pixel’s label is class . Then, for each class , inside each channel of , we distill the knowledge over all the locations which belong to class . There are channels in total, one for each class. Hence, we get a
-dimensional vectorfor class , with its -th element as averaging over all locations in the -th class channel. Formally, this procedure is represented as:
Next, we compute scaled into a probability distribution using softmax. This distilled knowledge aggregates how the network’s prediction probabilities for the pixels of class distribute across all other classes. Temperature scaling is employed as:
where is the temperature scalar  for softer output to enhance small values. We set in our experiments, with being the ordinary softmax. We empirically observe that the model performance is not very sensitive to this hyper-parameter, within a reasonable range of considered.
Similarly, we can get an array of distilled semantic knowledge . Each vector element conveys how the model’s predictions for pixels of a particular class would distribute across all classes. We can distill such knowledge for each modality, denoted by and for CT and MRI, respectively. We encourage the network’s distilled knowledge from high-level representations of both modalities to be aligned. Intuitively, if one class in CT is often confused by another, this situation may also happen in MRI. For instance, as shown in top-right confusion matrix (the blue and green grid planes) of Fig. 1 (d), the confusions between class 1 and 2 are more obvious, and this is consistent in CT and MRI under our learning scheme. These two classes correspond to the left and right kidney (in human-body view) in abdominal images.
In our scheme, both and are updated together during the dynamic training process. We compute their relative entropy between vectors of each class and design a loss term to minimize their KullbackLeibler (KL) divergence. Our knowledge distillation based loss term (KD-loss) is as follows:
We compute a symmetric version of KL-divergence between two modalities, and is similar to . By explicitly enhancing alignment of the distilled knowledge, the shared kernels can extract features capturing cross-modality patterns co-existing in CT and MRI at multi-modal learning.
Iii-C Overall loss function and training procedure
The compact segmentation network is trained with the loss function as follows:
where and are ordinary segmentation loss for each modality, for which we combine Dice loss  and pixel-wise weighted cross-entropy loss in our experiments. We multiply a scalar for to average over the symmetric KL-divergence. The is generally set as 0.5, and we also study this hyper-parameter in ablation experiments. The last term is a L2 regularizer for shared kernels and modality-specific normalization parameters. The weight is fixed as .
The multi-modal images are sampled to similar voxel-spacing, and normalized to zero mean, unit variance for each modality at the input layer. In each training iteration, we input half of the batch as CT and the other half as MRI. The images go through the shared convolutional kernels and respective internal normalization layers. This can be easily implemented in TensorFlow, by defining the scope of involved variables. The KD-loss is computed with activation tensors of the samples from each modality. All trainable parametersare updated together with the loss function in Eq. 5 using an Adam optimizer. Note that our multi-modal learning scheme is architecture-independent. It can be easily integrated into various existing 2D and 3D CNN models.
We extensively evaluate our multi-modal learning approach on two different multi-class tasks: 1) cardiac structure segmentation; and 2) multi-organ segmentation. We implement 2D and 3D models with different network architectures to demonstrate the flexibility and general efficacy of our method.
Iv-a Datasets and networks
Task-1: We perform multi-class cardiac structure segmentation using the MICCAI 2017 Multi-Modality Whole Heart Segmentation Challenge  dataset, which consists of unpaired 20 CT and 20 MRI images from different patients and sites. The multi-class segmentation includes four structures: left ventricle myocardium (LVM), left atrium blood cavity (LAC), left ventricle blood cavity (LVC) and ascending aorta (AA). We crop the heart regions in the images. Both modalities are resampled to a voxel-spacing at around 1.01.01.0 with size of 256256 in the coronal plane. Before inputting to the network, we conduct intensity normalization to zero mean and unit variance separately for each modality. Each modality is randomly divided into 70% for training, 10% for validation and 20% for testing.
Network-1: For the cardiac segmentation, we implement a 2D CNN with dilated convolutions, following the architecture used in  which employ the same dataset. The inputs are three adjacent slices with the middle slice providing a ground truth mask. The batch size is set as 8 for each modality. The learning rate is initialized as and decayed by in every 1000 iterations. The detailed network architecture is presented in Fig. 2
, with size of convolution kernels indicated within the each box. The number of channels of feature maps are indicated below each box. The normalization layer follows each convolution operation before applying ReLU non-linearity, except for the last knowledge distillation layer before applying softmax.
The layers at which we separate or merge the data streams in the “Y”-shaped and “X”-shaped architectures are indicated using the colored lines in the architectures. For the “Y”-shaped model, layers before green line are separate, and layers after green line are shared between modalities. For the “X”-shaped model, layers before the first blue line and after the second blue line are separate, and layers in-between the blue lines are shared between modalities.
Task-2: We perform multi-organ segmentation in abdominal images for liver, spleen, right kidney (R-kdy) and left kidney (L-kdy). We utilize public CT dataset of  with 30 patients (but one case was excluded due to low image quality), and our MRI data come from the ISBI 2019 CHAOS Challenge, with 9 cases available at the time we downloaded data. This enables us to observe how our method performs in the situation when one modality has much fewer samples than the other. We crop the original CT and MRI images at the areas of multi-organs by excluding the black margins. Since these two datasets have large variance in voxel-spacing. We resample them into around , with size of in transverse plane. The images are normalized to zero mean and unit variance for intensities within each modality before inputting to the network. Again, each modality is randomly divided into 70% for training, 10% for validation and 20% for testing.
Network-2: We employ a 3D U-Net  wise architecture as shown in Fig. 3, where dot-lines indicate skip connections in the network. The volumetric input has a size of , considering to contain all organs inside the transverse view and also the constrain from GPU memory. The batch size is set as 4 at training. Noting that at every training iteration, the balance of the number of samples in a mini-batch for each modality still holds, considering training stability. The learning rate is initialized as and decayed by in every 500 iterations. The indication for “Y”-shaped and “X”-shaped architectures with green and blue lines are the same as described above for Network-1.
|Methods||Param||Cardiac CT||Cardiac MRI||Overall|
|Dice Coefficient (avg.std., )|
|Payer et al. ||-||87.23.9||92.43.6||92.43.3||91.118.4||90.8||75.212.1||81.113.8||87.77.7||76.613.8||80.2||85.5|
|Hausdorff Distance (avg.std., )|
|Payer et al. ||-||-||-||-||-||-||-||-||-||-||-||-|
|Methods||Param||Abdominal CT||Abdominal MRI||Overall|
|Dice Coefficient (meanstd, )|
|Hausdorff Distance (meanstd, )|
Iv-B Experimental settings
We generally use BN layer, as it is the most widely-adopted normalization technique in medical image segmentation tasks.
Ablation study on other normalization layers is also conducted.
For comprehensive analysis and comparison, we design the following seven experimental settings, and implement on our datasets.
Network architecture and hyper-parameters are fixed for all the settings, for a fair comparison of different methods.
Individual: independently training a separate model for each individual modality.
Joint: training a single model with all network parameters (convolution kernels and BN) shared for CT and MRI.
Joint+KD: training a joint model for CT and MRI, with adding our proposed KD-loss.
“Y”-shaped : modality-specific encoders and shared decoders, which is a widely-used late fusion scheme for multi-modal learning.
“Chilopod”-shaped: our proposed architecture, i.e., sharing all CNN kernels while keeping internal feature normalization layers as modality-specific.
Ours: our full multi-modal learning scheme, i.e., using “Chilopod”-shaped architecture and KD-loss together.
Iv-C Segmentation results and comparison with state-of-the-arts
We evaluate the segmentation performance with the metrics of volume Dice coefficient (%) and surface Hausdorff distance (
) by calculating the average and standard deviation of the segmentation results for each class, as listed in TableI and Table II respectively for the two different tasks. The mean Dice coefficient and Hausdorff distance over each modality as well as over two modalities are also presented for a straight-forward comparison. Our implemented Individual models are the baselines from single modality training. We compare the performance of different multi-modal learning methods, including two state-of-the-art approaches [9, 15]. We also refer to the available winning performance of the challenge [38, 36] to demonstrate effectiveness of multi-modal learning.
Iv-C1 Results on multi-modal cardiac segmentation
In Table I, we see that Joint model obtains average segmentation Dice of 88.6% on CT and 84.0% on MRI, which are quite reasonable compared with Individual model (88.4% on CT and 84.8% on MRI). This indicates that networks present sufficient capacity to analyze both CT and MRI in a compact model, though the data distributions of these two modalities are very different. On top of Joint model, adding our proposed KD-loss can improve the segmentation performance to 89.0% on CT and 84.5% on MRI, thank to the explicit guidance from confusion matrix alignment of the distilled semantic knowledge. Next, we compare the different methods which use modality-specific parameters for multi-modal learning. We see that the models of “Y”-shaped, “X”-shaped and “Chilopod” generally get higher performance over “Individual” training. The three models employ the same segmentation loss function (i.e., combining Dice loss and cross-entropy loss), with different ways of designing modality-specific and shared parameters for feature fusion. Both  and  utilize independent encoders/decoders for each modality, while we just use modality-specific BN layers resulting in a more compact model. By further leveraging KD-loss as sort of cross-modality transductive bias, the segmentation performance is boosted to overall Dice of 88.8% (specifically, 91.7% on CT and 86.0% on MRI), exceeding our own implemented “Indivudial” training as well as the MICCAI-MMWHS challenge winner Payer et al.  (overall Dice of 85.5%) which used single model learning. Our approach also achieves the lowest overall mean Hausdorff distance (i.e., ) among all the compared methods. Fig. 4 presents typical segmentation results of CT and MRI images, for a quantitative comparison of the different methods.
Iv-C2 Results on multi-modal abdominal organ segmentation
Table II lists results of multi-organ segmentation using 3D model with skip connections. When analyzing CT and MRI in a Joint model, the performance shows a large decrease compared with Individual training, i.e., overall average Dice dropping from 87.1% to 79.0% while mean Hausdorff distance increases from 2.74 to 3.85. This indicates that the multi-modal shift may present more challenges when learning in 3D. Adding our KD-loss to guide the convergence towards extracting universally representative features, the segmentation Dice is improved by 2.0% (from 79.0% to 81.0%). For multi-modal learning methods, We observe that “X”-shaped model is superior to “Y”-shaped model, which is consistent with the findings in . Our proposed “Chilopod”-shaped model, i.e., sharing all the convolution kernels but with modality-specific BN layers, achieves comparable or better performance than the “X”-shaped model, with a higher parameter efficiency. Further adding our KD-loss, our full multi-modal learning scheme achieves the best performance on average segmentation Dice of 90.8% and average Hausdorff distance of 2.34, outperforming all Individual and “X”-/“Y”-shaped models by a large margin. The Dice of challenge winners for the two datasets used in this multi-organ segmentation task are currently not available. Last but not least, it is worth noting that, Ours significantly improves the segmentation performance on MRI over Individual model by 5.6% (from 83.7% to 89.3%). This demonstrates that our method can effectively improve the performance on the modality with fewer training samples, by leveraging multi-modal learning. We notice that such improvement mainly comes from two organs: spleen and left kidney. These are located nearby and have similar appearance in MRI and pose challenges in the context of data scarcity. Multi-modal learning seems beneficial by enhancing high-level inter-class relationship alignment. The Fig. 5 presents typical segmentation results of CT and MRI images, for a quantitative comparison of these different methods.
Iv-C3 Statistical analysis on significance
We have computed p-values using Student’s t-tests when comparing our segmentation results with other methods, with numbers given in TableIII for 2D cardiac segmentation and Table IV for 3D abdominal organ segmentation. It is observed that we get in all settings, indicating a significant improvement for our approach. The statistical tests are conducted by jointly considering both results of CT and MRI in each setting.
|Metrics||Individual||Joint||Joint+KD||“Y”-shaped ||“X”-shaped ||“Chilopod”|
Iv-C4 Analysis on parameter efficiency
A benefit of our multi-modal learning network is its high compactness. With careful internal normalization of features, we can make more sufficient use of the remarkable capacity inherent in neural networks. We compute the parameter scales of all the models for our implemented seven different settings, as listed in the second column of Table I and Table II. A Joint model has 39.32M parameters for 2D model and 1916K parameters for 3D model. Our 2D model has more parameters because it is much deeper and wider than the 3D model. With individual training, we need double of the parameter scales (i.e., 78.64M for 2D and 3832K for 3D), since each single modality has its own model separately. Using Joint+KD adds no extra parameters, while the distilled knowledge helps stimulate underlying cross-modality information. The conventional “Y”-shaped and “X”-shaped multi-modal learning schemes consist more parameters due to their modality-specific encoders/decoders. Specifically, the “X”-shaped model almost doubles the parameter scale, for the cost of using more modality-specific layers. In comparison, our proposed multi-modal learning schemes (“Chilopod” and Ours) only need marginally extra parameters for separate internal feature normalization with BN (parameterized by a set of feature channel tied scalars . Specifically, our method only adds 0.02M for 2D network and 3K parameters for 3D network, which is quite parameter efficient compared with existing multi-modal schemes.
Iv-D Analytical ablation studies
Iv-D1 Different internal normalization layers
We demonstrate that the effectiveness of separate internal feature normalization is agnostic to different ways of grouping features for . Three additional popular feature normalization methods (i.e., Instance Norm , Layer Norm , and Group Norm ) are implemented for five ablation settings of our method, using the cardiac segmentation dataset. The results are listed in Table V. Ours consistently presents superior performance over Individual learning for all popular feature normalization methods. Comparing the results of “Joint” models under these different normalization methods, we notice that Instance Norm has the best average Dice, achieving 87.5%. This indicates that more refined internal feature normalization benefits multi-modal learning under great parameter sharing. This also meets our hypothesis on normalizing multi-modal images separately in our compact model.
Iv-D2 Different weights of KD-loss
We vary the trade-off weight of our KD-loss, to analyze its sensitivity to the hyper-parameter of in Eq. 5. Specifically, we range at a step of 0.1, and observe our multi-modal segmentation performance on the cardiac dataset. In Fig. 6, the box-plots present the mean of segmentation Dice across all classes on CT (top) and MRI (bottom). Note that the setting of corresponds to the seventh and twelfth columns in Table I. We observe that our method can generally improve the segmentation performance over baselines (e.g., 88.4% for CT and 84.8% for MRI at “Individual” training), while not being very sensitive to the value of .
|Normalization||Methods||Cardiac CT||Cardiac MRI||Overall|
Iv-D3 Learning curve of KD-loss
In Fig. 7, we present the learning curve of KD-loss computed on test data at multi-organ segmentation dataset. We observe that without constraints (i.e., ), the distilled knowledge from two modalities diverges. By activating KD-loss, the is stabilized, reflecting that the probability distributions across classes are better aligned. This observation also explains the performance gain from using the KD-loss as guidance of high-level semantic alignment.
Iv-D4 Evolution of prediction alignment between modalities
In Fig. 8, we visualize the evolution of the confusion matrices (i.e., and ) for both modalities, from the beginning of training until model convergence. To more clearly observe alignment of these matrices (i.e., Eq. 4), we compute their absolute difference plane by abstracting one from the other, as illustrated in the bottom row. It is observed that when the model is randomly initialized, the confusion matrices are invariable with no much difference between CT and MRI. As training goes on, the model starts learning and gradually produces meaningful confusion planes with stronger diagonal responses. Notably, with the model converging, the difference plane between CT and MRI returns to clean again, but for the reason of successful alignment of their confusion matrices.
This work tackles the challenging task of multi-class segmentation on unpaired CT and MRI images. The difficulties mainly arise from the significant distribution shift and absence of registration between the two modalities. This problem has not been well studied so far, compared with segmentation on paired multi-sequence of MRI images. Valindria et al.  is the current state-of-the-art study on this topic, with empirical results demonstrating that leveraging cross-modality information is superior to single model learning. To more explicitly utilize cross-modality knowledge, we present a novel approach by distilling the semantic representations in a high-level within a compact model. Notably, our multi-modal learning scheme is architecture-independent, therefore can be easily integrated into various existing 2D and 3D CNN models. Our approach is developed for unpaired multi-modal learning, and it does not make use of any information which relies on image alignment. In this regard, we think that matching the distributions of the predictions on aligned images would not bring much extra help to our approach as it is. Nevertheless, if such aligned images are available, our method is still directly applicable, and could be extended to leverage the fine-grained pixel-wise alignment.
The remarkable capacity of deep neural networks motivates our design of a highly compact model for multi-modal learning. Bilen et al. 
show that sharing all kernels (even including the final classifier) while using domain-specific batch normalization can work reasonably well for different easy tasks, e.g., MNIST and CIFAR-10. In medical imaging, Moeskops et al. build a single network for different segmentation tasks in different modalities. Karani et al.  successfully adapt an MRI brain segmentation model to different scanners/protocols by only fine-tuning the BN layers. Inspired by these findings, we argue that a single shared network has the potential to work well on very different multi-modal data with similar structures (e.g., CT and MRI) requiring only a few modality-specific parameters. Separate internal normalization may be sufficient to unleash the potential model capacity, not necessarily using a separate encoder for each modality. An explicit regularization loss towards cross-modality semantic alignment helps further stimulate the model capacity.
|Metrics||Individual||Joint||Joint+KD||“Y”-shaped ||“X”-shaped ||“Chilopod”||Ours|
We have conducted case study regarding an outlier case with image artefacts in the abdominal CT dataset. As illustrated in Fig. 9, the outlier case presents clear artefacts with worse image quality compared with other general cases, noting that all the training cases are good-quality images without artefacts. The mean results of the four abdominal organs using seven different settings are listed in Table VI, where we see that all comparison methods obtained a lower performance on this one case (compared with general results in Table II). Our approach achieved a higher Dice score with a smaller Hausdorff distance compared with the other methods, demonstrating our superior robustness at lower image quality.
One limitation of this paper lies in the plain network architectures used for multi-modal segmentation. The employed 2D dilation network for cardiac segmentation and 3D U-Net for abdominal multi-organ segmentation are relatively basic, compared with more complicated network designs for multi-modal learning. This is reasonable as we currently focus on studying separate feature normalization and knowledge distillation between modalities, but this may limit the segmentation accuracy. We plan to integrate our multi-modal learning scheme into more well-designed networks (e.g., consisting multi-scale feature fusion) in our future work, seeking more accurate segmentation of the images with multi-modal learning. This extension is quite natural, thanks to the flexibility of our proposed multi-modal learning scheme.
The characteristics of modality differences may play an important role affecting the performance of multi-modal learning methods. It would be interesting to explore further whether there are limitations on certain functional or statistical relationships between intensity distributions that cannot be handled by our proposed approach. The fact that it works for CT and MRI is encouraging that our approach may work for a large family of relationships. In preliminary synthetic experiments, we could also confirm that an anti-correlation between intensity (i.e., one modality is the inverse of the other) does not pose a problem. Regarding differences in image resolution between modalities, we would expect this not to impact the method too much due to the statistical approach of distribution matching. However, this would need to be confirmed for more extreme cases where one of the modalities is of much lower resolution.
In general, our proposed method of separate internal feature normalization and knowledge distillation loss can be applied to many other situations, when we use data with a mixture of distributions or domains. For instance, it can be used for model learning by aggregating images acquired from different clinical sites in real world scenarios. The data from different sites can be separately normalized within a network, so that one can make better use of multiple data sources. The knowledge distillation loss derived from semantic representations can be applied for domain adaptation problems by leveraging alignment of inter-class relationships. We will explore these extensions in future work.
We present a novel multi-modal learning scheme for unpaired CT and MRI segmentation, with high parameter efficiency and a new KD-loss term. Our method is general for multi-modal segmentation tasks, and we have demonstrated its effectiveness on two different tasks and both 2D and 3D network architectures. Integrating analysis of multi-modal data into a single parameter efficient network helps to ease deployment and improve usability of the model in clinical practice. Moreover, our KD-loss encouraging robust features has a potential to tackle model generalization challenges in medical image segmentation applications.
-  K. Nikolaou, H. Alkadhi, F. Bamberg, S. Leschka, and B. J. Wintersperger, “Mri and ct in the diagnosis of coronary artery disease: indications and applications,” Insights into imaging, vol. 2, no. 1, pp. 9–24, 2011.
R. Karim, L. E. Blake, J. Inoue, Q. Tao, S. Jia, R. J. Housden et al.
, “Algorithms for left atrial wall segmentation and thickness–evaluation on an open-source ct and mri image database,”Medical image analysis, vol. 50, pp. 36–53, 2018.
-  H. Chen, Q. Dou, L. Yu, J. Qin, and P.-A. Heng, “Voxresnet: Deep voxelwise residual networks for brain segmentation from 3d mr images,” NeuroImage, vol. 170, pp. 446–455, 2018.
-  W. Zhang, R. Li, H. Deng, L. Wang, W. Lin et al., “Deep convolutional neural networks for multi-modality isointense infant brain image segmentation,” NeuroImage, vol. 108, pp. 214–224, 2015.
-  P. Moeskops, M. A. Viergever, A. M. Mendrik, L. S. de Vries, M. J. Benders, and I. Išgum, “Automatic segmentation of mr brain images with a convolutional neural network,” IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1252–1261, 2016.
-  L. Fidon, W. Li, L. C. Garcia-Peraza-Herrera, J. Ekanayake, N. Kitchen, S. Ourselin et al., “Scalable multimodal convolutional networks for brain tumour segmentation,” in MICCAI, 2017, pp. 285–293.
-  K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon et al., “Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation,” Medical image analysis, vol. 36, pp. 61–78, 2017.
-  S. Valverde, M. Cabezas, E. Roura, S. González-Villà, D. Pareto, J. C. Vilanova et al., “Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach,” NeuroImage, vol. 155, pp. 159–168, 2017.
-  D. Nie, L. Wang, Y. Gao, and D. Shen, “Fully convolutional network for multi-modality isointense infant brain segmentation,” in ISBI, 2016, pp. 1342–1345.
-  X. Li, Q. Dou, H. Chen, C. W. Fu, X. Qi, D. L. Belavỳ et al., “3d multi-scale fcn with random modality voxel dropout learning for intervertebral disc localization and segmentation from multi-modality mr images,” Medical image analysis, vol. 45, pp. 41–54, 2018.
-  S. Wang, K. Burtt, B. Turkbey, P. Choyke, and R. M. Summers, “Computer aided-diagnosis of prostate cancer on multiparametric mri: a technical review of current research,” BioMed research international, vol. 2014, 2014.
-  J. Dolz, K. Gopinath, J. Yuan, H. Lombaert, C. Desrosiers, and I. B. Ayed, “Hyperdense-net: A hyper-densely connected cnn for multi-modal image segmentation,” IEEE transactions on medical imaging, vol. 38, no. 5, pp. 1116–1126, 2018.
-  J. Dolz, C. Desrosiers, and I. B. Ayed, “Ivd-net: Intervertebral disc localization and segmentation in mri with a multi-modal unet,” in International Workshop and Challenge on Computational Methods and Clinical Applications for Spine Imaging. Springer, 2018, pp. 130–143.
-  J. Li, Z. L. Yu, Z. Gu, H. Liu, and Y. Li, “Mman: Multi-modality aggregation network for brain segmentation from mr images,” Neurocomputing, vol. 358, pp. 10–19, 2019.
-  V. V. Valindria, N. Pawlowski, M. Rajchl, I. Lavdas, A. G. Rockall et al., “Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI,” in WACV, 2018, pp. 547–556.
-  Z. Zhang, L. Yang, and Y. Zheng, “Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network,” in
-  Q. Dou, C. Ouyang, C. Chen, H. Chen, B. Glocker et al., “Pnp-adanet: Plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation,” IEEE Access, vol. 7, pp. 99 065–99 076, 2019.
-  Y. Huo, Z. Xu, H. Moon, S. Bao, A. Assad, T. K. Moyo et al., “Synseg-net: Synthetic segmentation without target modality ground truth,” IEEE transactions on medical imaging, vol. 38, no. 4, pp. 1016–1025, 2018.
-  G. van Tulder and M. de Bruijne, “Learning cross-modality representations from multi-modal images,” IEEE transactions on medical imaging, vol. 38, no. 2, pp. 638–648, 2019.
-  H. Bilen and A. Vedaldi, “Universal representations: The missing link between faces, text, planktons, and cat breeds,” arXiv:1701.07275, 2017.
-  N. Karani, K. Chaitanya, C. Baumgartner, and E. Konukoglu, “A lifelong learning approach to brain MR segmentation across scanners and protocols,” in MICCAI, 2018, pp. 476–484.
-  G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
-  E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko, “Simultaneous deep transfer across domains and tasks,” in ICCV, 2015, pp. 4068–4076.
-  S. Hou, X. Pan, C. Change Loy, Z. Wang, and D. Lin, “Lifelong learning via progressive distillation and retrospection,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 437–452.
-  N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in 2016 IEEE Symposium on Security and Privacy (SP). IEEE, 2016, pp. 582–597.
S. H. Lee, D. H. Kim, and B. C. Song, “Self-supervised knowledge distillation using singular value decomposition,” inEuropean Conference on Computer Vision. Springer, 2018, pp. 339–354.
-  H. Wang, D. Zhang, Y. Song, S. Liu, Y. Wang, D. Feng et al., “Segmenting neuronal structure in 3d optical microscope images via knowledge distillation with teacher-student network,” in ISBI. IEEE, 2019.
-  E. Kats, J. Goldberger, and H. Greenspan, “Soft labeling by distilling anatomical knowledge for improved MS lesion segmentation,” arXiv:1901.09263, ISBI, 2019.
-  S. Christodoulidis, M. Anthimopoulos, L. Ebner, A. Christe, and S. Mougiakakou, “Multisource transfer learning with convolutional neural networks for lung pattern analysis,” IEEE journal of biomedical and health informatics, vol. 21, no. 1, pp. 76–84, 2016.
-  S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015.
-  D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022, 2016.
-  J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016.
-  Y. Wu and K. He, “Group normalization,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 3–19.
-  Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International conference on medical image computing and computer-assisted intervention. Springer, 2016, pp. 424–432.
-  F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in 2016 Fourth International Conference on 3D Vision (3DV). IEEE, 2016, pp. 565–571.
-  X. Zhuang, L. Li, C. Payer, D. Stern, M. Urschler, M. P. Heinrich et al., “Evaluation of algorithms for multi-modality whole heart segmentation: An open-access grand challenge,” preprint arXiv:1902.07880, 2019.
-  B. Landman, Z. Xu, J. Igelsias, M. Styner, T. Langerak, and A. Klein, “2015 miccai multi-atlas labeling beyond the cranial vault – workshop and challenge,” 2015, https://www.synapse.org/#!Synapse:syn3193805/wiki/89480.
-  C. Payer, D. Štern, H. Bischof, and M. Urschler, “Multi-label whole heart segmentation using cnns and anatomical label configurations,” in International Workshop on Statistical Atlases and Computational Models of the Heart. Springer, 2017, pp. 190–198.
P. Moeskops, J. M. Wolterink, B. H. van der Velden, K. G. Gilhuijs, T. Leiner
, “Deep learning for multi-task medical image segmentation in multiple modalities,” inMICCAI, 2016, pp. 478–486.