Over the years, public medical imaging datasets have emerged which enable researchers to benchmark the performance of their algorithms . Data is mostly acquired from patients who have volunteered to be part of a clinical research study and are subject to a strict study protocol. If the study involves the acquisition of Magnetic Resonance Imaging (MRI) scans, the study protocol might dictate the scanner choice as well as the acquisition parameters to be used . In the real unconstrained clinical setting however, MRIs are more likely to be acquired from different machines under different acquisition protocols and parameters. There is no guarantee that a particular sequence will be available, no guarantee on the number of available modalities, no guarantee that modalities will be unique (e.g. same sequence acquired with different orientations and contrasts), and no guarantee that any of the modalities will be labeled appropriately for algorithmic use. If hospitals are to benefit from advances in neuroimaging, algorithms that can cope with this lack of available modalities are necessary. We argue that an algorithm which is to be deployed in this setting should have two key properties: 1)permutation invariance, i.e permuting the order of the input images should not affect the output and 2)robustness to missing modalities. To this end we propose a segmentation model, with neural networks as building blocks, which can learn with limited data and segment scans without MR modality labels. In this work we focus on the task of segmenting white matter hyperintensities (WMH). In studies involving WMH segmentation the most common modalities are T1, T2 and T2-FLAIR which provide complementary information about the imaged tissue. Although T1 and T2 modalities are created from different underlying physical signals (longitudinal and transverse relaxation time respectively) the scans produced will almost always be a combination of both (hence the name attribute -weighted). By varying the acquisition parameters, such as the echo and relaxation times, these underlying physical signals are observed in different proportions . Modality labels are a discrete approximation of a continuous acquisition parameter landscape and we use this as inspiration for the model we present.
In order to address missing modalities, research has focused mostly on generative models where missing MRI scans are synthesized or imputed . In the work of  the authors handle missing modalities without using generative models of MR modalities. Instead of synthesizing the missing modalities, their model, Hetero-modal Image Segmentation (HeMIS), is trained to handle missing input modalities. More details about HeMIS can be found in section 2.0.1. Although HeMIS is successful at dealing with missing modalities, it assumes that the MR modalities in a test case will be labeled. The authors of 
tackle the issue of generalizing to unseen protocols and scanners. In order to be robust to different scanners and protocols, they propose a tuning of the batch normalization parameters of a CNN. However, their method still requires approximately four scans with their associated segmentations from the unseen protocol to perform well.
We introduce a model that learns to build intermediate representations of the images as a linear combination of the available inputs which are more continuous than their original labels. The proposed model does not assume the modality is known and has the ability to generalize to unseen scanners/protocols, taking in unordered input scans with no modality labels to produce accurate segmentation masks. We provide results on a variety of datasets featuring WMH with large variability in scanner type and acquisition parameters and show that our model is both permutation invariant and robust to missing modalities. We demonstrate that it can perform comparatively well with an algorithm which utilizes the modality labels having never seen an image from that particular protocol. Furthermore, our model can outperform the baseline method (HeMIS) in the case where it has seen MR modality labels of the same protocol it is being tested on.
In HeMIS each available modality, , is embedded with a modality specific function
denoted the “back-end” to produce embeddings. An “abstraction” layer then operates on these embeddings by computing the mean and variance across their
dimensions and concatenating the two resulting vectors, where x M is the number of modalities and D is the spatial dimensions of the input. Let
be a fixed dimensional tensor which represents an input of variable size. This forms the input to the final portion of the network referred to as the “frontend” which will output a semantic segmentation map. The network is trained using a Dice loss, first proposed in
as a loss function for training neural networks.
During training, random modalities are set to zero, encouraging robustness to missing modalities. HeMIS, shown in Figure 1, forms part of our architecture.
2.0.2 Our approach
We propose a method which at test time takes in an arbitrary number of scans (denoted ) which do not have corresponding MR modality labels and produces a permutation invariant representation that is also robust to missing modalities. In theory this common representation could be applied to a variety of tasks. In this paper we focus on white matter hyperintensity segmentation.
The inputs are fed into an MR modality classifierwhich outputs a distribution over modalities for a given scan as its prediction. These modality scores are combined with the inputs, , to produce modified inputs denoted as . In the attention literature a distinction is drawn between “soft” and “hard” attention . Soft attention generally involves a probabilistic weighted sum whilst a hard attention is a categorical argmax over the inputs. With this in mind, we explore two methods for performing : and . The function is defined as,
Each component of the modified input is formed by taking a weighted sum of each input
according to the probabilities provided by. is defined as,
The modified input now consists of a finite number of modalities. The mapping is illustrated in the blue block in Figure 1.
Each MR modality is designed to capture fundamentally different physical properties which justifies having individual feature extractors, , for each modality representation. The output of these modality-specific feature extractors is collected into one tensor by taking the mean and the variance across modalities and concatenating the result to give where is given by the choice of filter depth in . This feeds into a final network, which produces a segmentation prediction. This use of modality specific models, pooling and a separate segmentation network is the same as HeMIS and is illustrated in the grey block in Figure 1.
A convolutional neural network was used for
. A network with 36 layers using skip connections and ReLU non-linearities inspired by the residual network (ResNet) proposed in is used. The network was trained with the categorical cross-entropy loss which we refer to as . where is a one-hot encoded modality label and is the modality score. Each of the branches as well as were two convolutional layers with ReLU non-linearities (more details in section 2.0.3). The parameters of and were found by minimizing which is the binary Dice Loss.
For two of our variants these losses were trained separately (or “offline”). However, we also trained an “online” variant where the parameters of the modality classifier are learned using a multi-objective loss function. This loss is defined as, , where is some choice of weighting or parametrized weighting function. Although the loss consists of multiple objectives this should not be considered “multi-task learning”. There is no conditional independence between the tasks and no representation sharing — instead this can be seen as a differentiable attention mechanism. The four variants trained are summarized below,
- using labels, trained separately from , , &
- used to create , trained separately from , , & ,
- used to create , trained separately from , , & ,
- used to create , trained jointly with , , & .,
2.0.3 Implementation Details
It is important to note that the network architecture takes in 2D patches from the image as was done in . Specifically we take patches of size from 3D scans which have all been resampled to . This theoretical framework permits any spatial dimension and future work will train and run inference in full 3D.
All results were obtained using the NiftyNet framework 
, which is a wrapper around TensorFlow designed for medical imaging.uses a standard ResNet design with nine blocks per resolution, each with three convolutions and relu activations. The network is trained using the Adam optimizer with a learning rate of . A batch size of was used on this network and weight decay regularization of .
For each and the implementation details from  were recreated. Two convolutional layers with 48 filters,two convolutional layers were used, one with 16 filters, kernel sizes, zero padding and ReLU activation the last convolutional layer had 2 filters, a kernel size of
, zero padding and a softmax activation which provided the per class predictions. We also utilized the pseudo-curriculum learning approach from HeMIS. Random modalities are set to zero but the chance of setting only one or no modalities to zero is higher. The online model was harder to train than the offline ones. The joint training lead to odd dynamics between the classification loss and the segmentation loss. To help stabilize the training an exponential decay weighting was used on the classification loss in order to encourage training it towards the start and remove its importance later on so that the model could experiment with representations which do not match the provided labels and not be punished by. Our best performing “online” model used where is the current iteration and
is a decay constant hyperparameter set to.
This same ResNet architecture was used as in the online case in order to make a fair comparison in terms of number of parameters. However, in the online setting, the batch size had to be reduced as a practical consideration as the combination of both modality and backend models proved too large to fit in GPU memory. All experiments were run on a single NVIDIA Titan Xp.
3 Experiments and Results
Data used in this work comes from a variety of sources, chosen to try and capture the acquisition variability observed in a practical setting due to multiple MRI scanners/protocols. A subset of 973 subjects each with T1 and FLAIR scans were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database . The data in this study was collected from multiple scanners, but used the same protocol for setting the acquisition parameters. We therefore deem this dataset one of relatively low variance between subjects. We also utilise data collected from the longitudinal SABRE study . The data contains one cohort of 586 subjects with T1, T2 and FLAIR obtained using the same scanner (low variance) and another of 1263 with T1, T2 and FLAIR obtained from multiple scanners with multiple settings (high variance). Additionally we use a dataset of 626 patients with T1 and FLAIR obtained from multiple scanners using multiple field strengths. As no manual annotations were available for this large collection of MRI scans, the outputs of BaMoS , a fully unsupervised WM lesion segmentation algorithm, were quality controlled by an experienced human rater and subsequently used as silver-standard training labels. Additionally, we evaluate our trained models on a manually annotated dataset from the MICCAI 2017 White Matter Hyperintensity Challenge .
The split between training, validation and test sets was chosen in order to measure the ability of our method at generalizing to unseen scanners and protocols. Three separate holdouts were created, defined as follows,
- Mixed Holdout
- Random 10% subset of the full data minus ADNI/MICCAI17,
- Silver Protocol Holdout
- ADNI: 973 subjects with silver standard labels.
- Gold Protocol Holdout
- MICCAI2017: 60 subjects with human rater labels.
Overall there was a 80/10/10 split between training, validation and test using the 2474 subjects that are not in the gold or silver protocol holdouts. All four models described in section 1 were trained with this subset.
For the mixed holdout it was found that the classification accuracy was between all three modalities. For unseen protocols the accuracy was lower, for ADNI and for MICCAI17 which showed that the inter-scanner variance was harder to model than the inter-subject variance. For each of the holdout sets, results are presented on all possible subsets of the available modalities. The quantitative and qualitative results are shown in Table 1 and Figure 2, respectively. The brains shown are selected from the 95%, 50% and 20% percentile of Dice score on the dataset holdout for a model shown all available modalities. We note that the samples of very high Dice score are often the ones with large lesions which the algorithm has managed to capture well and there is poor performance when the contrast settings are significantly different.
We utilise the Wilcoxon signed-rank test to test whether the Dice scores from each of our models outperforms the baseline (HeMIS). Bold values in Tables 1 denotes that the model is better than HeMIS with a statistical significance of . We compare ground truths and predictions using the Dice score as well as the average symmetric distance in order to provide a geometric evaluation.
The “hard” setting converges to HeMIS as the accuracy of the modality classifier tends to 1. This is observed in practice. Note that the results of HeMIS are similar to “hard” in the mixed holdout set where the modality classifier has had access to the test set distribution and consistently worse in the Silver Protocol holdout. It does comparatively better on the Gold Protocol as the modality classifier has better performance on these scans than on Silver. The “soft” version matches or improves on the performance of HeMIS and “hard” on the mixed holdout, but does not outperform HeMIS on other holdouts. The fact that “soft” outperfoms “hard” is evidence towards our hypothesis that mixing the input images can lead to better representations which improve performance on a visual task. This can be interpreted as a coarse attention mechanism as the transformation from to
is linear with few degrees of freedom.
The “online” model outperforms the baseline in the mixed holdout set with statistical significance in 6/7 cases when using the Dice score. Although the median average symmetric distance (ASD) is higher, the average is lower in 4/7 cases with a much lower 95 percentile. There is some improvement over the baseline model even in the protocol holdout but the gains seen in Dice score are not reflected in the ASD. Qualitatively this is explained by the “online” method overpredicting the positive class leading to a higher Dice score and yet missing lesions altogether leading to a larger ASD. This gives us insights as to how we can improve the model.
Future work will extend the “online” model to an unsupervised setting in terms of scan labels. This is appealing not only due to the lack of modality labels currently available in certain hospital databases but also in order to go beyond the information contained in the modality label and towards a representation which is more true to the underlying physical structure.
We have presented PIMMS, a segmentation algorithm for MRI scans which simultaneously addresses the problem of missing modalities and missing modality labels in a clinical setting. We present three variants which all include a convolutional neural network and are trained to perform modality classification in a supervised setting. We argue that by mixing the input modalities in ratios other than those provided by the labels we can achieve better performance. This could be due to more accurately capturing the underlying distribution of physical quantities, but future work is needed to make this claim. Evidence is presented with statistical significance which suggests that a model which mixes inputs can perform better than one which does not with all other factors kept identical.
The results show that the modality classifier almost replicates modality labels when trained and tested on the same protocol while the categorical accuracy reaches 88% when protocols differ at training and testing times.
Our model serves as a proof of concept for a system that could utilize all the MR scans associated with a patient in a hospital and provide accurate segmentation predictions.
Acknowledgements We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. Zach Eaton-Rosen is supported by the EPSRC Doctoral Prize. Carole H. Sudre is supported by the Biomedical Junior Fellowship from Alzheimer’s Society. Parashkev Nachev is funded by the Wellcome Trust and the UCLH NIHR Biomedical Research Centre. M. Jorge Cardoso is funded by Wellcome Trust.
-  White Matter Segmentation Grand Challenge at MICCAI 2017
-  Chartsias, A., et al.: Multimodal mr synthesis via modality-invariant latent representation. IEEE transactions on medical imaging (2017)
-  Fischl, B., et al.: Sequence-independent segmentation of magnetic resonance images. Neuroimage 23, S69–S84 (2004)
-  Ghafoorian, M., et al.: Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. CoRR abs/1610.04834 (2016)
-  Gibson, E., et al.: Niftynet: a deep-learning platform for medical imaging. arXiv preprint arXiv:1709.03485 (2017)
-  Havaei, M., et al.: Hemis: Hetero-modal image segmentation. In: MICCAI. pp. 469–477. Springer (2016)
He, K., et al.: Identity mappings in deep residual networks. In: European Conference on Computer Vision. pp. 630–645. Springer (2016)
-  Iglesias, J.E., et al.: Is synthesizing mri contrast useful for inter-modality analysis? In: MICCAI. pp. 631–638. Springer (2013)
-  Jack, C.R., et al.: The alzheimer’s disease neuroimaging initiative (adni): Mri methods. Journal of magnetic resonance imaging 27(4), 685–691 (2008)
-  Karani, N., et al.: A lifelong learning approach to brain mr segmentation across scanners and protocols. arXiv:1805.10170 (2018)
-  Milletari, F., et al.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3DV. pp. 565–571. IEEE (2016)
-  Sudre, C.H., et al.: Bayesian model selection for pathological neuroimaging data applied to white matter lesion segmentation. IEEE TMI 34(10), 2079–2102 (2015)
-  Tillin, T., et al.: Southall and brent revisited: Cohort profile of sabre, a uk population-based comparison of cardiovascular disease and diabetes in people of european, indian asian and african caribbean origins. IJEpid 41(1), 33–42 (2010)
Xu, K., et al.: Show, attend and tell: Neural image caption generation with visual attention. In: International Conference on Machine Learning. pp. 2048–2057 (2015)