A Lifelong Learning Approach to Brain MR Segmentation Across Scanners and Protocols

by   Neerav Karani, et al.

Convolutional neural networks (CNNs) have shown promising results on several segmentation tasks in magnetic resonance (MR) images. However, the accuracy of CNNs may degrade severely when segmenting images acquired with different scanners and/or protocols as compared to the training data, thus limiting their practical utility. We address this shortcoming in a lifelong multi-domain learning setting by treating images acquired with different scanners or protocols as samples from different, but related domains. Our solution is a single CNN with shared convolutional filters and domain-specific batch normalization layers, which can be tuned to new domains with only a few (≈ 4) labelled images. Importantly, this is achieved while retaining performance on the older domains whose training data may no longer be available. We evaluate the method for brain structure segmentation in MR images. Results demonstrate that the proposed method largely closes the gap to the benchmark, which is training a dedicated CNN for each scanner.


page 2

page 6


SUSAN: Segment Unannotated image Structure using Adversarial Network

Segmentation of magnetic resonance (MR) images is a fundamental step in ...

Nonlinear Markov Random Fields Learned via Backpropagation

Although convolutional neural networks (CNNs) currently dominate competi...

Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation

Magnetic Resonance Imaging (MRI) is widely used in routine clinical diag...

Automatic segmentation of MR brain images with a convolutional neural network

Automatic segmentation in MR brain images is important for quantitative ...

Deeply-Supervised CNN for Prostate Segmentation

Prostate segmentation from Magnetic Resonance (MR) images plays an impor...

Parkinson's Disease Detection with Ensemble Architectures based on ILSVRC Models

In this work, we explore various neural network architectures using Magn...

1 Introduction

Segmentation of brain MR images is a critical step in many diagnostic and surgical applications. Accordingly, several approaches have been proposed for tackling this problem such as atlas-based segmentation [1]

, methods based on machine learning techniques such as CNNs 

[2], among many others as detailed in this recent survey [3]. One of the important challenges in many MRI analysis tasks, including segmentation, is robustness to differences in statistical characteristics of image intensities. These differences might arise due to using different scanners in which factors like drift in scanner SNR over time [4], gradient non-linearities [5] and others play an important role. Intensity variations may even arise when scanning protocol parameters (flip angle, echo or repetition time, etc.) are slightly changed on the same scanner. Fig. 1(a,b) shows 2D slices from two T1-weighted MRI datasets from different scanners, along with their intensity histograms which show the aforementioned variations. Segmentation algorithms are often very sensitive to such changes. Furthermore, images acquired with different MR modalities, such as T1 and T2-weighted images, may have considerably high-level of similarity in image content (see Fig. 1). While analyzing these images, humans can leverage such commonalities easily and it would be highly desirable if learning-based algorithms could mimic this trait.

Figure 1: Image slices (top) and corresponding histograms (bottom) of normalized T1w (a,b) and T2w (c,d) MRIs from different scanners. Despite high-level information similarity, there exists considerable intensity and contrast differences, which segmentation algorithms are often sensitive to.

In the parlance of transfer machine learning, images acquired from different scanners, protocols or similar MR modalities may be viewed as data points sampled from different domains, with the degree of domain shift potentially indicated by the differences in their intensity statistics. This perspective motivates us to employ ideas from the literature of domain adaptation [6], multi-domain learning [7] and lifelong learning [8]

to the problem of brain segmentation across scanners / protocols. Domain adaptation / transfer learning refers to a situation where a learner trained on a source domain is able to perform well on a target domain, of which only a few labelled examples are available. However, in this case, the performance on the source domain may not be necessarily maintained after adaptation. Multi-domain learning aims to train a learner that can simultaneously perform well on multiple domains. Finally, in lifelong learning, a multi-domain learner is able to incorporate new domains with only few labelled examples, while preserving performance on previous domains.

Variants of image intensity standardization [9, 10] and atlas intensity renormalization [11] have been proposed as pre-processing steps to insure conventional segmentation methods from inter-scanner differences. Among learning methods based on hand-crafted features, transfer learning approaches have been employed for multi-site segmentation [12] and classification [13]

. While adaptive support vector machines used by 

[12] may be adapted for new scanners in a lifelong learning sense, they are likely to be limited by the quality of the hand-crafted features. Using CNNs, [14] propose to deal with inter-protocol differences by learning domain invariant representations. This approach may be limited to work with the least common denominator between the domains, while, as shown in [15], providing a few separate parameters for each domain allows for learning of domain specific nuances. Further, it is unclear how [14] can be extended to deal with new domains that may be encountered after the initial training. In the computer vision literature, several adaptations of batch normalization (BN) [16] have been suggested for domain adaptation [17, 18] and multi-domain learning [15, 19] for object recognition using CNNs. Broadly, these works employ BN for domain-specific scaling to account for domain shifts, while sharing the bulk of the CNN parameters to leverage the similarity between the domains.

In this work, we extend approaches based on adaptive BN layers for segmentation across scanning protocols in a lifelong learning setting. In particular, we train a CNN with common convolutional filters and specific BN parameters for each protocol/scanner. The network is initially trained with images from a few scanners to learn appropriate convolutional filters. By fine-tuning the BN parameters with a few labelled images, it can then be adapted to new protocols/scanners. Crucially, this is achieved without performance degradation on the older scanners, whose training data is not available after the initial training.

2 Method

Batch normalization (BN) was introduced in [16] to enable faster training of deep neural networks by preventing saturated gradients via normalization of inputs before each non-linear activation layer. In a BN layer, each batch is normalized as shown in Eq. 1. During training, and

are the mean and variance of

, while at test time, they are the estimated population mean and variance as approximated by a moving average over training batches.

, are learnable parameters that allow the network to undo the normalization, if required. Inspired by [15], we propose to use separate batch normalization for each protocol / scanner.


Notwithstanding variations in image statistics due to inter-scanner differences, a segmentation network would be confronted with images of the same organ, acquired with the same modality (MR). Hence, it is reasonable to postulate common characteristics between the domains and thus, shared support in an appropriate representation space. Following [15], we hypothesize that such a representation space can be found by using domain-agnostic convolutional filters and that the inter-domain differences can be handled by appropriate normalization via domain-specific BN modules. This approach is not only in line with the previous domain adaptation works [18], but also embodies the normalization idea of conventional proposals for dealing with inter-scanner variations [9, 10, 11]. Further, like [19], once suitable shared convolutional filters have been learned, we adapt the domain-specific BN layers to new related domains.

The training procedure in our framework is as follows. We use superscript to indicate a network with domain-specific BN layers. We initially train a network, on domains, with shared convolutional filters and separate BN parameters, , for each domain . During training, each batch consists of only one domain, with all domains covered successively. In a training iteration when the batch consists of domain , for k’k are frozen. Now, consider a new domain , with a few labelled images . We split this small dataset into two halves, using one for training, and the other for validation, . We evaluate the performance of on , using each learned , . If leads to the best accuracy, we infer that among the already learned domains, is the closest to . Then, keeping the convolutional filter weights fixed, an additional set of BN parameters is initialized with and fine-tuned using

with standard stochastic gradient descent minimization. The optimization is stopped when the performance on

stops improving. Now, the network can segment all domains , for using their respective .

In the spirit of lifelong learning, this approach allows learning on new domains with only a few labelled examples. This is enabled by utilizing the knowledge obtained from learning on the old domains, in the form of the trained domain-agnostic parameters. The fact that the number of domain-specific parameters is small comes with two advantages. One, that they can be tuned for a new domain by training with a few labelled images quickly and with minimal risk of overfitting. Secondly, they can be saved for each domain without significant memory footprint. Finally, catastrophic forgetting [20] by performance degradation on previous domains does not arise in this approach by construction because of the explicit separate modeling of shared and private parameters.

3 Experiments and Results

Datasets: Brain MR datasets from several scanners, hospitals, or acquisition protocols are required to test the applicability of the proposed method for lifelong multi-domain learning. To the best of our knowledge, there are only a few publicly available brain MRI datasets with ground truth segmentation labels from human experts. Therefore, we use FreeSurfer [1] to generate pseudo ground truth annotations. While annotations from human experts would be ideal, we believe that FreeSurfer annotations can serve as a reasonable proxy to test our approach to lifelong multi-scanner learning.

We use images from 4 publicly available datasets: Human Connectome Project (HCP) [21], Alzheimer’s Disease Neuroimaging Initiative (ADNI)111adni.loni.usc.edu, Autism Brain Imaging Data Exchange (ABIDE) [22] and Information eXtraction from Images (IXI)222brain-development.org/ixi-dataset/. The datasets are split into different domains, as shown in Table 1. Domains , , are treated as initially available, and , as new. The number of training and test images for each domain indicated in the table are explained later while describing the experiments.

Training details: While the domain-specific BN layers can be incorporated in any standard CNN, we work with the widely used U-Net [2]

architecture with minor alterations. Namely, our network has a reduced depth with three max-pooling layers and a reduced number of kernels: 32,64,128,256 in the convolutional blocks on the contracting path and 128,64,32 on the upscaling path. Also, bilinear interpolation is preferred to deconvolutional layers for upscaling in view of potential checkerboard artifacts 

[23]. The network is trained to minimize the dice loss, as introduced in [24] to reduce sensitivity to imbalanced classes. Per image volume, the intensities are normalized by dividing with their 98 %tile. The initial network trains in about 6 hours, while the domain-specific BN modules can be updated for a new domain in about 1 hour.

Domain Dataset Field MR Modality
D HCP 3T T1w 30 30 20
D HCP 3T T2w 30 30 20
D ADNI 1.5T T1w 30 30 20
D ABIDE, Caltech 3T T1w 4 20 20
D IXI 3T T2w 4 20 20
Table 1: Details of the datasets used for our experiments.

Experiments: We train three types of networks, as described below.

  • Individual networks : Trained for each domain , with training images (see Table 1). For the known domains (, , ), the accuracy of serves as a baseline that the other networks with shared parameters must preserve. For the new domains (, ), the performance of is the benchmark that we seek to achieve by training on much fewer training examples () and using the knowledge of the previously learned domains.

  • A shared network : Trained on , , with images, with all parameters shared including the BN layers, bn. In contrast to the training regime of described in Section 2, while training each batch randomly contains images from all domains to ensure that the shared BN parameters can be tuned for all domains. Histogram equalization [25] is applied to a new domain before being tested . For adapting to , its parameters are fine-tuned with images of the new domain and the modified network is referred to as .

  • A lifelong multi-domain learning network : Trained on , , , with shared convolutional layers and domain-specific BN layers. The updated network after extending for a new domain according to the procedure described in Sec. 2 is called , where is the closest domain to .

Results: All networks are evaluated based on their mean Dice score for images from the appropriate domain (see Table 1). Quantitative results of our experiments are shown in Table 2. The findings can be summarized as follows:

  • preserves the performance of , , . Thus, a single network can learn to segment multiple domains, provided sufficient training data is available from all the domains at once. However, its performance severely degrades for unseen domains and . Histogram equalization (denoted by ) to the closest domain is unable to improve performance significantly, while fine-tuning the network for the new domains causes catastrophic forgetting [20], that is, degradation in performance on the old domains.

  • also preserves the performance of , , . For a new domain , using the parameters of the trained lead to the best performance. Thus, we infer that is the closest to among , , . After fine-tuning the parameters of BN to obtain those of BN, the dice scores for all the structures improve dramatically and are comparable to the performance of . Crucially, as the original for k=1,2,3 are saved, the performance on , , in the updated network is exactly the same as in . Similar results can be seen for the other new domain, . The improvement in the segmentations for new domains after fine-tuning the BN parameters can also be observed qualitatively in Fig. 2.


Network Test BN Thal Hipp Amyg Ventr Caud Puta Pall Avg


N D bn 0.919 0.861 0.849 0.901 0.9 0.887 0.747 0.866
N D bn 0.912 0.84 0.836 0.891 0.889 0.876 0.736 0.854
N D bn 0.913 0.872 0.81 0.944 0.864 0.879 0.853 0.876
N D bn 0.924 0.879 0.853 0.933 0.912 0.9 0.851 0.893
N D bn 0.884 0.79 0.773 0.803 0.793 0.818 0.791 0.81


N D bn 0.909 0.846 0.824 0.891 0.878 0.877 0.745 0.853
N D bn 0.888 0.838 0.815 0.876 0.863 0.86 0.701 0.834
N D bn 0.905 0.851 0.792 0.938 0.863 0.873 0.828 0.864
N D bn 0.745 0.249 0.057 0.787 0.428 0.324 0.071 0.38
N D bn 0.641 0.428 0.175 0.754 0.628 0.579 0.303 0.501
N D bn 0.91 0.856 0.74 0.922 0.894 0.859 0.786 0.852
N D bn 0.869 0.809 0.773 0.867 0.861 0.722 0.667 0.795
N D bn 0.676 0.418 0.512 0.105 0.635 0.4 0.411 0.451
N D bn 0.801 0.762 0.65 0.753 0.728 0.715 0.772 0.74
N D bn 0.418 0.178 0.182 0.438 0.268 0.197 0.025 0.244
N D bn 0.294 0.143 0.16 0.437 0.261 0.293 0.01 0.228
N D bn 0.861 0.777 0.761 0.799 0.76 0.796 0.741 0.785
N D bn 0.267 0.022 0.173 0.004 0.05 0.002 0.004 0.075
N D bn 0.574 0.574 0.564 0.739 0.657 0.521 0.526 0.594
N D bn 0.147 0.029 0.16 0.006 0.114 0.039 0.003 0.071


N D bn 0.916 0.852 0.84 0.894 0.893 0.884 0.729 0.858
N D bn 0.91 0.853 0.843 0.887 0.882 0.873 0.749 0.857
N D bn 0.911 0.868 0.818 0.944 0.867 0.879 0.846 0.876
N D bn 0.621 0.288 0.218 0.173 0.676 0.576 0.457 0.43
N D bn 0.162 0 0.001 0.001 0.04 0.017 0 0.032
N D bn 0.721 0.271 0.305 0.549 0.569 0.515 0.297 0.461
N D bn 0.878 0.83 0.772 0.907 0.875 0.852 0.772 0.841
N D bn 0.001 0.019 0.062 0.008 0.004 0 0 0.013
N D bn 0.354 0.123 0.268 0.225 0.407 0.276 0.366 0.288
N D bn 0 0.003 0.031 0.001 0 0 0 0.005
N D bn 0.774 0.687 0.687 0.761 0.669 0.714 0.713 0.715


Table 2: Segmentation Dice scores for different domains for the three different types of networks, trained as explained in the experiments section.
Figure 2: Qualitative results: (a) images from domains D, segmentations predicted by (b) N, bn, (c) N, bn, (d) N and (e) ground truth annotations, with {, } as {4, 3} (top) and {5, 2} (bottom).

4 Conclusion

In this article, we presented a lifelong multi-domain learning approach to learn a segmentation CNN that can be for related MR modalities and across scanners/protocols. Further, it can be adapted to new scanners or protocols with only a few labelled images and without degrading performance on the previous scanners. This was achieved by learning batch normalization parameters for each scanner, while sharing the convolutional filters between all scanners. In future work, we intend to investigate the possibility of extending this approach to MR modalities that were not present during the initial training.

To the best of our knowledge, this is the first work to tackle the lifelong machine learning problem for CNNs in the context of medical image analysis. We believe that this may set an important precedent for more research in this vein to handle data distribution changes which are ubiquitous in clinical data.


  • [1] B Fischl, “Freesurfer,” Neuroimage, vol. 62, no. 2, pp. 774–781, 2012.
  • [2] O Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  • [3] I Despotović et al., “Mri segmentation of the human brain: challenges, methods, and applications,” Computational and mathematical methods in medicine, vol. 2015, 2015.
  • [4] G Preboske et al., “Common mri acquisition non-idealities significantly impact the output of the boundary shift integral method of measuring brain atrophy on serial mri,” Neuroimage, vol. 30, no. 4, pp. 1196–1202, 2006.
  • [5] J Jovicich et al., “Reliability in multi-site structural mri studies: effects of gradient non-linearity correction on phantom and human data,” Neuroimage, vol. 30, no. 2, pp. 436–443, 2006.
  • [6] S Pan et al., “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2010.
  • [7] M Dredze et al., “Multi-domain learning by confidence-weighted parameter combination,” Machine Learning, vol. 79, no. 1-2, pp. 123–149, 2010.
  • [8] S Thrun, “Lifelong learning algorithms,” in Learning to learn, pp. 181–209. Springer, 1998.
  • [9] Y Zhuge et al., “Intensity standardization simplifies brain mr image segmentation,” Computer vision and image understanding, vol. 113, no. 10, pp. 1095–1103, 2009.
  • [10] N Weisenfeld et al.,

    “Normalization of joint image-intensity statistics in mri using the kullback-leibler divergence,”

    in Biomedical Imaging: Nano to Macro, 2004. IEEE International Symposium on. IEEE, 2004, pp. 101–104.
  • [11] X Han et al., “Atlas renormalization for improved brain mr image segmentation across scanner platforms,” IEEE transactions on medical imaging, vol. 26, no. 4, pp. 479–486, 2007.
  • [12] A Van Opbroek, M A Ikram, M Vernooij, and M De Bruijne, “Transfer learning improves supervised image segmentation across imaging protocols,” IEEE transactions on medical imaging, vol. 34, no. 5, pp. 1018–1030, 2015.
  • [13] V Cheplygina, I Pena, J Pedersen, D Lynch, L Sorensen, and M de Bruijne, “Transfer learning for multi-center classification of chronic obstructive pulmonary disease,” IEEE Journal of Biomedical and Health Informatics, 2017.
  • [14] L Kamnitsas et al., “Unsupervised domain adaptation in brain lesion segmentation with adversarial networks,” in International Conference on Information Processing in Medical Imaging. Springer, 2017, pp. 597–609.
  • [15] H Bilen and A Vedaldi, “Universal representations: The missing link between faces, text, planktons, and cat breeds,” arXiv preprint arXiv:1701.07275, 2017.
  • [16] S Ioffe et al., “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning, 2015, pp. 448–456.
  • [17] Y Li, N Wang, J Shi, J Liu, and X Hou, “Revisiting batch normalization for practical domain adaptation,” arXiv preprint arXiv:1603.04779, 2016.
  • [18] F Carlucci and L Porzi, “Autodial: Automatic domain alignment layers,” .
  • [19] S Rebuffi et al., “Learning multiple visual domains with residual adapters,” in Advances in Neural Information Processing Systems, 2017, pp. 506–516.
  • [20] R French, “Catastrophic forgetting in connectionist networks,” Trends in cognitive sciences, vol. 3, no. 4, pp. 128–135, 1999.
  • [21] D Van Essen et al., “The wu-minn human connectome project: an overview,” Neuroimage, vol. 80, pp. 62–79, 2013.
  • [22] A Di Martino et al., “The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism,” Molecular psychiatry, vol. 19, no. 6, pp. 659, 2014.
  • [23] A Odena et al., “Deconvolution and checkerboard artifacts,” Distill, 2016.
  • [24] F Milletari et al., “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in 3D Vision (3DV), 2016 Fourth International Conference on. IEEE, 2016, pp. 565–571.
  • [25] L Nyúl et al., “New variants of a method of mri scale standardization,” IEEE transactions on medical imaging, vol. 19, no. 2, pp. 143–150, 2000.