A post-processing method to improve the white matter hyperintensity segmentation accuracy for randomly-initialized U-net

by   Yue Zhang, et al.

White matter hyperintensity (WMH) is commonly found in elder individuals and appears to be associated with brain diseases. U-net is a convolutional network that has been widely used for biomedical image segmentation. Recently, U-net has been successfully applied to WMH segmentation. Random initialization is usally used to initialize the model weights in the U-net. However, the model may coverage to different local optima with different randomly initialized weights. We find a combination of thresholding and averaging the outputs of U-nets with different random initializations can largely improve the WMH segmentation accuracy. Based on this observation, we propose a post-processing technique concerning the way how averaging and thresholding are conducted. Specifically, we first transfer the score maps from three U-nets to binary masks via thresholding and then average those binary masks to obtain the final WMH segmentation. Both quantitative analysis (via the Dice similarity coefficient) and qualitative analysis (via visual examinations) reveal the superior performance of the proposed method. This post-processing technique is independent of the model used. As such, it can also be applied to situations where other deep learning models are employed, especially when random initialization is adopted and pre-training is unavailable.


page 1

page 2

page 3

page 4


Anatomical Priors for Image Segmentation via Post-Processing with Denoising Autoencoders

Deep convolutional neural networks (CNN) proved to be highly accurate to...

Transfer Learning U-Net Deep Learning for Lung Ultrasound Segmentation

Transfer learning (TL) for medical image segmentation helps deep learnin...

Prostate Segmentation from 3D MRI Using a Two-Stage Model and Variable-Input Based Uncertainty Measure

This paper proposes a two-stage segmentation model, variable-input based...

Post-DAE: Anatomically Plausible Segmentation via Post-Processing with Denoising Autoencoders

We introduce Post-DAE, a post-processing method based on denoising autoe...

Tropical and Extratropical Cyclone Detection Using Deep Learning

Extracting valuable information from large sets of diverse meteorologica...

Analysis of effectiveness of thresholding in perfusion ROI detection on T2-weighted MR images with abnormal brain anatomy

The brain perfusion ROI detection being a preliminary step, designed to ...

I Introduction

White matter hyperintensity (WMH) is commonly found in elderly people and appears to be associated with various brain diseases, such as ischemic stroke, Alzheimer s disease and multiple sclerosis [1]. The location and size of WMH regions are important biomarkers of these diseases. Manual delineation by experienced neuroradiologists is a reliable way to segment the WMH regions but it is laborious and time-consuming and has high intra- as well as inter-rater variability. As such, automatically segmenting the WMH regions is of great importance, especially in the context of large-scale neuroimaging studies. However, automatic WMH segmentation is challenging, especially for the small WMH lesions [1].

Recently, deep learning techniques have been successfully employed to segment WMH regions [2, 3, 4, 5, 6]. For example, Li and colleagues have used U-net [7] in a WMH segmentation challenge [2]. In their proposed approach, they adopted three identical U-nets with different randomly-initialized weights during training and averaged the score maps at the testing stage. U-net has been widely used in segmenting biomedical images because it can work efficiently, when the training samples are few, via data augmentation through elastic deformation [7]. However, the performance of U-net is unstable depending on the randomly initialized weights.

For a neural network, its weights are usually randomly initialized so as to break symmetry and make the network coverage faster


. The logistic sigmoid activation function has been widely used in various binary classification tasks because it can map any real number to an interval between 0 and 1

[9]. However, Glorot and Bengio found that the logistic sigmoid activation is inappropriate for deep learning networks with random initialization because its mean value may drive the top hidden layer into saturation [10].

Pre-training is usually employed to initialize the weights of a neural network to have a good starting point [11][12]

, and it is helpful for the network to converge to a local optimum during back propagation. However, Pre-training is challenging in medical imaging related classification tasks given that the number of medical images with ground truth labeling is usually very limited, especially for magnetic resonance images (MRIs). Medical image segmentation can be treated as a specific type of binary classification task, with the background being classified as one class (with label value 0) and the region of interest (ROI) being classified as another class (with label value 1).

In the work reporting the highest WMH segmentation accuracy, the authors have averaged the outputs of three U-nets to make their model more robust [2]. On such basis, to further reduce the false negative rate, we propose a new post-processing method by thresholding the outputs of the three U-nets followed by averaging. According to the dice similarity coefficient (DSC) analysis, our proposed method can boost the segmentation accuracy of [2] by 1% .

This paper is organized as follows. Section II describes the dataset and the evaluation criteria. Section III details the proposed method. Section IV presents our experimental results. Section V discuss the advantages and potential limitations of the proposed method.

Ii Background

Ii-a Dataset

The datasets used in this study came from a WMH segmentation challenge in conjunction with MICCAI 2017 [13]. There are three datasets acquired from three scanners with each dataset containing 20 subjects. For each subject, there are multi-slice FLAIR and T1 MR images along with ground truth images of WMH regions. Inhomogeneity correction was performed using SPM 12 [14] for all MR images. We performed cross-scanner analysis by using 40 subjects from two scanners as the training data and the other 20 subjects as the testing data. Please note, we used 2D images at the axial direction as our samples rather than the entire 3D images.

Ii-B Evaluation Criteria

To evaluate the accuracy of an automatically segmented WMH region, we employed the DSC score, which is also known as the similarity index. DSC is a statistic used for comparing the similarity of two sets, which is defined as


where represents the gold standard segmentation of a WMH region, represents the corresponding automatic segmentation, and refers to the overlap region. represents the sum of the entries of matrix.

Iii Method

Iii-a Preprocessing and neural network

To unify the image sizes, we cropped or padded the axial slices of each image to be of size

. Gaussian normalization was then applied to rescale the voxel intensities of each image. We also conducted data augmentation via rotation, shearing and scaling. The preprocessed FLAIR and T1 images were concatenated to form a tensor, which was served as the input to our neural network. The output was the manually-delineated ground truth segmentation mask.

We built a U-net based on the work of Li [2] and Ronneberger [7]. As shown in Fig. 1, the U-net consists of a down-convolutional part (left side) and up-convolutional part (right side). The left side aims at extracting features for classifying each voxel into WMH and non-WMH regions. And the right side aims at locating WMH regions more precisely. The down-convolutional part consists of two

convolution layers, each followed by a rectified linear unit (ReLU) and a

max pooling layer for down-sampling. For the first two convolutional layers, a kernel of size was used to handle different transformations. Each step in the up-convolutional part involved up-sampling the feature map followed by a convolution layer that reduces the number of the feature channels and two convolutions, each followed by a ReLU. The concatenation was performed between the down-convolutional and up-convolutional parts as shown in Fig. 1 using the gray line. Random initialization was used to initialize the model weights.

Fig. 1: The framwork of the training procedure

Iii-B Post-processing

U-net may suffer abnormal local optima because of the random initialization. We assembled the binarization and averaging operations in two different post-processing manners, as shown in Fig. 2. In that figure, panel (a) demonstrates the mean score maps (MSM) method in which the score maps were averaged first and then binarized, and panel (b) shows the mean binary masks (MBM) method in which the score maps were binarized first and then averaged. Both binarization and averaging were performed pixel-wisely. The output of the averaging operation is the mean value of the three input pixel values, and the binarization is conducted as below


where denotes the pixel value at the specific location in score map, which is the output of a U-net in the testing stage. And denotes the pixel value of the binary mask at the corresponding location, which can be treated as the segmentation results.

In Table I, we demonstrate the difference between MSB and MBM. The score values at a specific pixel obtained from the three randomly-initialized U-net 1, U-net 2, and U-net 3 are 0.6, 0.7, 0.1 respectively. The output of the MSM method is 0 whereas the output of the MBM is 1. A possible situation is that both U-net 1 and U-net 2 have converged to a normal optimum whereas U-net 3 suffered an abnormal local optimum. In this case, MBM performed better than MSM because it can reduce the false negative rate caused by the abnormal model which may have been induced by inappropriate randomly-initialized weights.

U-net 1 2 3
Score 0.6 0.7 0.1
Average 0.4667
Threshold 0
U-net 1 2 3
Score 0.6 0.7 0.1
Threshold 1 1 0
Average 0.6667
Threshold 1
TABLE I: An Example demonstrating how MSM (Top) and MBM (Bottom) works
Fig. 2: Demonstration of the two post-processing techniques

Iii-C Implementation

The proposed method was implemented in Python language, using Keras with Tensorflow backend. All experiments were conducted on a Linux machine running Ubuntu 16.04 with 32 GB RAM memory. The U-net training was carried out on a single GTX 1080 Ti with 11 GB RAM memory. We used Adam optimizer with initial learning rate 0.001 and 8 batch size for training.

Iv Results

Iv-a Visual examinations

In Fig. 3, we demonstrate the segmentation results for three representative slices. From top to bottom, we respectively show the FLAIR images, the ground truth segmentations overlaid on top of the FLAIR images, the results obtained from the three U-nets (U-net 1, U-net 2, and U-net 3), as well as those obtained from the post-processing methods (MSM and MBM). To better show the segmentation results, selected patches were zoomed by four times and shown on the upper left corners. To clearly reveal the difference between the ground truth segmentation and the segmentation obtained from each automatic method, the ground truth segmentation was shown in red, the automatic segmentation was shown in blue, and their overlapping regions were shown in green. As such, at the lowest five rows (from U-net 1 to MBM), green denotes the overlapping region, red denotes the false negative, and blue denotes the false positive.

Fig. 3: Representative segmentation results from the three U-nets and the two post-processing methods

From the leftmost column of Fig. 3, we find that the segmentation results from both U-net 1 and U-net 2 are reasonable, whereas that from U-net 3 is far from satisfactory. In such case, MBM performed much better than MSM. This conclusion holds for the other two examples as well (the middle column and the rightmost column).

Iv-B DSC analysis

Fig. 4 shows the distributions of the DSC scores of the three U-nets, MSM, and MBM, obtained from 20 testing subjects. It is evident that the segmentation performance of U-net 3 is relatively poor. The MBM post-processing method can improve the overall segmentation accuracy, being superior to MSM.

Fig. 4: Box plot of DSC for the 20 testing data. (MSM: mean score maps, MSM: mean binary masks)

In Table II, we tabulate a variety of statistics on the DSC scores of the three U-nets and the two post-processing methods. According to those statistics, we observe that both MSM and MBM are better than each individual U-net and MBM performed even better. With one U-net delivering poor segmentation results, both MSM and MBM can weaken the influence of that poorly-behaving U-net, delivering segmentation results that are superior to those obtained from the best U-net, but MBM is even superior to MSM.

U-net 1 U-net 2 U-net 3 MSM MBM
Mean 0.6702 0.6444 0.5759 0.6810 0.6929
Max 0.8118 0.8269 0.7742 0.8235 0.8228
Top 75% 0.7510 0.7300 0.6753 0.7722 0.7641
Median 0.7060 0.6607 0.5992 0.7066 0.7258
Top 25% 0.6204 0.5896 0.4545 0.5928 0.6329
Min 0.4256 0.3788 0.3024 0.4294 0.4618
TABLE II: Various statistics of the DSC scores for the three U-nets, MBM and MSM

In clinical application, it is not possible to guarantee that a model will necessarily converge to a good optimum like U-net 1 did. Our post-processing method can reduce the influence of a bad local optimum like U-net 3 has induced and make the model robust.

V Discussion and Conclusion

In this work, we compared two post-processing methods, MSM and MBM, for WMH segmentation in the framework of U-net based learning. We found MBM can further reduce the false negative rate compared to MSM. One potential limitation of this work is that we have not tested the proposed post-processing method in other deep learning models and applications, which will be one of our future endeavors. We believe that the post-processing method can be successfully applied to various deep learning models, especially when pre-training is not available. In addition, there is space for improvement by modifying the structure of the U-net which is the key component in our WMH segmentation approach. Lastly, applying the proposed technique to real-world clinical applications is another future plan of ours.


We thank the WMH challenge organizer Dr. Hugi J. Kuijf and the joint efforts of the UMC Ultrecht, VU Amsterdam, and NUHS Singapore for making the datasets available for this research. We thank Hongwei Li for his open source code and generous help to rebuild his method. This work was supported by the Shenzhen Science and Technology Innovation Committee funds (JCYJ20160301113918121), the National Key R&D Program of China (2017YFC0112404) and the National Natural Science Foundation of China (81501546).


  • [1] A. A. Gouw, A. Seewann, W. M. Van Der Flier, F. Barkhof, A. M. Rozemuller, P. Scheltens, and J. J. Geurts, “Heterogeneity of small vessel disease: a systematic review of MRI and histopathology correlations,” Journal of Neurology, Neurosurgery & Psychiatry, pp. jnnp–2009, 2010.
  • [2] H. Li, G. Jiang, R. Wang, J. Zhang, Z. Wang, W.-S. Zheng, and B. Menze, “Fully convolutional network ensembles for white matter hyperintensities segmentation in MR images,” arXiv preprint arXiv:1802.05203, 2018.
  • [3]

    S. Andermatt, S. Pezold, and P. Cattin, “Multi-dimensional gated recurrent units for the segmentation of biomedical 3D-data,” in

    Deep Learning and Data Labeling for Medical Applications.   Springer, 2016, pp. 142–151.
  • [4]

    M. Ghafoorian, N. Karssemeijer, T. Heskes, I. W. Uden, C. I. Sanchez, G. Litjens, F.-E. Leeuw, B. Ginneken, E. Marchiori, and B. Platel, “Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities,”

    Scientific Reports, vol. 7, no. 1, p. 5110, 2017.
  • [5] D. Jin, Z. Xu, A. P. Harrison, and D. J. Mollura, “White matter hyperintensity segmentation from T1 and FLAIR images using fully convolutional neural networks enhanced with residual connections,” arXiv preprint arXiv:1803.06782, 2018.
  • [6] S. Valverde, M. Cabezas, E. Roura, S. González-Villà, D. Pareto, J. C. Vilanova, L. Ramió-Torrentà, À. Rovira, A. Oliver, and X. Lladó, “Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach,” NeuroImage, vol. 155, pp. 159–168, 2017.
  • [7] A. A. Gouw, A. Seewann, W. M. Van Der Flier, F. Barkhof, A. M. Rozemuller, P. Scheltens, and J. J. Geurts, “Heterogeneity of small vessel disease: a systematic review of MRI and histopathology correlations,” Journal of Neurology, Neurosurgery & Psychiatry, pp. jnnp–2009, 2010.
  • [8] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in

    International conference on machine learning

    , 2013, pp. 1139–1147.
  • [9] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” Medical image analysis, vol. 42, pp. 60–88, 2017.
  • [10] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in

    Proceedings of the thirteenth international conference on artificial intelligence and statistics

    , 2010, pp. 249–256.
  • [11] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, “Why does unsupervised pre-training help deep learning?” Journal of Machine Learning Research, vol. 11, no. Feb, pp. 625–660, 2010.
  • [12] D. Yu, L. Deng, and G. Dahl, “Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition,” in Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010.
  • [13] H. J. Kuijf, “WMH segmentation challenge,” 2017, http://wmh.isi.uu.nl/.
  • [14] J. Ashburner, G. Barnes, C. Chen, J. Daunizeau, G. Flandin, K. Friston, S. Kiebel, J. Kilner, V. Litvak, R. Moran et al., “SPM 12 manual,” London: Wellcome Trust, 2014.