Partial supervision for the FeTA challenge 2021

11/03/2021
by   Lucas Fidon, et al.
5

This paper describes our method for our participation in the FeTA challenge2021 (team name: TRABIT). The performance of convolutional neural networks for medical image segmentation is thought to correlate positively with the number of training data. The FeTA challenge does not restrict participants to using only the provided training data but also allows for using other publicly available sources. Yet, open access fetal brain data remains limited. An advantageous strategy could thus be to expand the training data to cover broader perinatal brain imaging sources. Perinatal brain MRIs, other than the FeTA challenge data, that are currently publicly available, span normal and pathological fetal atlases as well as neonatal scans. However, perinatal brain MRIs segmented in different datasets typically come with different annotation protocols. This makes it challenging to combine those datasets to train a deep neural network. We recently proposed a family of loss functions, the label-set loss functions, for partially supervised learning. Label-set loss functions allow to train deep neural networks with partially segmented images, i.e. segmentations in which some classes may be grouped into super-classes. We propose to use label-set loss functions to improve the segmentation performance of a state-of-the-art deep learning pipeline for multi-class fetal brain segmentation by merging several publicly available datasets. To promote generalisability, our approach does not introduce any additional hyper-parameters tuning.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

05/27/2020

Segmentation Loss Odyssey

Loss functions are one of the crucial ingredients in deep learning-based...
04/22/2019

Reducing the Hausdorff Distance in Medical Image Segmentation with Convolutional Neural Networks

The Hausdorff Distance (HD) is widely used in evaluating medical image s...
04/04/2019

Improved Inference via Deep Input Transfer

Although numerous improvements have been made in the field of image segm...
03/08/2019

Joint Learning of Brain Lesion and Anatomy Segmentation from Heterogeneous Datasets

Brain lesion and anatomy segmentation in magnetic resonance images are f...
07/13/2021

Learning from Partially Overlapping Labels: Image Segmentation under Annotation Shift

Scarcity of high quality annotated images remains a limiting factor for ...
08/16/2020

Training CNN Classifiers for Semantic Segmentation using Partially Annotated Images: with Application on Human Thigh and Calf MRI

Objective: Medical image datasets with pixel-level labels tend to have a...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

Objective

The Fetal Brain Tissue Annotation and Segmentation Challenge (FeTA) aims at comparing algorithms for multi-class automatic segmentation of fetal brain 3D T2 MRI. Seven tissue types are considered [17]:

  1. extra-axial cerebrospinal fluid,

  2. cortical gray matter,

  3. white matter,

  4. ventricular system,

  5. cerebellum,

  6. deep gray matter,

  7. brainstem.

This paper describes our method for our participation in the FeTA challenge 2021 (team name: TRABIT).

The performance of convolutional neural networks for medical image segmentation is thought to correlate positively with the number of training data [1]. The FeTA challenge does not restrict participants to using only the provided training data but also allows for using other publicly available sources. Yet, open access fetal brain data remains limited. An advantageous strategy could thus be to expand the training data to cover broader perinatal brain imaging sources. Perinatal brain MRIs, other than the FeTA challenge data, that are currently publicly available, span normal and pathological fetal atlases as well as neonatal scans [8, 9, 11, 22]. However, perinatal brain MRIs segmented in different datasets typically come with different annotation protocols. This makes it challenging to combine those datasets to train a deep neural network.

We recently proposed a family of loss functions, the label-set loss functions [5], for partially supervised learning. Label-set loss functions allow to train deep neural networks with partially segmented images, i.e. segmentations in which some classes may be grouped into super-classes. We propose to use label-set loss functions [5] to improve the segmentation performance of a state-of-the-art deep learning pipeline for multi-class fetal brain segmentation by merging several publicly available datasets. To promote generalisability, our approach does not introduce any additional hyper-parameters tuning.

Methods and Materials

Figure 1: Overview of the training process with partial supervision. The 3D MRIs from datasets a. [17] and b.[8, 9] are fully-segmented while the 3D MRIs from dataset c. [11] were only segmented manually for white matter, ventricular system, cerebellum, and the full brain. We propose to use a label-set loss function [5] to allow to train a 3D U-Net [2] with fully and partially segmented 3D MRIs.

In this section, we give the detail of our segmentation pipeline and the data used for training the deep neural networks. Our segmentation software is publicly available at https://github.com/LucasFidon/feta-inference.

0.0.1 FeTA challenge amended training data.

The original FeTA challenge training data provides fetal brain 3D T2 MRIs with manual manual segmentations of all 7 target tissue types [17]. fetal brain 3D MRIs were reconstructed using MIAL [19] and fetal brain 3D MRIs were reconstructed using Simple IRTK [12].

For the MIAL 3D MRIs, corrections of the segmentations were performed by authors MA, LF, and PD using ITK-SNAP [23] to reduce the variability against the published segmentation guidelines that was released with the FeTA dataset [17]. Those corrections were performed as part of our previous work [5] and are publicly available (DOI: 10.5281/zenodo.5148612). Two spina bifida cases were excluded (sub-feta007 and sub-feta009) because we considered that the image quality did not allow to segment them reliably for all the tissue types. Only the remaining 3D MRIs were used for training.

0.0.2 Other public training data.

We also included average fetal brain 3D T2 MRIs from a neurotypical fetal brain atlas111http://crl.med.harvard.edu/research/fetal_brain_atlas/ [9], average fetal brain 3D T2 MRIs from a spina bifida fetal brain atlas222https://www.synapse.org/#!Synapse:syn25887675/wiki/611424 [8]. Segmentations for all 7 tissue types are available for all the atlas data. In addition, we used neonatal brain MRIs from the developing human connectome project [11] (dHCP data release 2). We excluded the brain MRIs of babies with a gestational age higher than weeks. We started from the brain masks and the automatic segmentations publicly available for the neonatal brain MRIs for white matter, ventricular system, and cerebellum [13] and we modified them manually to match the annotation protocol of the FeTA dataset [17] using ITK-SNAP [23]. Ground-truth segmentations for the other tissue types in the dHCP data were not available for our training.

0.0.3 Pre-processing.

A brain mask for the fetal brain 3D MRI is computed by affine registration of template volumes from two fetal brain atlases [8, 9]. We use all the template volumes with a gestational age that does not differ to the gestation age of fetal brain by more than weeks. The affine registrations are computed using a symmetric block-matching approach [14] as implemented in NiftyReg [15]. The affine transformations are initialized by a translation that aligns the centre of gravity of the non zero intensity regions of the two volumes. The brain mask is obtained by averaging the warped brain mask and thresholding at .

After a brain mask has been computed, the fetal brain 3D MRI is registered rigidly to a neurotycal fetal brain atlas [9] and the 3D MRI resampled to a resolution of mm isotropic. The rigid registration is computed using NiftyReg [14, 15] and the transformation is initialized by a translation that aligns the centre of gravity of the brain masks.

0.0.4 Deep learning pipeline.

We used an ensemble of 3D U-Nets [2]. We used the DynU-Net of MONAI [16] to implement a 3D U-Net with one input block, down-sampling blocks, one bottleneck block, upsampling blocks, features in the first level, instance normalization [20]

, and leaky-ReLU with slope

. An illustration of the architecture is provided in Fig. 1. The CNN used has trainable parameters. The patch size was set to . All the pre-processed 3D MRIs on which the pipeline was tested fitted inside a patch of size . Every input volume is skull stripped after dilating the brain mask by

voxels and cropped or padded with zeros to fit the patch size. The non zeros image intensity values are clipped for the values above percentile

, and normalized to zeros mean and unit variance. Test-time augmentation 

[21] with all the combinations of flipping along the three spatial dimensions is performed ( predictions). The score map predictions are averaged to obtain the output of each CNN. Ensembling is obtained by averaging the softmax predictions of the CNNs. The deep learning pipeline was implemented using MONAI v by authors LF and SS.

0.0.5 Loss function.

We used the sum of two label-set loss functions as loss function: the Leaf-Dice loss [5] and the marginalized cross entropy loss [5].

0.0.6 Optimization.

For each network in the ensemble, the training dataset was split into training and validation at random. The random initialization of the 3D U-Net weights was performed using He initialization [10]

. We used SGD with Nesterov momentum, batch size

, weight decay , initial learning rate , and polynomial learning rate decay with power for a total of epochs. The CNN parameters used at inference corresponds to the last epoch. We used deep supervision with levels during training. Training each 3D U-Net required GB of GPU memory and took on average days. We have trained exactly CNNs and used all of them for the ensemble submitted to the challenge.

0.0.7 Data augmentation.

We used random zoom (zoom ratio range

drawn uniformly at random; probability of augmentation

), random rotation (rotation angle range for all dimensions drawn uniformly at random; probability of augmentation ), random additive Gaussian noise (mean

, standard deviation

; probability of augmentation ), random Gaussian spatial smoothing (standard deviation range in voxels for all dimensions drawn uniformly at random; probability of augmentation ), random gamma augmentation (gamma range drawn uniformly at random; probability of augmentation ), and random flip along all dimension (probability of augmentation for each dimension).

0.0.8 Post-processing.

The mean softmax prediction is resampled to the original 3D MRI using the inverse of the rigid transformation computed in the pre-processing step to register the 3D MRI to the template space. This image registration is computed using NiftyReg [15]

with an interpolation order equal to

. After resampling, the final multi-class segmentation prediction is obtained by taking the argmax of the mean softmax.

Results

We evaluated our method on 20 spina bifida MRIs acquired at University Hospital Leuven that were previously used in [5, 6] using the Dice score [3, 7]

. Those 20 3D MRIs and their brain masks were computed using the super-resolution and reconstruction software

NiftyMIC [4, 18]. Our method achieves mean (standard deviation) Dice scores: white matter (), ventricular system (), cerebellum (), extra-axial CSF (), cortical grey matter (), deep grey matter (), and brainstem ().

Conclusion

Partially supervised learning can be used to train deep neural networks using multiple publicly available perinatal brain 3D MRI datasets that have different level of segmentations available. We used label-set loss functions [5] to train an ensemble of 3D U-Nets using four publicly available datasets [8, 9, 11, 17]. We have submitted our segmentation algorithm to the FeTA challenge 2021.

Funding Sources

This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement TRABIT No 765148. This work was supported by core and project funding from the Wellcome [203148/Z/16/Z; 203145Z/16/Z; WT101957], and EPSRC [NS/A000049/1; NS/A000050/1; NS/A000027/1]. TV is supported by a Medtronic / RAEng Research Chair [RCSRF1819\7\34].

References

  • [1] Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R.T., Berger, C., Ha, S.M., Rozycki, M., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018)
  • [2] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. pp. 424–432. Springer (2016)
  • [3] Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)
  • [4] Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P.A., Aughwane, R., Melbourne, A., Doel, T., Dymarkowski, S., De Coppi, P., et al.: An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI. NeuroImage 206, 116324 (2020)
  • [5] Fidon, L., Aertsen, M., Emam, D., Mufti, N., Guffens, F., Deprest, T., Demaerel, P., David, A.L., Melbourne, A., Ourselin, S., et al.: Label-set loss functions for partial supervision: application to fetal brain 3d mri parcellation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 647–657. Springer (2021)
  • [6]

    Fidon, L., Aertsen, M., Mufti, N., Deprest, T., Emam, D., Guffens, F., Schwartz, E., Ebner, M., Prayer, D., Kasprian, G., et al.: Distributionally robust segmentation of abnormal fetal brain 3d mri. In: Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis, pp. 263–273. Springer (2021)

  • [7] Fidon, L., Li, W., Garcia-Peraza-Herrera, L.C., Ekanayake, J., Kitchen, N., Ourselin, S., Vercauteren, T.: Generalised wasserstein dice score for imbalanced multi-class segmentation using holistic convolutional networks. In: International MICCAI Brainlesion Workshop. pp. 64–76. Springer (2017)
  • [8] Fidon, L., Viola, E., Mufti, N., David, A., Melbourne, A., Demaerel, P., Ourselin, S., Vercauteren, T., Deprest, J., Aertsen, M.: A spatio-temporal atlas of the developing fetal brain with spina bifida aperta. Open Research Europe 1(123) (2021). https://doi.org/10.12688/openreseurope.13914.1
  • [9] Gholipour, A., Rollins, C.K., Velasco-Annis, C., Ouaalam, A., Akhondi-Asl, A., Afacan, O., Ortinau, C.M., Clancy, S., Limperopoulos, C., Yang, E., et al.: A normative spatiotemporal MRI atlas of the fetal brain for automatic segmentation and analysis of early brain growth. Scientific reports 7(1), 1–13 (2017)
  • [10]

    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp. 1026–1034 (2015)

  • [11] Hughes, E.J., Winchman, T., Padormo, F., Teixeira, R., Wurie, J., Sharma, M., Fox, M., Hutter, J., Cordero-Grande, L., Price, A.N., et al.: A dedicated neonatal brain imaging system. Magnetic resonance in medicine 78(2), 794–804 (2017)
  • [12]

    Kuklisova-Murgasova, M., Quaghebeur, G., Rutherford, M.A., Hajnal, J.V., Schnabel, J.A.: Reconstruction of fetal brain mri with intensity matching and complete outlier removal. Medical image analysis

    16(8), 1550–1564 (2012)
  • [13] Makropoulos, A., Robinson, E.C., Schuh, A., Wright, R., Fitzgibbon, S., Bozek, J., Counsell, S.J., Steinweg, J., Vecchiato, K., Passerat-Palmbach, J., et al.: The developing human connectome project: A minimal processing pipeline for neonatal cortical surface reconstruction. Neuroimage 173, 88–112 (2018)
  • [14] Modat, M., Cash, D.M., Daga, P., Winston, G.P., Duncan, J.S., Ourselin, S.: Global image registration using a symmetric block-matching approach. Journal of Medical Imaging 1(2), 024003 (2014)
  • [15] Modat, M., Ridgway, G.R., Taylor, Z.A., Lehmann, M., Barnes, J., Hawkes, D.J., Fox, N.C., Ourselin, S.: Fast free-form deformation using graphics processing units. Computer methods and programs in biomedicine 98(3), 278–284 (2010)
  • [16] MONAI Consortium: MONAI: Medical open network for AI (3 2020). https://doi.org/10.5281/zenodo.4323058, https://github.com/Project-MONAI/MONAI
  • [17] Payette, K., de Dumast, P., Kebiri, H., Ezhov, I., Paetzold, J.C., Shit, S., Iqbal, A., Khan, R., Kottke, R., Grehten, P., et al.: An automatic multi-tissue human fetal brain segmentation benchmark using the fetal tissue annotation dataset. Scientific Data 8(1), 1–14 (2021)
  • [18] Ranzini, M., Fidon, L., Ourselin, S., Modat, M., Vercauteren, T.: MONAIfbs: MONAI-based fetal brain MRI deep learning segmentation. arXiv preprint arXiv:2103.13314 (2021)
  • [19] Tourbier, S., Bresson, X., Hagmann, P., Thiran, J.P., Meuli, R., Cuadra, M.B.: An efficient total variation algorithm for super-resolution in fetal brain mri with adaptive regularization. NeuroImage 118, 584–597 (2015)
  • [20] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
  • [21]

    Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T.: Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing

    338, 34–45 (2019)
  • [22] Wu, J., Sun, T., Yu, B., Li, Z., Wu, Q., Wang, Y., Qian, Z., Zhang, Y., Jiang, L., Wei, H.: Age-specific structural fetal brain atlases construction and cortical development quantification for chinese population. NeuroImage p. 118412 (2021)
  • [23] Yushkevich, P.A., Gao, Y., Gerig, G.: ITK-SNAP: an interactive tool for semi-automatic segmentation of multi-modality biomedical images. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 3342–3345. IEEE (2016)