Automatic Segmentation and Overall Survival Prediction in Gliomas using Fully Convolutional Neural Network and Texture Analysis

by   Varghese Alex, et al.

In this paper, we use a fully convolutional neural network (FCNN) for the segmentation of gliomas from Magnetic Resonance Images (MRI). A fully automatic, voxel based classification was achieved by training a 23 layer deep FCNN on 2-D slices extracted from patient volumes. The network was trained on slices extracted from 130 patients and validated on 50 patients. For the task of survival prediction, texture and shape based features were extracted from T1 post contrast volume to train an XGBoost regressor. On BraTS 2017 validation set, the proposed scheme achieved a mean whole tumor, tumor core and active dice score of 0.83, 0.69 and 0.69 respectively and an accuracy of 52 overall survival prediction.



There are no comments yet.


page 6

page 7

page 8


Brain Tumor Segmentation and Survival Prediction using Automatic Hard mining in 3D CNN Architecture

We utilize 3-D fully convolutional neural networks (CNN) to segment glio...

Multi-region segmentation of bladder cancer structures in MRI with progressive dilated convolutional networks

Precise segmentation of bladder walls and tumor regions is an essential ...

Deep radiomic features from MRI scans predict survival outcome of recurrent glioblastoma

This paper proposes to use deep radiomic features (DRFs) from a convolut...

Training a Fully Convolutional Neural Network to Route Integrated Circuits

We present a deep, fully convolutional neural network that learns to rou...

Brain Tumor Survival Prediction using Radiomics Features

Surgery planning in patients diagnosed with brain tumor is dependent on ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In this paper, we utilize a 23 layer deep FCNN for the task of segmentation of gliomas from MR scans. In the field of medical image analysis, U-net [1] is one of the oft used architecture. The network used in this work has a similar architecture as that of the aforementioned network. The network was trained on 2-D axial slices (240

240) extracted from FLAIR, T2, T1, and T1 post contrast sequences. The architecture of the network enables semantic segmentation i.e. classify of all voxels in a slice in a single forward pass. Due the reason stated above, the inference time associated with FCNN based network are lower when compared to traditional patch based CNNs.

Convolutional neural network and its variants being deterministic approaches tend to mis-classify voxels as lesion in regions like brain stem, cerebellum where occurrence of gliomas is anatomically impossible. We utilize 3-D connected component analysis to discard components below a certain threshold for false positive reduction. Unlike previous years BraTS competitions, apart from segmentation of gliomas from MR volumes, an additional challenge of predicting the prognosis of subject using pre-operative scan was made part of the BraTS 2017 challenge.

For the prognosis challenge, the overall survival rate was categorized into three groups namely short survivors (prognosis, 10 months), mid survivors (10 months 15 months) & long survivors (

15 months). The segmentations produced by the FCNN was used to extract first order texture and shaped based features such as entropy, skewness, circularity of lesion constituent etc. The extracted features along with age of the subject was fed to an Extreme gradient boosting (XGBOOST) regressor to predict the prognosis of the subject.

2 Materials and Methods

The proposed technique comprises of following stages:

  1. Pre-processing of data

  2. Segmentation of gliomas using FCNN

  3. Post-processing using 3-D connected components

  4. Feature extraction for survival rate prediction.

  5. Prediction of survival rate using XGBoost Regressor.

The flowchart of the proposed technique is given in Fig. (1).

Figure 1: Flow chart of the proposed technique

2.1 Data

The network was trained and validated on the BraTS 2017 training data [2],[3], [4],[5]. The training data comprises of 210 HGG volumes and 75 LGG volumes collected from multiple centers. Each patient comprises of FLAIR, T2, T1, T1 post contrast and the associated ground truth labeled by experts. Each sequence was skull stripped and was re-sampled to 1mm 1mm 1mm (isotropic resolution).

For the overall survival challenge, age & prognosis of the patient post treatment were supplied by the organizers. The training set for the challenge comprised of 163 High Grade Glioma patients of which 43 patients had survival rate between 10 and 15 months (mid survivors), while 65 patients had prognosis less than 10 months (short survivors) and 56 patients had prognosis greater than 15 months (long survivors).

2.2 Fully Convolutional Neural Network

A typical FCNN comprises of convolution operations, max pooling layers and transpose convolution layers. The absence of fully connected layers in FCNNs reduces the number of parameters in the network & in-turn accepts inputs of arbitrary sizes. The max pooling layer helps in reducing the spatial dimension of the feature maps in the deeper layers and also aids in capturing translational invariant features in the data.

The dimensionality of the feature maps are brought back to size of the input by either using up-sampling modules such as bilinear interpolation of feature maps or transposed convolution. The use of transposed convolution in the networks makes the scaling procedure of feature maps a parameter to be learned during the training process. Concatenation of feature maps between different layers of the network enables the classifier in the network to make use of both low and complex level features for better classification results.

FCNNs have an inherent advantage of classifying all pixels in the image by using single forward pass of the image and thus makes FCNNs an ideal choice for semantic segmentation related tasks. Similar to traditional CNNs, the parameters of the network are learned by minimizing the cross entropy.

3 Preprocessing of Data

3.1 Histogram Matching

Multi center data and magnetic field inhomogeneities contribute to the non-uniform intensity variation in MR images. The voxel intensities of all volumes were standardized by matching histograms to an arbitrary chosen reference image from the training database, (Fig.4).

Figure 4: Histogram Matching. (a) Raw histograms of reference volume & test volume. (b) Histograms of reference & test volume post histogram matching.

3.2 Z-score Normalization

The histogram matched volumes were normalized to have zero mean and unit variance using Eq.

1, where is the MR volume, and

are the mean and standard deviation of the volume and

is the normalized volume.


4 Segmentation of Gliomas using proposed network

4.1 Network Architecture

The architecture of the network is given in Fig. (7 (a)). Each Conv

block in the network comprises of 2 sets of convolution by 3x3 kernels, batch normalization and a non linearity (ReLU), (Fig. (

7 (b))). The number of filters in each layer is given inside parenthesis in the Conv and UpConv block. The concatenation of feature maps is presented in the architecture as blue arrows.

The stride, kernel size & padding of the transposed convolution were chosen so as to produce feature maps of similar dimensions as that of the feature maps of the adjoining

Conv block. This enables concatenation of feature maps without the need of cropping feature maps from the Conv block. The network makes use of convolution with 1x1 filters in the hindmost convolution block and results in generating the segmentation map.

Figure 7: Architecture of the proposed network. (a) Proposed FCNN. (b) Composition of the Conv block.

4.2 Training

The network was trained and validated using with slices extracted from 120 and 50 HGG patients respectively. The weights and biases in each layer was initialized using the Xavier initialization [6]

. The network was trained for 30 epochs and the weights and biases were learned by minimizing the cross entropy loss function with ADAM

[7] as the optimizer.

The imbalance amongst classes in the dataset were addressed by:

  1. Training and validating the network using slices that comprises of atleast one pixel of lesion.

  2. Performing data augmentation on the extracted slices which include horizontal flipping of the data.

  3. Using a weighted cross entropy as the loss function for training the network. The weight assigned to normal, necrotic, edema and enhancing were 1, 5, 2 and 3 respectively.

4.3 Testing

During the testing phase, axial slices from all 4 sequences were fed to the trained network to generate the segmentation mask/ volumes.

4.4 Post processing

CNNs being deterministic techniques tend to mis-classify voxels as lesions at certain locations such as cerebellum, brain stem etc. were occurrence of gliomas is physiologically impossible. The false positives in predictions made by the trained network were removed by using 3-D connected component analysis. All components below a certain threshold (T=2000) were discarded while the rest were retained.

4.5 Survival prediction

The segmentation mask generated by the network was binarized to form 4 different volumes namely whole lesion mask, edema mask, necrosis mask and enhancing mask, (Fig.

13). A total of 19 first order texture based features, Table 1 and 16 shape based features of the lesion, Table 2, were extracted from T1 post contrast sequences using each of the aforementioned masks. The texture and shape based features were extracted from MR volume using a python package called Pyradiomics [8]. Apart from texture and shape based features, the age of subject was used as a feature for the regressor to train and predict the prognosis.

Figure 13: Mask generated from segmentation maps. (a) Prediction made by the segmentation network. (b) whole lesion mask. (c) edema mask. (d) necrosis mask. (e) enhancing mask. In figure (a), green, red and yellow indicate edema, necrotic core and enhancing tumor in the lesion respectively.
First order Texture Based Feature
1. Volume 11. Range
2. Total Energy 12. Mean Absolute Deviation
3. Entropy 13. Robust Mean Absolute Deviation
4. Minimum 14. Root Mean Squared
5. 10th percentile 15. Standard Deviation
6. 90th percentile 16. Skewness
7. Maximum

17. Kurtosis

8. Mean 18. Variance
9. Median 19. Uniformity
10. Interquartile Range
Table 1: First order texture features extracted from T1 post contrast sequence to predict overall survival of a subject.
Shape Based Feature
1. Volume 9. Maximum 2D diameter (coronal)
2. Surface area 10. Maximum 2D diameter (sagital)
3. Surface area to Volume Ratio 11. Major Axis
4. Sphericity 12. Minor Axis
5. Spherical Disproportion 13. Least Axis
6. Compactness 1 14. Elongation
7. Maximum 3D diameter 15. Flatness
8. Maximum 2D diameter (axial) 16. Compactness 2
Table 2: Shape based features used to predict prognosis of a subject

5 Results

The performance of network on the entire BraTS on the local test set (n=40) [HGG-25, LGG-15] is given in Table (3). Fig.(22) shows the performance of the network on 2 different patients from the local test data.

Figure 22: Performance of the network on local test data. (a) FLAIR. (b) T2. (c) Prediction. (d) Ground Truth. (e) FLAIR. (f) T2. (g) Prediction. (h) Ground Truth. In figures c,d,g & h, green- Edema, yellow- Enhancing Tumor, red- Necrotic Core.
Whole Tumor Tumor Core Active Tumor
Mean 0.84 0.84 0.77
Std. Dev. 0.19 0.20 0.19
Median 0.90 0.90 0.83
Table 3: Performance of the network on local test data (HGG=25 and LGG=15)

The post processing technique improves the performance of the network. On the local test data, the improvement in performance was in the order of 2.44% for whole tumor dice score, 2.44% for tumor core and 1.31% for active tumor. Fig.(27) shows an example were the proposed post processing technique aids in eliminating false positives.

Figure 27: Reduction of False positive using connected components. (a) FLAIR. (b) Raw Prediction. (c) Post Processed image. (d) Ground Truth. In figures b, c, d, green- Edema, yellow- Enhancing Tumor, red- Necrotic Core.

For the task of overall survival rate prediction, it was observed that texture based & shape based features extracted from T1 post contrast sequence performed better than extracting features from other MR sequences. We observed that using features extracted all four sequences had negative impact on performance of the regressor.

The performance of the network on the BraTS 2017 validation set is given in Table (6). It was observed that the network maintains similar whole tumor scores on the local test data and on the validation data. However, a dip in performance was observed in the tumor core & active tumor compartments. The performance of the proposed technique for survival prediction on the validation data is given in Table (7).

Whole Tumor Tumor Core Active Tumor
Mean 0.83 0.69 0.72
Std. Dev. 0.16 0.30 0.32
Median 0.90 0.83 0.85
Table 4: Performance of the network on BraTS 2017 validation data
Accuracy MSE Median SE Std SE SpearmanR
0.52 221203.54 59035.10 505184.81 0.27
Table 5: Survival prediction on BraTS 2017 validation data

The trained network was tested on BraTS 2017 challenge data (n=146). The performance of the proposed algorithm for the task of segmentation of gliomas from multi modal MR images is given in Table 4. For the overall survival prediction, the proposed technique achieved an accuracy of 47% & and a Spearman coefficient of 0.41, (Table 5).

The proposed technique which makes use of a single network, produces good segmentations on the challenge data & its performance was found to be comparable with techniques that uses an ensemble of networks.

Whole Tumor Tumor Core Active Tumor
Mean 0.79 0.65 0.63
Std. Dev. 0.20 0.33 0.33
Median 0.87 0.82 0.77
Table 6: Performance of the network on BraTS 2017 testing data (n=146)
Accuracy MSE Median SE Std SE SpearmanR
0.47 217755.772 39264.206 668015.465 0.411
Table 7: Survival prediction on BraTS 2017 testing data (n=95)

6 Conclusion

In this paper, we propose a fully automatic technique for segmentation of gliomas from MR volume and predict the prognosis of the patient using first order texture and shape based features. A fully convolutional neural network was utilized for task segmentation of gliomas into its various constituents namely edema, necrotic core and enhancing tumor. A 3-D connected component analysis was used to remove false positives in the predictions made by the network. The network produces good segmentation on the BraTS test data and achieved a whole tumor, tumor core and active tumor dice score of 0.79, 0.65 and 0.63 respectively. The segmentation produced by the network was used to generate 4 different masks namely whole tumor mask, edema mask, necrotic mask, enhancing mask. Using each mask 19 different first order texture features and 16 shaped based features were extracted from T1 post contrast sequence to train a XGBOOST regressor to predict the prognosis of a subject. On the BraTS 2017 validation data and test data, the regressor achieved an accuracy of 52 % and 47% respectively.


  • [1] Ronneberger O, Fischer P, Brox T. “U-net: Convolutional networks for biomedical image segmentation”. In International Conference on Medical Image Computing and Computer-Assisted Intervention 2015 Oct 5 (pp. 234-241). Springer, Cham.
  • [2] Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, Lanczi L, Gerstner E, Weber MA, Arbel T, Avants BB, Ayache N, Buendia P, Collins DL, Cordier N, Corso JJ, Criminisi A, Das T, Delingette H, Demiralp Ç, Durst CR, Dojat M, Doyle S, Festa J, Forbes F, Geremia E, Glocker B, Golland P, Guo X, Hamamci A, Iftekharuddin KM, Jena R, John NM, Konukoglu E, Lashkari D, Mariz JA, Meier R, Pereira S, Precup D, Price SJ, Raviv TR, Reza SM, Ryan M, Sarikaya D, Schwartz L, Shin HC, Shotton J, Silva CA, Sousa N, Subbanna NK, Szekely G, Taylor TJ, Thomas OM, Tustison NJ, Unal G, Vasseur F, Wintermark M, Ye DH, Zhao L, Zhao B, Zikic D, Prastawa M, Reyes M, Van Leemput K. ”The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)”, IEEE Transactions on Medical Imaging 34(10), 1993-2024 (2015
  • [3] Bakas S et al. “Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features”, Nature Scientific Data, (2017) [In Press]
  • [4] Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby J, Freymann J, Farahani K, Davatzikos C. ”Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-GBM collection”, The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.KLXWJJ1Q
  • [5] Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby J, Freymann J, Farahani K, Davatzikos C. ”Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-LGG collection”, The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.GJQ7R0EF
  • [6]

    X. Glorot et al., “Understanding the difficulty of training deep feedforward neural networks,” in International conference on artificial intelligence and statistics, pp. 249–256, 2010

  • [7] Kingma D, Ba J. “Adam: A method for stochastic optimization”. arXiv preprint arXiv:1412.6980. 2014 Dec 22
  • [8] Joost JM van Griethuysen et al., “Computational Radiomics System to Decode the Radiographic Phenotype”; Accepted Cancer Research, 2017