In this paper, we utilize a 23 layer deep FCNN for the task of segmentation of gliomas from MR scans. In the field of medical image analysis, U-net  is one of the oft used architecture. The network used in this work has a similar architecture as that of the aforementioned network. The network was trained on 2-D axial slices (240
240) extracted from FLAIR, T2, T1, and T1 post contrast sequences. The architecture of the network enables semantic segmentation i.e. classify of all voxels in a slice in a single forward pass. Due the reason stated above, the inference time associated with FCNN based network are lower when compared to traditional patch based CNNs.
Convolutional neural network and its variants being deterministic approaches tend to mis-classify voxels as lesion in regions like brain stem, cerebellum where occurrence of gliomas is anatomically impossible. We utilize 3-D connected component analysis to discard components below a certain threshold for false positive reduction. Unlike previous years BraTS competitions, apart from segmentation of gliomas from MR volumes, an additional challenge of predicting the prognosis of subject using pre-operative scan was made part of the BraTS 2017 challenge.
For the prognosis challenge, the overall survival rate was categorized into three groups namely short survivors (prognosis, 10 months), mid survivors (10 months 15 months) & long survivors (
15 months). The segmentations produced by the FCNN was used to extract first order texture and shaped based features such as entropy, skewness, circularity of lesion constituent etc. The extracted features along with age of the subject was fed to an Extreme gradient boosting (XGBOOST) regressor to predict the prognosis of the subject.
2 Materials and Methods
The proposed technique comprises of following stages:
Pre-processing of data
Segmentation of gliomas using FCNN
Post-processing using 3-D connected components
Feature extraction for survival rate prediction.
Prediction of survival rate using XGBoost Regressor.
The flowchart of the proposed technique is given in Fig. (1).
The network was trained and validated on the BraTS 2017 training data ,, ,. The training data comprises of 210 HGG volumes and 75 LGG volumes collected from multiple centers. Each patient comprises of FLAIR, T2, T1, T1 post contrast and the associated ground truth labeled by experts. Each sequence was skull stripped and was re-sampled to 1mm 1mm 1mm (isotropic resolution).
For the overall survival challenge, age & prognosis of the patient post treatment were supplied by the organizers. The training set for the challenge comprised of 163 High Grade Glioma patients of which 43 patients had survival rate between 10 and 15 months (mid survivors), while 65 patients had prognosis less than 10 months (short survivors) and 56 patients had prognosis greater than 15 months (long survivors).
2.2 Fully Convolutional Neural Network
A typical FCNN comprises of convolution operations, max pooling layers and transpose convolution layers. The absence of fully connected layers in FCNNs reduces the number of parameters in the network & in-turn accepts inputs of arbitrary sizes. The max pooling layer helps in reducing the spatial dimension of the feature maps in the deeper layers and also aids in capturing translational invariant features in the data.
The dimensionality of the feature maps are brought back to size of the input by either using up-sampling modules such as bilinear interpolation of feature maps or transposed convolution. The use of transposed convolution in the networks makes the scaling procedure of feature maps a parameter to be learned during the training process. Concatenation of feature maps between different layers of the network enables the classifier in the network to make use of both low and complex level features for better classification results.
FCNNs have an inherent advantage of classifying all pixels in the image by using single forward pass of the image and thus makes FCNNs an ideal choice for semantic segmentation related tasks. Similar to traditional CNNs, the parameters of the network are learned by minimizing the cross entropy.
3 Preprocessing of Data
3.1 Histogram Matching
Multi center data and magnetic field inhomogeneities contribute to the non-uniform intensity variation in MR images. The voxel intensities of all volumes were standardized by matching histograms to an arbitrary chosen reference image from the training database, (Fig.4).
3.2 Z-score Normalization
4 Segmentation of Gliomas using proposed network
4.1 Network Architecture
The architecture of the network is given in Fig. (7 (a)). Each Conv7 (b))). The number of filters in each layer is given inside parenthesis in the Conv and UpConv block. The concatenation of feature maps is presented in the architecture as blue arrows.
The network was trained and validated using with slices extracted from 120 and 50 HGG patients respectively. The weights and biases in each layer was initialized using the Xavier initialization 7] as the optimizer.
The imbalance amongst classes in the dataset were addressed by:
Training and validating the network using slices that comprises of atleast one pixel of lesion.
Performing data augmentation on the extracted slices which include horizontal flipping of the data.
Using a weighted cross entropy as the loss function for training the network. The weight assigned to normal, necrotic, edema and enhancing were 1, 5, 2 and 3 respectively.
During the testing phase, axial slices from all 4 sequences were fed to the trained network to generate the segmentation mask/ volumes.
4.4 Post processing
CNNs being deterministic techniques tend to mis-classify voxels as lesions at certain locations such as cerebellum, brain stem etc. were occurrence of gliomas is physiologically impossible. The false positives in predictions made by the trained network were removed by using 3-D connected component analysis. All components below a certain threshold (T=2000) were discarded while the rest were retained.
4.5 Survival prediction
The segmentation mask generated by the network was binarized to form 4 different volumes namely whole lesion mask, edema mask, necrosis mask and enhancing mask, (Fig.13). A total of 19 first order texture based features, Table 1 and 16 shape based features of the lesion, Table 2, were extracted from T1 post contrast sequences using each of the aforementioned masks. The texture and shape based features were extracted from MR volume using a python package called Pyradiomics . Apart from texture and shape based features, the age of subject was used as a feature for the regressor to train and predict the prognosis.
|First order Texture Based Feature|
|1. Volume||11. Range|
|2. Total Energy||12. Mean Absolute Deviation|
|3. Entropy||13. Robust Mean Absolute Deviation|
|4. Minimum||14. Root Mean Squared|
|5. 10th percentile||15. Standard Deviation|
|6. 90th percentile||16. Skewness|
|8. Mean||18. Variance|
|9. Median||19. Uniformity|
|10. Interquartile Range|
|Shape Based Feature|
|1. Volume||9. Maximum 2D diameter (coronal)|
|2. Surface area||10. Maximum 2D diameter (sagital)|
|3. Surface area to Volume Ratio||11. Major Axis|
|4. Sphericity||12. Minor Axis|
|5. Spherical Disproportion||13. Least Axis|
|6. Compactness 1||14. Elongation|
|7. Maximum 3D diameter||15. Flatness|
|8. Maximum 2D diameter (axial)||16. Compactness 2|
The performance of network on the entire BraTS on the local test set (n=40) [HGG-25, LGG-15] is given in Table (3). Fig.(22) shows the performance of the network on 2 different patients from the local test data.
|Whole Tumor||Tumor Core||Active Tumor|
The post processing technique improves the performance of the network. On the local test data, the improvement in performance was in the order of 2.44% for whole tumor dice score, 2.44% for tumor core and 1.31% for active tumor. Fig.(27) shows an example were the proposed post processing technique aids in eliminating false positives.
For the task of overall survival rate prediction, it was observed that texture based & shape based features extracted from T1 post contrast sequence performed better than extracting features from other MR sequences. We observed that using features extracted all four sequences had negative impact on performance of the regressor.
The performance of the network on the BraTS 2017 validation set is given in Table (6). It was observed that the network maintains similar whole tumor scores on the local test data and on the validation data. However, a dip in performance was observed in the tumor core & active tumor compartments. The performance of the proposed technique for survival prediction on the validation data is given in Table (7).
|Whole Tumor||Tumor Core||Active Tumor|
|Accuracy||MSE||Median SE||Std SE||SpearmanR|
The trained network was tested on BraTS 2017 challenge data (n=146). The performance of the proposed algorithm for the task of segmentation of gliomas from multi modal MR images is given in Table 4. For the overall survival prediction, the proposed technique achieved an accuracy of 47% & and a Spearman coefficient of 0.41, (Table 5).
The proposed technique which makes use of a single network, produces good segmentations on the challenge data & its performance was found to be comparable with techniques that uses an ensemble of networks.
|Whole Tumor||Tumor Core||Active Tumor|
|Accuracy||MSE||Median SE||Std SE||SpearmanR|
In this paper, we propose a fully automatic technique for segmentation of gliomas from MR volume and predict the prognosis of the patient using first order texture and shape based features. A fully convolutional neural network was utilized for task segmentation of gliomas into its various constituents namely edema, necrotic core and enhancing tumor. A 3-D connected component analysis was used to remove false positives in the predictions made by the network. The network produces good segmentation on the BraTS test data and achieved a whole tumor, tumor core and active tumor dice score of 0.79, 0.65 and 0.63 respectively. The segmentation produced by the network was used to generate 4 different masks namely whole tumor mask, edema mask, necrotic mask, enhancing mask. Using each mask 19 different first order texture features and 16 shaped based features were extracted from T1 post contrast sequence to train a XGBOOST regressor to predict the prognosis of a subject. On the BraTS 2017 validation data and test data, the regressor achieved an accuracy of 52 % and 47% respectively.
-  Ronneberger O, Fischer P, Brox T. “U-net: Convolutional networks for biomedical image segmentation”. In International Conference on Medical Image Computing and Computer-Assisted Intervention 2015 Oct 5 (pp. 234-241). Springer, Cham.
-  Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, Lanczi L, Gerstner E, Weber MA, Arbel T, Avants BB, Ayache N, Buendia P, Collins DL, Cordier N, Corso JJ, Criminisi A, Das T, Delingette H, Demiralp Ç, Durst CR, Dojat M, Doyle S, Festa J, Forbes F, Geremia E, Glocker B, Golland P, Guo X, Hamamci A, Iftekharuddin KM, Jena R, John NM, Konukoglu E, Lashkari D, Mariz JA, Meier R, Pereira S, Precup D, Price SJ, Raviv TR, Reza SM, Ryan M, Sarikaya D, Schwartz L, Shin HC, Shotton J, Silva CA, Sousa N, Subbanna NK, Szekely G, Taylor TJ, Thomas OM, Tustison NJ, Unal G, Vasseur F, Wintermark M, Ye DH, Zhao L, Zhao B, Zikic D, Prastawa M, Reyes M, Van Leemput K. ”The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)”, IEEE Transactions on Medical Imaging 34(10), 1993-2024 (2015
-  Bakas S et al. “Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features”, Nature Scientific Data, (2017) [In Press]
-  Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby J, Freymann J, Farahani K, Davatzikos C. ”Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-GBM collection”, The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.KLXWJJ1Q
-  Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby J, Freymann J, Farahani K, Davatzikos C. ”Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-LGG collection”, The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.GJQ7R0EF
X. Glorot et al., “Understanding the difficulty of training deep feedforward neural networks,” in International conference on artificial intelligence and statistics, pp. 249–256, 2010
-  Kingma D, Ba J. “Adam: A method for stochastic optimization”. arXiv preprint arXiv:1412.6980. 2014 Dec 22
-  Joost JM van Griethuysen et al., “Computational Radiomics System to Decode the Radiographic Phenotype”; Accepted Cancer Research, 2017