Automatic segmentation methods have the potential to provide accurate and reproducible labels leading to improved tumor prognosis and treatment planning, especially for cases where access to expert radiologists is limited.
In the BraTS competition, we seek to segment multimodal MR images of glioma patients. Common brain MRI modalities include post-Gadolinium T1 (used to enhance contrast and visualization of the blood-brain barrier), T2 and FLAIR (to highlight different tissue fluid intensities), and T1. We use the data for these four modalities to generate the segmentations using a methodology that we outline below.
Contributions: In most image classification tasks deep neural networks (DNNs) have been a very powerful technique that tends to outperform other approaches and BraTS is no different. From past BraTS competitions two main DNN architectures have emerged: DeepMedic  and U-Net 
. How can we further improve this approach? Most research efforts have been on further improving these architectures, as well as coupling them with post-processing and ensemble techniques. In our work here, we propose a framework to work around the relatively small training datasets used in the BraTS competition. Indeed, in comparison to other popular classification challenges like ImageNet (which consists of one million images for training), the BraTS training set contains only 285 instances (multimodal 3D MR images), a number that is several orders of magnitude smaller than the typical number of instances required for DNNs to work well. These observations have motivated this work, whose contributions we summarize below.
Data augmentation: We propose a biophysics based domain adaptation strategy to add synthetic tumor-bearing MR images to the training examples. There have been many notable works to simulate tumor growth (see [18, 9, 6, 7, 12, 5]). We use an in-house PDE based multispecies tumor growth model  to simulate synthetic tumors. Since simulated data does not contain the correct intensities distribution of a real MR image, we train an auxiliary neural network to transform the simulated images to match real MRIs. This network gets a multimodal input and transforms this data to match the distribution of BraTS images by imposing certain cycle consistency constraints. As we will show, this is a very promising approach.
Extended segmentation: We extend the segmentation to the healthy parenchyma. This is done in two steps. First, we segment the training dataset using an atlas-based ensemble registration (using an in-house diffeomorphic registration code). Second, we train our DNN network to segment both tumor and healthy tissue (four classes, glial matter, cerebrospinal fluid, gray, and white matter). Our approach adds important information about healthy tissue delineation, which is actually used by radiologists. It also reduces the inherent class imbalance problem.
Our data augmentation is different than the recent work of , which uses GANs to automatically generate data. To the best of our knowledge, our work here is the first to use a biophysics based domain adaptation framework for automatic data generation, and our approach achieves five percentage points higher dice score as compared to , even though we use a 2D neural network architecture (which has suboptimal performance as compared to 3D network used in ).
Recently, deep learning approaches using convolutional neural networks (CNNs) have demonstrated excellent performance in semantic segmentation tasks in medical imaging. Seminal works for segmentation stem from fully-convolutional networks (FCNs). U-Net  is another popular architecture for medical segmentation, which merges feature maps from the contracting path of an FCN to its expanding path to preserve local contextual information. Multiscale information is often incorporated by using parallel convolutional pathways of various resolutions  or by using dilated convolutions and cascading network architectures . Post-processing and ensemble methods are also usually used after training with these models. The most commonly used post processing step is Conditional Random Fields (CRF) , which has been found to significantly reduce false positives and sharpen the segmentation. Ensembling is also very important to reduce overfitting with deep neural networks. The winning algorithm of the Multimodal Brain Tumor Image Segmentation Benchmark (BraTS) challenge in 2017 was based on Ensembles of Multiple Models and Architectures (EMMA) , which bagged a heterogeneous collection of networks (including DeepMedic (winner of ISLES 2015 ), U-Nets and FCNs) to build a robust and generic segmentation model.
There are established techniques to address training with small datasets, such as regularization, or ensembling, which was the approach taken by the winning team of BraTS’17. However, in this paper we propose an orthogonal method to address this problem.
Limitations: Currently, our framework only supports 2D domain transformations. Hence, we are limited to transforming 3D brains slice-by-slice and using only 2D neural network architectures. This is sub-optimal as 3D CNNs can demonstrably utilize volumetric medical imaging data more efficiently leading to better and more robust performance (see [11, 10, 8]). Hence, extending our framework to 3D is the focus of our future work and can potentially lead to greater improvements in performance.
The outline of the paper is as follows. In §2, we discuss the methodology for domain adaptation (§2.1), and the whole brain segmentation (§2.2) . In §3 we present preliminary results for the BraTS’18 challenge [14, 1, 2, 3]. Our method achieves a Dice score of [79.15,90.81,81.91] for enhancing tumor, whole tumor and tumor core, respectively for the BraTS’18 validation dataset.
2.1 Domain Adaptation
As mentioned above, one of the main challenges in medical imaging is the scarcity of training data. To address this issue, we use a novel domain adaptation strategy and generate synthetic tumor-bearing MR images to enrich the training dataset. This is performed by first solving an in-house PDE based multispecies tumor model using an atlas brain . This model captures the time evolution of enhancing and necrotic tumor concentrations along with tumor-induced brain edema. The governing equations for the model are reaction-diffusion-advection equations for the tumor species along with a diffusion equation for oxygen and other nutrients. We couple this model with linear elasticity equations with variable elasticity material properties to simulate the deformation of surrounding brain tissue due to tumor growth, also known as “mass effect”. However, this data cannot be used directly due to the difference in intensity distributions between a BraTS MRI scan and a synthetic MRI scan. Directly using synthetic data during the training process will adversely guide the neural network to learn features which do not exist in a real MR image, resulting in poor performance.
To address this issue, we use CycleGAN  to perform domain adaptation from the generated synthetic data to the real BraTS images. This is done by learning a mapping such that the distribution of images from is indistinguishable from the distribution using an adversarial loss, as shown in Fig. LABEL:f:cyclegan. Here, is the simulated tumor data, and is the corresponding data which matches the BraTS distribution. Because this mapping is highly under-constrained, it is coupled with an inverse mapping and a cycle consistency loss is introduced to enforce (and vice versa).
For training the domain adaptation network, we first computationally simulate synthetic tumors in a healthy brain atlas, located approximately at the whole tumor center taken from each BraTS image. Hence, every synthetic tumorous brain is paired with the corresponding data from a real BraTS image. Then, we perform a pre-processing step to transform our synthetic results to intensities. We produce a segmentation map for every tissue (healthy and tumorous) class and sample intensities for each class from a real MRI scan. We assign these sampled intensities to every voxel in our synthetic segmentation map to finally obtain our synthetic MRI scans. Then, we train with these pre-processed synthetic MRI scans and their corresponding BraTS images. Samples of our adaptation results are shown Fig. 1, which demonstrate an almost indistinguishable adaptation of the simulated data with the real images.
2.2 Whole Brain Segmentation With Healthy Tissues
An orthogonal approach that we propose for data augmentation, is an extended segmentation BraTS training data. That is, we segment the healthy parenchyma int gray/white matter, cerebrospinal fluid, and glial cells. The delineation of these healthy tissues contain important information which is actually used by radiologists. For example, the delineation of the tissues could be compressed due to tumor growth in the confined space of the brain. Providing this information to the classifier can help in better segmenting tumorous regions. However, such data is not readily available in the BraTS training dataset, since labelling the tumorous regions itself is laborious, let alone annotating full healthy tissues which is orders of magnitude more time consuming. We propose a novel automated approach to compute this information through image registration. In our method, we only need one (or preferably a few) fully segmented brains. Then given an input 3D brain, we perform the following automatic steps to obtain the extended segmentation:
Affine registration of each atlas image to the brats image.
Diffeomorphic registration of each atlas image to the BraTS image: This step aims to find a deformation map that would “translate” a healthy atlas to match the structure of a given BraTS training example. We compute this deformation by solving a PDE-constrained optimization problem. We refer to [16, 15, 17] for details on solving this optimization problem.
Majority voting to fuse labels of all deformed atlases to get the final healthy tissue segmentation: The votes are weighted with the quality of diffeomorphic registration measured by the norm of the residual between each deformed atlas and brats image. This ensures the highest weight for the deformed atlas closest to the BraTS image.
We show an exemplary segmentation for an MRI scan from the BraTS training data in Fig. 2.
Here, we report our segmentation results on BRATS’18 dataset.
Baseline network for healthy and tumor segmentation
We first obtain the healthy tissue segmentation for all the BraTS training data using the image registration method discussed above, and use the fine grained data to train a neural network. Given that our current domain adaptation framework only supports 2D transformations, we follow a two stage segmentation routine using both a 3D and a 2D UNet. The 3D U-Net has ten layers with multiclass dice loss (based on the works of 
, implemented in TensorFlow/Keras) as the baseline network to localize the tumor. Then we train a second 2D U-Net with the domain adaptation results to obtain the final segmentation. The training is performed for 500 epochs using a five-fold validation split of the training data with ADAM optimizer and ensemble the splits to obtain the baseline results. We show the healthy segmentation for a validation MRI scan in Fig.3.
Data augmentation through domain adaptation
In order to avoid noisy segmentations and reduce the class imbalance inherent in the BraTS dataset, we create crops of the tumor regions in every slice and use them for training. During testing, we use the segmentation generated by our baseline 3D model to create crops around the whole tumor. We use fixed sizes for our crops (specifically and
). This is to ensure no loss of information due to strided operations when we go deeper in the neural network. We train three U-Nets corresponding to the axial, sagittal or coronal view of the MRI scan and ensemble them (similar to the multi view fusion method outlined in). As before, we train five-fold cross-validation splits and ensemble them to avoid overfitting to the training data.
To augment our data with domain adaptation results, we simulate a synthetic tumor in our atlas corresponding to the whole tumor center of mass of every BraTS training image. We transfer the synthetic brain to the BraTS domain for every axial slice. Hence, our augmented dataset consists of approximately twice the amount of training brains. Our final neural network is the 2D multiview (MV) U-Net with masks generated using the baseline and data augmentation using domain adaptation.
We trained the framework using the BraTS’18 data. The fine-grained segmentation result from the first stage 3D U-Net is shown in Fig. 3. As one can see, this involves both the tumor segmentation, shown in red/yellow/green, as well as healthy structure of the brain shown in purple/cyan/gray. This data is used for localizing the tumor boundaries. We then use this data and create multiview slices around the tumor bearing region. The multiviews include the three directions of axial, sagittal, and coronal directions. Then, this data is passed through the second stage 2D U-Net which was trained along with the domain adaptation data, and fused together to obtain the final segmentation as shown in Fig. 5.
We show quantitative values for the Dice score in Tab. 1, with the corresponding box plots shown in Fig. 4. The baseline network has a dice score of [73.86,89.49,79.94] for Enhancing Tumor (ET), Whole Tumor (WT), and Tumor Core (TC). Using our proposed data augmentation framework leads to a dice score of [79.15,90.81,81.91]. These could be further improved by using a 3D network instead of a 2D one, by developing a 3D domain adaptation framework which is part of our future work.
We presented a new framework for biophysics-based medical image segmentation. Our contributions include an automatic healthy tissue segmentation of the BraTS dataset, and a novel Generative Adversarial Network to enrich the training dataset using a model to generate synthetic phenomenological structures of a glioma. We demonstrated that our approach yields promising results on the BraTS’18 validation dataset. Our framework is not specific to a particular model, and could be used with other proposed neural networks for the BraTS challenge.
-  Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J., Freymann, J., Farahani, K., Davatzikos, C.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Nature Scientific Data (2017), in press
-  Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J., Freymann, J., Farahani, K., Davatzikos, C.: Segmentation labels for the pre-operative scans of the TCGA-GBM collection (2017), http://doi.org/10.7937/k9/tcia.2017.klxwjj1q
-  Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J., Freymann, J., Farahani, K., Davatzikos, C.: Segmentation labels for the pre-operative scans of the TCGA-LGG collection (2017), http://doi.org/10.7937/k9/tcia.2017.gjq7r0ef
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. pp. 248–255. Ieee (2009)
-  Gholami, A.: Fast Algorithms for Biophysically-Constrained Inverse Problems in Medical imaging. Ph.D. thesis, The University of Texas at Austin (2017)
-  Hawkins-Daarud, A., Rockne, R.C., Anderson, A.R.A., Swanson, K.R.: Modeling tumor-associated edema in gliomas during anti-angiogenic therapy and its imapct on imageable tumor. Front Oncol 3(66) (2013)
-  Hawkins-Daarud, A., van der Zee, K.G., Tinsley Oden, J.: Numerical simulation of a thermodynamically consistent four-species tumor growth model. International journal for numerical methods in biomedical engineering 28(1), 3–24 (2012)
-  Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., Maier-Hein, K.H.: Brain tumor segmentation and radiomics survival prediction: Contribution to the BRATS 2017 challenge. CoRR abs/1802.10508 (2018), http://arxiv.org/abs/1802.10508
-  Ivkovic, S., Beadle, C., Noticewala, S., Massey, S.C., Swanson, K.R., Toro, L.N., Bresnick, A.R., Canoll, P., Rosenfeld, S.S.: Direct inhibition of myosin ii effectively blocks glioma invasion in the presence of multiple motogens. Molecular biology of the cell 23(4), 533–542 (2012)
-  Kamnitsas, K., Bai, W., Ferrante, E., McDonagh, S.G., Sinclair, M., Pawlowski, N., Rajchl, M., Lee, M.C.H., Kainz, B., Rueckert, D., Glocker, B.: Ensembles of multiple models and architectures for robust brain tumour segmentation. CoRR abs/1711.01468 (2017), http://arxiv.org/abs/1711.01468
-  Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical Image Analysis 36, 61 – 78 (2017), http://www.sciencedirect.com/science/article/pii/S1361841516301839
-  Lima, E., Oden, J., Hormuth, D., Yankeelov, T., Almeida, R.: Selection, calibration, and validation of models of tumor growth. Mathematical Models and Methods in Applied Sciences 26(12), 2341–2368 (2016)
-  Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. CoRR abs/1411.4038 (2014), http://arxiv.org/abs/1411.4038
-  Maier, O., Menze, B.H., von der Gablentz, J., Häni, L., Heinrich, M.P., Liebrand, M., Winzeck, S., Basit, A., Bentley, P., Chen, L., Christiaens, D., Dutil, F., Egger, K., Feng, C., Glocker, B., Götz, M., Haeck, T., Halme, H.L., Havaei, M., Iftekharuddin, K.M., Jodoin, P.M., Kamnitsas, K., Kellner, E., Korvenoja, A., Larochelle, H., Ledig, C., Lee, J.H., Maes, F., Mahmood, Q., Maier-Hein, K.H., McKinley, R., Muschelli, J., Pal, C., Pei, L., Rangarajan, J.R., Reza, S.M.S., Robben, D., Rueckert, D., Salli, E., Suetens, P., Wang, C.W., Wilms, M., Kirschke, J.S., Krämer, U.M., Münte, T.F., Schramm, P., Wiest, R., Handels, H., Reyes, M.: Isles 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. Medical Image Analysis 35 (2017)
-  Mang, A., Biros, G.: A semi-lagrangian two-level preconditioned newton–krylov solver for constrained diffeomorphic image registration. SIAM Journal on Scientific Computing 39(6) (2017)
-  Mang, A., Gholami, A., Biros, G.: Distributed-memory large deformation diffeomorphic 3d image registration. SC16: International Conference for High Performance Computing, Networking, Storage and Analysis (2016)
-  Mang, A., Gholami, A., Davatzikos, C., Biros, G.: Claire: A distributed-memory solver for constrained large deformation diffeomorphic image registration. arXiv preprint arXiv:1808.04487 (2018)
-  Oden, J.T., Lima, E.A., Almeida, R.C., Feng, Y., Rylander, M.N., Fuentes, D., Faghihi, D., Rahman, M.M., DeWitt, M., Gadde, M., et al.: Toward predictive multiscale modeling of vascular tumor growth. Archives of Computational Methods in Engineering 23(4), 735–779 (2016)
-  Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. CoRR abs/1505.04597 (2015), http://arxiv.org/abs/1505.04597
-  Shin, H.C., Tenenholtz, N.A., Rogers, J.K., Schwarz, C.G., Senjem, M.L., Gunter, J.L., Andriole, K.P., Michalski, M.: Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In: International Workshop on Simulation and Synthesis in Medical Imaging. pp. 1–11. Springer (2018)
-  Subramanian, S., Gholami, A., Biros, G.: Simulation of glioblastoma growth using a 3d multispecies tumor model with mass effect. (In preparation)
-  Wang, G., Li, W., Ourselin, S., Vercauteren, T.: Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. CoRR abs/1709.00382 (2017), http://arxiv.org/abs/1709.00382
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networkss. In: Computer Vision (ICCV), 2017 IEEE International Conference on (2017)