1 Introduction
Segmentation of medical images is an essential task for many applications such as anatomical structure modeling, tumor growth measurement, surgical planing and treatment assessment (Sharma and Aggarwal, 2010). Despite the breadth and depth of current research, it is very challenging to achieve accurate and reliable segmentation results for many targets (Withey and Koles, 2007). This is often due to poor image quality, inhomogeneous appearances brought by pathology, various imaging protocols and large variations of the segmentation target among patients. Therefore, uncertainty estimation of segmentation results is critical for understanding how reliable the segmentations are. For example, for many images, the segmentation results of pixels near the boundary are likely to be uncertain because of the low contrast between the segmentation target and surrounding tissues, where uncertainty information of the segmentation can be used to indicate potential missegmented regions or guide user interactions for refinement (Prassni et al., 2010; Wang et al., 2018b).
In recent years, deep learning with convolutional neural networks (CNN) has achieved the stateoftheart performance for many medical image segmentation tasks
(Milletari et al., 2016; Abdulkadir et al., 2016; Kamnitsas et al., 2017). Despite their impressive performance and the ability of automatic feature learning, these approaches do not by default provide uncertainty estimation for their segmentation results. In addition, having access to a large training set plays an important role for deep CNNs to achieve humanlevel performance (Esteva et al., 2017; Rajpurkar et al., 2017). However, for medical image segmentation tasks, collecting a very large dataset with pixelwise annotations for training is usually difficult and timeconsuming. As a result, current medical image segmentation methods based on deep CNNs use relatively small datasets compared with those for natural image recognition(Russakovsky et al., 2015). This is likely to introduce more uncertain predictions for the segmentation results, and also leads to uncertainty of downstream analysis, such as volumetric measurement of the target. Therefore, uncertainty estimation is highly desired for deep CNNbased medical image segmentation methods.Several works have investigated uncertainty estimation for deep neural networks (Kendall and Gal, 2017; Lakshminarayanan et al., 2017; Zhu and Zabaras, 2018; Ayhan and Berens, 2018). They focused mainly on image classification or regression tasks, where the prediction outputs are highlevel image labels or bounding box parameters, therefore uncertainty estimation is usually only given for the highlevel predictions. In contrast, pixelwise predictions are involved in segmentation tasks, therefore pixelwise uncertainty estimation is highly desirable. In addition, in most interactive segmentation cases, pixelwise uncertainty information is more helpful for intelligently guiding the user to give interactions. However, previous works have rarely demonstrated uncertainty estimation for deep CNNbased medical image segmentation. As suggested by Kendall and Gal (2017), there are two major types of predictive uncertainties for deep CNNs: epistemic uncertainty and aleatoric uncertainty. Epistemic uncertainty is also known as model uncertainty that can be explained away given enough training data, while aleatoric uncertainty depends on noise or randomness in the input testing image.
In contrast to previous works focusing mainly on classification or regressionrelated uncertainty estimation, and recent works of Nair et al. (2018) and Roy et al. (2018) investigating only testtime dropoutbased (epistemic) uncertainty for segmentation, we extensively investigate different kinds of uncertainties for CNNbased medical image segmentation, including not only epistemic but also aleatoric uncertainties for this task. We also propose a more general estimation of aleatoric uncertainty that is related to not only image noise but also spatial transformations of the input, considering different possible poses of the object during image acquisition. To obtain the transformationrelated uncertainty, we augment the input image at test time, and obtain an estimation of the distribution of the prediction based on testtime augmentation. Testtime augmentation (e.g., rotation, scaling, flipping) has been recently used to improve performance of image classification (Matsunaga et al., 2017) and nodule detection (Jin et al., 2018). Ayhan and Berens (2018) also showed its utility for uncertainty estimation in a fundus image classification task. However, the previous works have not provided a mathematical or theoretical formulation for this. Motivated by these observations, we propose a mathematical formulation for testtime augmentation, and analyze its performance for the general aleatoric
uncertainty estimation in medical image segmentation tasks. In the proposed formulation, we represent an image as a result of an acquisition process which involves geometric transformations and image noise. We model the hidden parameters of the image acquisition process with prior distributions, and predict the distribution of the output segmentation for a test image with a Monte Carlo sampling process. With the samples from the distribution of the predictive output based on the same pretrained CNN, the variance/entropy can be calculated for these samples, which provides an estimation of the
aleatoric uncertainty for the segmentation.The contribution of this work is threefold. First, we propose a theoretical formulation of testtime augmentation for deep learning. Testtime augmentation has not been mathematically formulated by existing works, and our proposed mathematical formulation is general for image recognition tasks. Second, with the proposed formulation of testtime augmentation, we propose a general aleatoric uncertainty estimation for medical image segmentation, where the uncertainty comes from not only image noise but also spatial transformations. Third, we analyze different types of uncertainty estimation for the deep CNNbased segmentation, and validate the superiority of the proposed general aleatoric uncertainty with both 2D and 3D segmentation tasks.
2 Related Works
2.1 Segmentation Uncertainty
Uncertainty estimation has been widely investigated for many existing medical image segmentation tasks. As way of examples, Saad et al. (2010) used shape and appearance prior information to estimate the uncertainty for probabilistic segmentation of medical imaging. Shi et al. (2011) estimated the uncertainty of graph cutbased cardiac image segmentation, which was used to improve the robustness of the system. Sankaran et al. (2015) estimated lumen segmentation uncertainty for realistic patientspecific blood flow modeling. Parisot et al. (2014) used segmentation uncertainty to guide contentdriven adaptive sampling for concurrent brain tumor segmentation and registration. Prassni et al. (2010) visualized the uncertainty of a random walkerbased segmentation to guide volume segmentation of brain Magnetic Resonance Images (MRI) and abdominal Computed Tomography (CT) images. Top et al. (2011)
combined uncertainty estimation with active learning to reduce user time for interactive 3D image segmentation.
2.2 Uncertainty Estimation for Deep CNNs
For deep CNNs, both epistemic and aleatoric uncertainties have been investigated in recent years. For model (epistemic
) uncertainty, exact Bayesian networks offer a mathematically grounded method, but they are hard to implement and computationally expensive. Alternatively, it has been shown that dropout at test time can be cast as a Bayesian approximation to represent model uncertainty
(Gal and Ghahramani, 2016; Li et al., 2017). Zhu and Zabaras (2018)used Stochastic Variational Gradient Descent (SVGD) to perform approximate Bayesian inference on uncertain CNN parameters. A variety of other approximation methods such as Markov chain Monte Carlo (MCMC)
(Neal, 2012), Monte Carlo Batch Normalization (MCBN)
(Teye et al., 2018) and variational Bayesian methods (Graves, 2011; Louizos and Welling, 2016) have also been developed. Lakshminarayanan et al. (2017) proposed ensembles of multiple models for uncertainty estimation, which was simple and scalable to implement. For test imagebased (aleatoric) uncertainty, Kendall and Gal (2017) proposed a unified Bayesian deep learning framework to learn mappings from input data to aleatoric uncertainty and composed them with epistemic uncertainty, where the aleatoric uncertainty was modeled as learned loss attenuation and further categorized into homoscedastic uncertainty and heteroscedastic uncertainty. Ayhan and Berens (2018) used testtime augmentation for aleatoric uncertainty estimation, which was an efficient and effective way to explore the locality of a testing sample. However, its utility for medical image segmentation has not been demonstrated.2.3 TestTime Augmentation
Data augmentation was originally proposed for the training of deep neural networks. It was employed to enlarge a relatively small dataset by applying transformations to its samples to create new ones for training (Krizhevsky et al., 2012). The transformations for augmentation typically include flipping, cropping, rotating, and scaling training images. Abdulkadir et al. (2016) and Ronneberger et al. (2015) also used elastic deformations for biomedical image segmentation. Several studies have empirically found that combining predictions of multiple transformed versions of a test image helps to improve the performance. For example, Matsunaga et al. (2017) geometrically transformed test images for skin lesion classification. Radosavovic et al. (2017) used a single model to predict multiple transformed copies of unlabeled images for data distillation. Jin et al. (2018) tested on samples extended by rotation and translation for pulmonary nodule detection. However, all these works used testtime augmentation as an ad hoc method, without detailed formulation or theoretical explanation, and did not apply it to uncertainty estimation for segmentation tasks.
3 Method
The proposed general aleatoric uncertainty estimation is formulated in a consistent mathematical framework including two parts. The first part is a mathematical representation of ensembles of predictions of multiple transformed versions of the input. We represent an image as a result of an image acquisition model with hidden parameters in Section 3.1. Then we formulate testtime augmentation as inference with hidden parameters following given prior distributions in Section 3.2. The second part calculates the diversity of the prediction results of an augmented test image, and it is used to estimate the aleatoric uncertainty related to image transformations and noise. This is detailed in Section 3.3. Our proposed aleatoric uncertainty is compared and combined with epistemic uncertainty, which is described in Section 3.4. Finally, we apply our proposed method to structurewise uncertainty estimation in Section 3.5.
3.1 Image Acquisition Model
The image acquisition model describes the process by which the observed images have been obtained. This process is confronted with a lot of factors that can be related or unrelated to the imaged object, such as blurring, downsampling, spatial transformation, and system noise. While blurring and downsampling are commonly considered for image superresolution
(Yue et al., 2016), in the context of image recognition they have a relatively lower impact. Therefore, we focus on the spatial transformation and noise, and highlight that adding more complex intensity changes or other forms of data augmentation such as elastic deformations is a straightforward extension. The image acquisition model is:(1) 
where is an underlying image in a certain position and orientation, i.e., a hidden variable. is a transformation operator that is applied to . is the set of parameters of the transformation, and represents the noise that is added to the transformed image. denotes the observed image that is used for inference at test time. Though the transformations can be in spatial, intensity or feature space, in this work we only study the impact of reversible spatial transformations (e.g., flipping, scaling, rotation and translation), which are the most common types of transformations occurring during image acquisition and used for data augmentation purposes. Let denote the inverse transformation of , then we have:
(2) 
Similarly to data augmentation at training time, we assume that the distribution of covers the distribution of . In a given application, this assumption leads to some prior distributions of the transformation parameters and noise. For example, in a 2D slice of fetal brain MRI, the orientation of the fetal brain can span all possible directions in a 2D plane, therefore the rotation angle can be modeled with a uniform prior distribution
. The image noise is commonly modeled as a Gaussian distribution, i.e.,
, where andare the mean and standard deviation respectively. Let
and represent the prior distribution of and respectively, therefore we have and .Let and be the labels related to and respectively. For image classification, and
are categorical variables, and they should be invariant with regard to transformations and noise, therefore
. For image segmentation, and are discretized label maps, and they are equivariant with the spatial transformation, i.e., .3.2 Inference with Hidden Variables
In the context of deep learning, let be the function represented by a neural network, and represent the parameters learned from a set of training images with their corresponding annotations. In a standard formulation, the prediction of a test image is inferred by:
(3) 
For regression problems, refers to continuous values. For segmentation or classification problems, refers to discretized labels obtained by argmax operation in the last layer of the network. Since is only one of many possible observations of the underlying image , direct inference with may lead to a biased result affected by the specific transformation and noise associated with . To address this problem, we aim at inferring it with the help of the latent instead:
(4) 
where the exact values of and for are unknown. Instead of finding a deterministic prediction of , we alternatively consider the distribution of for a robust inference given the distributions of and .
(5) 
For regression problems, we obtain the final prediction for by calculating the expectation of using the distribution .
(6)  
Calculating with Eq. (6) is computationally expensive, as and may take continuous values and
is a complex joint distribution of different types of transformations. Alternatively, we estimate
by using Monte Carlo simulation. Let represent the total number of simulation runs. In the th simulation run, the prediction is:(7) 
where . To obtain , we first randomly sample and from the prior distributions and , respectively. Then we obtain one possible hidden image with and based on Eq. (2), and feed it into the trained network to get its prediction, which is transformed with to obtain according to Eq. (4). With the set sampled from , is estimated as the average of and we use it as the final prediction for :
(8) 
For classification or segmentation problems, is a discretized distribution. We obtain the final prediction for by maximum likelihood estimation:
(9) 
where Mode() is the most frequent element in . This corresponds to majority voting of multiple predictions.
3.3 Aleatoric Uncertainty Estimation with TestTime Augmentation
The uncertainty is estimated by measuring how diverse the predictions for a given image are. Both the variance and entropy of the distribution can be used to estimate uncertainty. However, variance is not sufficiently representative in the context of multimodal distributions. In this paper we use entropy for uncertainty estimation:
(10) 
With the Monte Carlo simulation in Section 3.2, we can approximate from the simulation results . Suppose there are unique values in . For classification tasks, this typically refers to labels. Assume the frequency of the th unique value is , then is approximated as:
(11) 
For segmentation tasks, pixelwise uncertainty estimation is desirable. Let denote the predicted label for the th pixel. With the Monte Carlo simulation, a set of values for are obtained . The entropy of the distribution of is therefore approximated as:
(12) 
where is the frequency of the th unique value in .
3.4 Epistemic Uncertainty Estimation
To obtain model (epistemic) uncertainty estimation, we follow the testtime dropout method proposed by Gal and Ghahramani (2016). In this method, let be an approximating distribution over the set of network parameters
with its elements randomly set to zero according to Bernoulli random variables.
can be achieved by minimizing the KullbackLeibler divergence between
and the posterior distribution of given a training set. After training, the predictive distribution of a test image can be expressed as:(13) 
The distribution of the prediction can be sampled based on Monte Carlo samples of the trained network (i.e, MC dropout): where is a Monte Carlo sample from . Assume the number of samples is , and the sampled set of the distribution of is . The final prediction for can be estimated by Eq. (8) for regression problems or Eq. (9) for classification/segmentation problems. The epistemic uncertainty estimation can therefore be calculated based on variance or entropy of the sampled predictions. To keep consistent with our aleatoric uncertainty, we use entropy for this purpose, which is similar to Eq. (12). Testtime dropout may be interpreted as a way of ensembles of networks for testing. In the work of Lakshminarayanan et al. (2017), ensembles of neural networks was explicitly proposed as an alternative solution of testtime dropout for estimating epistemic uncertainty.
3.5 Structurewise Uncertainty Estimation
Nair et al. (2018) and Roy et al. (2018) used Monte Carlo samples generated by testtime dropout for structure/lesionwise uncertainty estimation. Following these works, we extend the structurewise uncertainty estimation method by using Monte Carlo samples generated by not only testtime dropout, but also testtime augmentation described in 3.2. For samples from the Monte Carlo simulation, let denote the set of volumes of the segmented structure, where is the volume of the segmented structure in the th simulation. Let and denote the mean value and standard deviation of respectively. We use the volume variation coefficient (VVC) to estimate the structurewise uncertainty:
(14) 
where VVC is agnostic to the size of the segmented structure.
4 Experiments and Results
We validated our proposed testing and uncertainty estimation method with two segmentation tasks: 2D fetal brain segmentation from MRI slices and 3D brain tumor segmentation from multimodal MRI volumes. The implementation details for 2D and 3D segmentation are described in Section 4.1 and Section 4.2 respectively.
In both tasks, we compared different types of uncertainties for the segmentation results: 1) the proposed aleatoric uncertainty based on our formulated testtime augmentation (TTA), 2) the epistemic uncertainty based on testtime dropout (TTD) described in Section 3.4, and 3) hybrid uncertainty that combines the aleatoric and epistemic uncertainties based on TTA + TTD. For each of these three methods, the uncertainty was obtained by Eq. (12) with
predictions. For TTD and TTA + TTD, the dropout probability was set as a typical value of 0.5
(Gal and Ghahramani, 2016).We also evaluated the segmentation accuracy of these different prediction methods: TTA, TTD, TTA + TTD and the baseline that uses a single prediction without TTA and TTD. For a given training set, all these methods used the same model that was trained with data augmentation and dropout at training time. The augmentation during training followed the same formulation in Section 3.1. We investigated the relationship between each type of uncertainty and segmentation error in order to know which uncertainty has a better ability to indicate potential missegmentations. Quantitative evaluations of segmentation accuracy are based on Dice score and Average Symmetric Surface Distance (ASSD).
(15) 
where , and are true positive, false positive and false negative respectively. The definition of ASSD is:
(16) 
where and denote the set of surface points of a segmentation result and the ground truth respectively. is the shortest Euclidean distance between a point and all the points in .
4.1 2D Fetal Brain Segmentation from MRI
Fetal MRI has been increasingly used for study of the developing fetus as it provides a better soft tissue contrast than the widely used prenatal sonography. The most commonly used imaging protocol for fetal MRI is SingleShot Fast Spin Echo (SSFSE) that acquires images in a fast speed and mitigates the effect of fetal motion, leading to stacks of thick 2D slices. Segmentation is a fundamental step for fetal brain study, e.g., it plays an important role in interslice motion correction and highresolution volume reconstruction (Tourbier et al., 2017; Ebner et al., 2018). Recently, CNNs have achieved the stateoftheart performance for 2D fetal brain segmentation (Rajchl et al., 2016; Salehi et al., 2017, 2018). In this experiment, we segment the 2D fetal brain using deep CNNs with uncertainty estimation.
4.1.1 Data and Implementation
We collected clinical T2weighted MRI scans of 60 fetuses in the second trimester with SSFSE on a 1.5 Tesla MR system (Aera, Siemens, Erlangen, Germany). The data for each fetus contained three stacks of 2D slices acquired in axial, sagittal and coronal views respectively, with pixel size 0.63 mm to 1.58 mm and slice thickness 3 mm to 6 mm. The gestational age ranged from 19 weeks to 33 weeks. We used 2640 slices from 120 stacks of 40 patients for training, 278 slices from 12 stacks of 4 patients for validation and 1180 slices from 48 stacks of 16 patients for testing. Two radiologists manually segmented the brain region for all the stacks slicebyslice, where one radiologist gave a segmentation first, and then the second senior radiologist refined the segmentation if disagreement existed, the output of which were used as the ground truth. We used this dataset for two reasons. First, our dataset fits with a typical medical image segmentation application where the number of annotated images is limited. This leads the uncertainty information to be of high interest for robust prediction and our downstream tasks such as fetal brain reconstruction and volume measurement. Second, the position and orientation of fetal brain have large variations, which is suitable for investigating the effect of data augmentation. For preprocessing, we normalized each stack by its intensity mean and standard deviation, and resampled each slice with pixel size 1.0 mm.
We used 2D networks of Fully Convolutional Network (FCN) (Long et al., 2015), UNet(Ronneberger et al., 2015) and PNet(Wang et al., 2018b)
. The networks were implemented in TensorFlow
^{1}^{1}1https://www.tensorflow.org(Abadi et al., 2016) using NiftyNet^{2}^{2}2http://www.niftynet.io(Li et al., 2017; Gibson et al., 2018). During training, we used Adaptive Moment Estimation (Adam) to adjust the learning rate that was initialized as
, with batch size 5, weight decay and iteration number . We represented the transformation parameter in the proposed augmentation framework as a combination of , and , where is a random variable for flipping along each 2D axis, is the rotation angle in 2D, and is a scaling factor. The prior distributions of these transformation parameters and random intensity noise were modeled as , , and . The hyperparameters for our fetal brain segmentation task were set as , , , and . For the random noise, we set and , as a medianfilter smoothed version of a normalized image in our dataset has a standard deviation around 0.95. We augmented the training data with this formulation, and during test time, TTA used the same prior distributions of augmentation parameters as used for training.4.1.2 Segmentation Results with Uncertainty
Fig. 1 shows a visual comparison of different types of uncertainties for segmentation of three fetal brain images in coronal, sagittal and axial view respectively. The results were based on the same trained model of UNet with traintime augmentation, and the Monte Carlo simulation number was 20 for TTD, TTA, and TTA + TTD to obtain epistemic, aleatoric and hybrid uncertainties respectively. In each subfigure, the first row presents the input and the segmentation obtained by the singleprediction baseline. The other rows show these three types of uncertainties and their corresponding segmentation results respectively. The uncertainty maps in odd columns are represented by pixelwise entropy of predictions and encoded by the color bar in the left top corner. In the uncertainty maps, purple pixels have low uncertainty values and yellow pixels have high uncertainty values. Fig. 1(a) shows a fetal brain in coronal view. In this case, the baseline prediction method achieved a good segmentation result. It can be observed that for epistemic uncertainty calculated by TTD, most of the uncertain segmentations are located near the border of the segmented foreground, while the pixels with a larger distance to the border have a very high confidence (i.e., low uncertainty). In addition, the epistemic uncertainty map contains some random noise in the brain region. In contrast, the aleatoric uncertainty obtained by TTA contains less random noise and it shows uncertain segmentations not only on the border but also in some challenging areas in the lower right corner, as highlighted by the white arrows. In that region, the result obtained by TTA has an oversegmentation, and this is corresponding to the high values in the same region of the aleatoric uncertainty map. The hybrid uncertainty calculated by TTA + TTD is a mixture of epistemic and aleatoric uncertainty. As shown in the last row of Fig. 1(a), it looks similar to the aleatoric uncertainty map except for some random noise.
Fig. 1(b) and Fig. 1(c) show two other cases where the singleprediction baseline obtained an oversegmentation and an undersegmentation respectively. It can be observed that the epistemic uncertainty map shows a high confidence (low uncertainty) in these missegmented regions. This leads to a lot of overconfident incorrect segmentations, as highlighted by the white arrows in Fig. 1(b) and (c). In comparison, the aleatoric uncertainty map obtained by TTA shows a larger uncertain area that is mainly corresponding to missegmented regions of the baseline. In these two cases, The hybrid uncertainty also looks similar to the aleatoric uncertainty map. The comparison indicates that the aleatoric uncertainty has a better ability than the epistemic uncertainty to indicate missegmentations of nonborder pixels. For these pixels, the segmentation output is more affected by different transformations of the input (aleatoric) rather than variations of model parameters (epistemic).
Fig. 1(b) and (c) also show that TTD using different model parameters seemed to obtain very little improvement from the baseline. In comparison, TTA using different input transformations corrected the large missegmentations and achieved a more noticeable improvement from the baseline. It can also be observed that the results obtained by TTA + TTD are very similar to those obtained by TTA, which shows TTA is more suitable to improving the segmentation than TTD.
4.1.3 Quantitative Evaluation
Train  Test  Dice (%)  ASSD (mm)  

FCN  UNet  PNet  FCN  UNet  PNet  
TrAug  Baseline  91.053.82  90.264.77  90.654.29  2.682.93  3.113.34  2.833.07 
TTD  91.133.60  90.384.30  90.934.04  2.612.85  3.042.29  2.692.90  
TTA  91.993.48*  91.644.11*  92.023.85*  2.262.56*  2.513.23*  2.282.61*  
TTA + TTD  92.053.58*  91.883.61*  92.173.68*  2.192.67*  2.402.71*  2.132.42*  
Tr+Aug  Baseline  92.033.44  91.933.21  91.983.92  2.212.52  2.122.23  2.322.71 
TTD  92.083.41  92.003.22  92.013.89  2.172.52  2.032.13  2.152.58  
TTA  92.793.34*  92.883.15*  93.052.96*  1.882.08  1.701.75  1.621.77*  
TTA + TTD  92.853.15*  92.903.16*  93.142.93*  1.841.92  1.671.76*  1.481.63* 
To quantitatively evaluate the segmentation results, we measured Dice score and ASSD of predictions by different testing methods with three network structures: FCN (Long et al., 2015), UNet (Ronneberger et al., 2015) and PNet (Wang et al., 2018b). For all of these CNNs, we used data augmentation at training time to enlarge the training set. At inference time, we compared the baseline testing method (without Monte Carlo simulation) with TTD, TTA and TTA + TTD. We first investigated how the segmentation accuracy changes with the increase of the number of Monte Carlo simulation runs . The results measured with all the testing images are shown in Fig. 2. We found that for all of these three networks, the segmentation accuracy of TTD remains close to that of the singleprediction baseline. For TTA and TTA + TTD, an improvement of segmentation accuracy can be observed when increases from 1 to 10. When is larger than 20, the segmentation accuracy for these two methods reaches a plateau.
In addition to the previous scenario using augmentation at both training and test time, we also evaluated the performance of TTD and TTA when data augmentation was not used for training. The quantitative evaluations of combinations of different training methods and testing methods (20) are shown in Table 1. It can be observed that for both training with and without data augmentation, TTA has a better ability to improve the segmentation accuracy than TTD. Combining TTA and TTD can further improve the segmentation accuracy, but it does not significantly outperform TTA (value 0.05).
Fig. 3 shows Dice distributions of five example stacks of fetal brain MRI. The results were based on the same trained model of UNet with traintime augmentation. Note that the baseline had only one prediction for each image, and the Monte Carlo simulation number was 20 for TTD, TTA and TTA + TTD. It can be observed that for each case, the Dice of TTD is distributed closely around that of the baseline. In comparison, the Dice distribution of TTA has a higher average than that of TTD, indicating TTA’s better ability of improving segmentation accuracy. The results of TTA also have a larger variance than that of TTD, which shows TTA can provide more structurewise uncertainty information. Fig. 3 also shows that the performance of TTA + TTD is close to that of TTA.
4.1.4 Correlation between Uncertainty and Segmentation Error
To investigate how our uncertainty estimation methods can indicate incorrect segmentation, we measured the uncertainty and segmentation error at both pixellevel and structurelevel. For pixellevel evaluation, we measured the joint histogram of pixelwise uncertainty and error rate for TTD, TTA, and TTA + TTD respectively. The histogram was obtained by statistically calculating the error rate of pixels at different pixelwise uncertainty levels in each slice. The results based on UNet with are shown in Fig. 4, where the joint histograms have been normalized by the number of total pixels in the testing images for visualization. For each type of pixelwise uncertainty, we calculated the average error rate at each pixelwise uncertainty level, leading to a curve of error rate as a function of pixelwise uncertainty, i.e., the red curves in Fig. 4. This figure shows that the majority of pixels have a low uncertainty with a small error rate. When the uncertainty increases, the error rate also becomes higher gradually. Fig. 4(a) shows the TTDbased uncertainty (epistemic). It can be observed that when the prediction uncertainty is low, the result has a steep increase of error rate. In contrast, for the TTAbased uncertainty (aleatoric), the increase of error rate is slower, shown in Fig. 4(b). This demonstrates that TTA has fewer overconfident incorrect predictions than TTD. The dashed ellipses in Fig. 4 also show the different levels of overconfident incorrect predictions for different testing methods.
For structurewise evaluation, we used VVC to represent structurewise uncertainty and Dice to represent structurewise segmentation error. Fig. 5 shows the joint distribution of VVC and Dice for different testing methods using UNet trained with data augmentation and = 20 for inference. The results of TTD, TTA, and TTA + TTD are shown in Fig. 5(a), (b) and (c) respectively. It can be observed that for all the three testing methods, the VVC value tends to become larger when Dice grows. However, the slope in Fig. 5(a) is smaller than those in Fig. 5(b) and Fig. 5(c). The comparison shows that TTAbased structurewise uncertainty estimation is highly related to segmentation error, and TTA leads to a larger scale of VVC than TTD. Combining TTA and TTD leads to similar results to that of TTA.
4.2 3D Brain Tumor Segmentation from MultiModal MRI
MRI has become the most commonly used imaging methods for brain tumors. Different MR sequences such as T1weighted (T1w), contrast enhanced T1weighted (T1wce), T2weighted (T2w) and Fluid Attenuation Inversion Recovery (FLAIR) images can provide complementary information for analyzing multiple subregions of brain tumors. Automatic brain tumor segmentation from multimodal MRI has a potential for better diagnosis, surgical planning and treatment assessment (Menze et al., 2015). Deep neural networks have achieved the stateoftheart performance on this task (Kamnitsas et al., 2017; Wang et al., 2018a). In this experiment, we analyze the uncertainty of deep CNNbased brain tumor segmentation and show the effect of our proposed testtime augmentation.
4.2.1 Data and Implementation
We used the BraTS 2017^{3}^{3}3http://www.med.upenn.edu/sbia/brats2017.html (Bakas et al., 2017) training dataset that consisted of volumetric images from 285 studies, with ground truth provided by the organizers. We randomly selected 20 studies for validation and 50 studies for testing, and used the remaining for training. For each study, there were four scans of T1w, T1wce, T2w and FLAIR images, and they had been coregistered. All the images were skullstripped and resampled to an isotropic 1 mm resolution. As a first demonstration of uncertainty estimation for deep learningbased brain tumor segmentation, we investigate segmentation of the whole tumor from these multimodal images (Fig. 6). We used 3D UNet (Abdulkadir et al., 2016), VNet (Milletari et al., 2016) and WNet (Wang et al., 2018a) implemented with NiftyNet (Gibson et al., 2018), and employed Adam during training with initial learning rate , batch size 2, weight decay and iteration number . WNet is a 2.5D network, and we compared using WNet only in axial view and a fusion of axial, sagittal and coronal views. These two implementations are referred to as WNet(A) and WNet(ASC) respectively. The transformation parameter in the proposed augmentation framework consisted of , , and , where is a random variable for flipping along each 3D axis, is the rotation angle along each 3D axis, is a scaling factor and is intensity noise. The prior distributions were: , , and according to the reduced standard deviation of a medianfiltered version of a normalized image. We used this formulated augmentation during training, and also employed it to obtain TTAbased results at test time.
4.2.2 Segmentation Results with Uncertainty
Fig. 6 demonstrates three examples of uncertainty estimation of brain tumor segmentation by different testing methods. The results were based on the same trained model of 3D UNet (Abdulkadir et al., 2016). The Monte Carlo simulation number was 40 for TTD, TTA, and TTA + TTD to obtain epistemic, aleatoric and hybrid uncertainties respectively. Fig. 6(a) shows a case of high grade glioma (HGG). The baseline of single prediction obtained an oversegmentation at the upper part of the image. The epistemic uncertainty obtained by TTD highlights some uncertain predictions at the border of the segmentation and a small part of the oversegmented region. In contrast, the aleatoric uncertainty obtained by TTA better highlights the whole oversegmented region, and the hybrid uncertainty map obtained by TTA + TTD is similar to the aleatoric uncertainty map. The second column of Fig. 6(a) shows the corresponding segmentations of these uncertainties. It can be observed that the TTDbased result looks similar to the baseline, while TTA and TTA + TTD based results achieve a larger improvement from the baseline. Fig. 6(b) demonstrates another case of HGG brain tumor, and it shows that the oversegmented region in the baseline prediction is better highlighted by TTAbased aleatoric uncertainty than TTDbased epistemic uncertainty. Fig. 6(c) shows a case of low grade glioma (LGG). The baseline of single prediction obtained an undersegmentation in the middle part of the tumor. The epistemic uncertainty obtained by TTD only highlights pixels on the border of the prediction, with a low uncertainty (high confidence) for the undersegmented region. In contrast, the aleatoric uncertainty obtained by TTA has a better ability to indicate the undersegmentation. The results also show that TTA outperforms TTD for better segmentation.
4.2.3 Quantitative Evaluation
For quantitative evaluations, we calculated the Dice score and ASSDe for the segmentation results obtained by the different testing methods that were combined with 3D UNet (Abdulkadir et al., 2016), VNet (Milletari et al., 2016) and WNet (Wang et al., 2018a) respectively. We also compared TTD and TTA with and without traintime data augmentation, respectively. We found that for these networks, the performance of the multiprediction testing methods reaches a plateau when is larger than 40. Table 2 shows the evaluation results with = 40. It can be observed that for each network and each training method, multiprediction methods lead to better performance than the baseline with a single prediction, and TTA outperforms TTD with higher Dice scores and lower ASSD values. Combining TTA and TTD has a slight improvement from using TTA, but the improvement is not significant (value 0.05).
Train  Test  Dice (%)  ASSD (mm)  

WNet (ASC)  3D UNet  VNet  WNet (ASC)  3D UNet  VNet  
TrAug  Baseline  87.817.27  87.267.73  86.848.38  2.041.27  2.621.48  2.861.79 
TTD  88.147.02  87.557.33  87.138.14  1.951.20  2.551.41  2.821.75  
TTA  89.166.48*  88.586.50*  87.866.97*  1.420.93*  1.791.16*  1.971.40*  
TTA + TTD  89.436.14*  88.756.34*  88.036.56*  1.370.89*  1.721.23*  1.951.31*  
Tr+Aug  Baseline  88.765.76  88.436.67  87.447.84  1.611.12  1.821.17  2.071.46 
TTD  88.925.73  88.526.66  87.567.78  1.571.06  1.761.14  1.991.33  
TTA  90.075.69*  89.416.05*  88.386.74*  1.130.54*  1.450.81  1.670.98*  
TTA + TTD  90.355.64*  89.605.95*  88.576.32*  1.100.49*  1.390.76*  1.620.95* 
4.2.4 Correlation between Uncertainty and Segmentation Error
To study the relationship between prediction uncertainty and segmentation error at voxellevel, we measured voxelwise uncertainty and voxelwise error rate at different uncertainty levels. For each of TTDbased (epistemic), TTAbased (aleatoric) and TTA + TTDbased (hybrid) voxelwise uncertainty, we obtained the normalized joint histogram of voxelwise uncertainty and voxelwise error rate. Fig. 7 shows the results based on 3D UNet trained with data augmentation and using for inference. The red curve shows the average voxelwise error rate as a function of voxelwise uncertainty. In Fig. 7(a), the average prediction error rate has a slight change when the TTDbased epistemic uncertainty is larger than 0.2. In contrast, Fig. 7(b) and (c) show that the average prediction error rate has a smoother increase with the growth of aleatoric and hybrid uncertainties. The comparison demonstrates that the TTAbased aleatoric uncertainty leads to fewer overconfident missegmentations than the TTDbased epistemic uncertainty.
For structurelevel evaluation, we also studied the relationship between structurelevel uncertainty represented by VVC and structurelevel error represented by Dice. Fig. 8 shows their joint distributions with three different testing methods using 3D UNet. The network was trained with data augmentation, and was set as 40 for inference. Fig. 8 shows that TTAbased VVC increases when Dice grows, and the slope is larger than that of TTDbased VVC. The results of TTA and TTA + TTD are similar, as shown in Fig. 8(b) and (c). The comparison shows that TTAbased structurewise uncertainty can better indicate segmentation error than TTDbased structurewise uncertainty.
5 Discussion and Conclusion
In our experiments, the number of training images was relatively small compared with many datasets of natural images such as PASCAL VOC, COCO and ImageNet. For medical images, it is typically very difficult to collect a very large dataset for segmentation, as pixelwise annotations are not only timeconsuming to collect but also require expertise of radiologists. Therefore, for most existing medical image segmentation datasets, such as those in Grand challenge
^{4}^{4}4 https://grandchallenge.org/challenges, the image numbers are also quite small. Therefore, investigating the segmentation performance of CNNs with limited training data is of high interest for medical image computing community. In addition, our dataset is not very large so that it is suitable for data augmentation, which fits well with our motivation of using data augmentation at training and test time. The need for uncertainty estimation is also stronger in cases where datasets are smaller.In our mathematical formulation of testtime augmentation based on an image acquisition model, we explicitly modeled spatial transformations and image noise. However, it can be easily extended to include more general transformations such as elastic deformations (Abdulkadir et al., 2016) or add a simulated bias field for MRI. In addition to the variation of possible values of model parameters, the prediction result is also dependent on the input data, e.g., image noise and transformations related to the object. Therefore, a good uncertainty estimation should take these factors into consideration. Fig. 1 and 6 show that model uncertainty alone is likely to obtain overconfident incorrect predictions, and TTA plays an important role in reducing such predictions. In Fig. 3 we show five example cases, where each subfigure shows the results for one patient. Table 1 shows the statistical results based on all the testing images. We found that for few testing images TTA +TTD failed to obtain higher Dice scores than TTA, but for the overall testing images, the average Dice of TTA + TTD is slightly larger than that of TTA. Therefore, this leads to the conclusion that TTA + TTD does not always perform better than TTA, and the average performance of TTA + TTD is close to that of TTA, which is also demonstrated in Fig. 1 and 6.
We have demonstrated TTA based on the image acquisition model for image segmentation tasks, but it is general for different image recognition tasks, such as image classification, object detection, and regression. For regression tasks where the outputs are not discretized category labels, the variation of the output distribution might be more suitable than entropy for uncertainty estimation. Table 2 shows the superiority of testtime augmentation for better segmentation accuracy, and it also demonstrates the combination of WNet in different views helps to improve the performance. This is an ensemble of three networks, and such an ensemble may be used as an alternative for epistemic uncertainty estimation, as demonstrated by Lakshminarayanan et al. (2017).
We found that for our tested CNNs and applications, the proper value of Monte Carlo sample that leads the segmentation accuracy to a plateau was around 20 to 40. Using an empirical value is large enough for our datasets. However, the optimal setting of hyperparameter may change for different datasets. Fixing for new applications where the optimal value of is smaller would lead to unnecessary computation and reduce efficiency. In some applications where the object has more spatial variations, the optimal value may be larger than 40. Therefore, in a new application, we suggest that the optimal should be determined by the performance plateau on the validation set.
In conclusion, we analyzed different types of uncertainties for CNNbased medical image segmentation by comparing and combining model (epistemic) and inputbased (aleatoric) uncertainties. We formulated a testtime augmentationbased aleatoric uncertainty estimation for medical images that considers the effect of both image noise and spatial transformations. We also proposed a theoretical and mathematical formulation of testtime augmentation, where we obtain a distribution of the prediction by using Monte Carlo simulation and modeling prior distributions of parameters in an image acquisition model. Experiments with 2D and 3D medical image segmentation tasks showed that uncertainty estimation with our formulated TTA helps to reduce overconfident incorrect predictions encountered by modelbased uncertainty estimation and TTA leads to higher segmentation accuracy than a singleprediction baseline and multiple predictions using testtime dropout.
6 Acknowledgements
This work was supported by the Wellcome/EPSRC Centre for Medical Engineering [WT 203148/Z/16/Z], an Innovative Engineering for Health award by the Wellcome Trust (WT101957); Engineering and Physical Sciences Research Council (EPSRC) (NS/A000027/1, EP/H046410/1, EP/J020990/1, EP/K005278), Wellcome/EPSRC [203145Z/16/Z], the National Institute for Health Research University College London Hospitals Biomedical Research Centre (NIHR BRC UCLH/UCL), the Royal Society [RG160569], and hardware donated by NVIDIA.
References

Abadi et al. (2016)
Abadi, M., Barham, P.,
Chen, J., Chen, Z.,
Davis, A., Dean, J.,
Devin, M., Ghemawat, S.,
Irving, G., Isard, M.,
Kudlur, M., Levenberg, J.,
Monga, R., Moore, S.,
Murray, D.G., Steiner, B.,
Tucker, P., Vasudevan, V.,
Warden, P., Wicke, M.,
Yu, Y., Zheng, X.,
Brain, G., 2016.
TensorFlow: A system for largescale machine learning, in: USENIX Symposium on Operating Systems Design and Implementation, pp. 265–284.
 Abdulkadir et al. (2016) Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O., 2016. 3D UNet: Learning dense volumetric segmentation from sparse annotation, in: International Conference on Medical Image Computing and ComputerAssisted Intervention, pp. 424–432.
 Ayhan and Berens (2018) Ayhan, M.S., Berens, P., 2018. Testtime data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks, in: Medical Imaging with Deep Learning, pp. 1–9.
 Bakas et al. (2017) Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., Davatzikos, C., 2017. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Nature Scientific Data , 170117.
 Ebner et al. (2018) Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P.A., Aughwane, R., Melbourne, A., Doel, T., David, A.L., Deprest, J., Ourselin, S., Vercauteren, T., 2018. An automated localization, segmentation and reconstruction framework for fetal brain MRI, in: International Conference on Medical Image Computing and ComputerAssisted Intervention, pp. 313–320.
 Esteva et al. (2017) Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S., 2017. Dermatologistlevel classification of skin cancer with deep neural networks. Nature 542, 115–118.
 Gal and Ghahramani (2016) Gal, Y., Ghahramani, Z., 2016. Dropout as a Bayesian approximation: representing model uncertainty in deep learning, in: International Conference on Machine Learning, pp. 1050–1059.
 Gibson et al. (2018) Gibson, E., Li, W., Sudre, C., Fidon, L., Shakir, D.I., Wang, G., EatonRosen, Z., Gray, R., Doel, T., Hu, Y., Whyntie, T., Nachev, P., Modat, M., Barratt, D.C., Ourselin, S., Cardoso, M.J., Vercauteren, T., 2018. NiftyNet: A deeplearning platform for medical imaging. Computer Methods and Programs in Biomedicine 158, 113–122.
 Graves (2011) Graves, A., 2011. Practical variational inference for neural networks, in: Advances in Neural Information Processing Systems, pp. 1–9.
 Jin et al. (2018) Jin, H., Li, Z., Tong, R., Lin, L., 2018. A deep 3D residual CNN for false positive reduction in pulmonary nodule detection. Medical Physics 45, 2097–2107.
 Kamnitsas et al. (2017) Kamnitsas, K., Ledig, C., Newcombe, V.F.J., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B., 2017. Efficient multiscale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36, 61–78.

Kendall and Gal (2017)
Kendall, A., Gal, Y., 2017.
What uncertainties do we need in Bayesian deep learning for computer vision?, in: Advances in Neural Information Processing Systems, pp. 5580–5590.
 Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. ImageNet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, pp. 1097–1105.
 Lakshminarayanan et al. (2017) Lakshminarayanan, B., Pritzel, A., Blundell, C., 2017. Simple and scalable predictive uncertainty estimation using deep ensembles, in: Advances in Neural Information Processing Systems, pp. 6405–6416.
 Li et al. (2017) Li, W., Wang, G., Fidon, L., Ourselin, S., Cardoso, M.J., Vercauteren, T., 2017. On the compactness, efficiency, and representation of 3D convolutional networks: brain parcellation as a pretext task, in: International Conference on Information Processing in Medical Imaging, pp. 348–360.

Long et al. (2015)
Long, J., Shelhamer, E.,
Darrell, T., 2015.
Fully convolutional networks for semantic segmentation, in: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440.
 Louizos and Welling (2016) Louizos, C., Welling, M., 2016. Structured and efficient variational eeep learning with matrix gaussian posteriors, in: International Conference on Machine Learning, pp. 1708–1716.
 Matsunaga et al. (2017) Matsunaga, K., Hamada, A., Minagawa, A., Koga, H., 2017. Image classification of melanoma, nevus and seborrheic keratosis by deep neural network ensemble. arXiv preprint arXiv:1703.03108 .
 Menze et al. (2015) Menze, B.H., Jakab, A., Bauer, S., KalpathyCramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M.A., Arbel, T., Avants, B.B., Ayache, N., Buendia, P., Collins, D.L., Cordier, N., Corso, J.J., Criminisi, A., Das, T., Delingette, H., Demiralp, Ç., Durst, C.R., Dojat, M., Doyle, S., Festa, J., Forbes, F., Geremia, E., Glocker, B., Golland, P., Guo, X., Hamamci, A., Iftekharuddin, K.M., Jena, R., John, N.M., Konukoglu, E., Lashkari, D., Mariz, J.A., Meier, R., Pereira, S., Precup, D., Price, S.J., Raviv, T.R., Reza, S.M., Ryan, M., Sarikaya, D., Schwartz, L., Shin, H.C., Shotton, J., Silva, C.A., Sousa, N., Subbanna, N.K., Szekely, G., Taylor, T.J., Thomas, O.M., Tustison, N.J., Unal, G., Vasseur, F., Wintermark, M., Ye, D.H., Zhao, L., Zhao, B., Zikic, D., Prastawa, M., Reyes, M., Van Leemput, K., 2015. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging 34, 1993–2024.
 Milletari et al. (2016) Milletari, F., Navab, N., Ahmadi, S.A., 2016. VNet: Fully convolutional neural networks for volumetric medical image segmentation, in: International Conference on 3D Vision, pp. 565–571.
 Nair et al. (2018) Nair, T., Precup, D., Arnold, D.L., Arbel, T., 2018. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation, in: International Conference on Medical Image Computing and ComputerAssisted Intervention, pp. 655–663.
 Neal (2012) Neal, R.M., 2012. Bayesian learning for neural networks. Springer Science & Business Media.
 Parisot et al. (2014) Parisot, S., Wells, W., Chemouny, S., Duffau, H., Paragios, N., 2014. Concurrent tumor segmentation and registration with uncertaintybased sparse nonuniform graphs. Medical Image Analysis 18, 647–659.
 Prassni et al. (2010) Prassni, J.S., Ropinski, T., Hinrichs, K., 2010. Uncertaintyaware guided volume segmentation. IEEE Transactions on Visualization and Computer Graphics 16, 1358–1365.
 Radosavovic et al. (2017) Radosavovic, I., Dollár, P., Girshick, R., Gkioxari, G., He, K., 2017. Data distillation: towards omnisupervised learning. arXiv preprint arXiv:1712.04440 .
 Rajchl et al. (2016) Rajchl, M., Lee, M.C.H., Schrans, F., Davidson, A., PasseratPalmbach, J., Tarroni, G., Alansary, A., Oktay, O., Kainz, B., Rueckert, D., 2016. Learning under distributed weak supervision. arXiv preprint arXiv:1606.01100 .
 Rajpurkar et al. (2017) Rajpurkar, P., Irvin, J., Bagul, A., Ding, D., Duan, T., Mehta, H., Yang, B., Zhu, K., Laird, D., Ball, R.L., Langlotz, C., Shpanskaya, K., Lungren, M.P., Ng, A., 2017. MURA dataset: towards radiologistlevel abnormality eetection in musculoskeletal radiographs. arXiv preprint arXiv:1712.06957 .
 Ronneberger et al. (2015) Ronneberger, O., Fischer, P., Brox, T., 2015. UNet: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and ComputerAssisted Intervention, pp. 234–241.
 Roy et al. (2018) Roy, A.G., Conjeti, S., Navab, N., Wachinger, C., 2018. Inherent brain segmentation quality control from fully convnet Monte Carlo sampling, in: International Conference on Medical Image Computing and ComputerAssisted Intervention, pp. 664–672.
 Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., FeiFei, L., 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115, 211–252.
 Saad et al. (2010) Saad, A., Hamarneh, G., Möller, T., 2010. Exploration and visualization of segmentation uncertainty using shape and appearance prior information. IEEE Transactions on Visualization and Computer Graphics 16, 1366–1375.
 Salehi et al. (2017) Salehi, S.S.M., Erdogmus, D., Gholipour, A., 2017. Autocontext convolutional neural network (autonet) for brain extraction in magnetic resonance imaging. IEEE Transactions on Medical Imaging 36, 2319 – 2330.
 Salehi et al. (2018) Salehi, S.S.M., Hashemi, S.R., VelascoAnnis, C., Ouaalam, A., Estroff, J.A., Erdogmus, D., Warfield, S.K., Gholipour, A., 2018. Realtime automatic fetal brain extraction in fetal MRI by deep learning, in: IEEE International Symposium on Biomedical Imaging, pp. 720–724.
 Sankaran et al. (2015) Sankaran, S., Grady, L., Taylor, C.A., 2015. Fast computation of hemodynamic sensitivity to lumen segmentation uncertainty. IEEE Transactions on Medical Imaging 34, 2562–2571.
 Sharma and Aggarwal (2010) Sharma, N., Aggarwal, L.M., 2010. Automated medical image segmentation techniques. Journal of Medical Physics 35, 3–14.
 Shi et al. (2011) Shi, W., Zhuang, X., Wolz, R., Simon, D., Tung, K., Wang, H., Ourselin, S., Edwards, P., Razavi, R., Rueckert, D., 2011. A multiimage graph cut approach for cardiac image segmentation and uncertainty estimation, in: International Workshop on Statistical Atlases and Computational Models of the Heart. Springer Berlin Heidelberg. volume 7085, pp. 178–187.
 Teye et al. (2018) Teye, M., Azizpour, H., Smith, K., 2018. Bayesian uncertainty estimation for batch normalized deep detworks. arXiv preprint arXiv:1802.06455 .
 Top et al. (2011) Top, A., Hamarneh, G., Abugharbieh, R., 2011. Active learning for interactive 3D image segmentation, in: International Conference on Medical Image Computing and ComputerAssisted Intervention, pp. 603–610.
 Tourbier et al. (2017) Tourbier, S., VelascoAnnis, C., Taimouri, V., Hagmann, P., Meuli, R., Warfield, S.K., Bach Cuadra, M., Gholipour, A., 2017. Automated templatebased brain localization and extraction for fetal brain MRI reconstruction. NeuroImage 155, 460–472.
 Wang et al. (2018a) Wang, G., Li, W., Ourselin, S., Vercauteren, T., 2018a. Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks, in: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Springer International Publishing, pp. 178–190.
 Wang et al. (2018b) Wang, G., Li, W., Zuluaga, M.A., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., Vercauteren, T., 2018b. Interactive medical image segmentation using deep learning with imagespecific finetuning. IEEE Transactions on Medical Imaging 37, 1562–1573.
 Withey and Koles (2007) Withey, D., Koles, Z., 2007. Medical image segmentation: methods and software, in: Noninvasive Functional Source Imaging of the Brain and Heart and the International Conference on Functional Biomedical Imaging, pp. 140–143.
 Yue et al. (2016) Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L., 2016. Image superresolution: the techniques, applications, and future. Signal Processing 128, 389–408.
 Zhu and Zabaras (2018) Zhu, Y., Zabaras, N., 2018. Bayesian deep convolutional encoderdecoder networks for surrogate modeling and uncertainty quantification. arXiv preprint arXiv:1801.06879 .
Comments
There are no comments yet.