1 Introduction
Convolutional neural networks (CNNs) have been shown to be powerful tools for learning visual representations from images. They often consist of multiple layers of nonlinear functions with a large number of trainable parameters. Hierarchical features can be obtained by training the CNNs discriminatively.
In the medical image computing domain, recent years have seen a growing number of applications using CNNs. Although there have been recent advances in tailoring CNNs to analyse volumetric images, most of the work to date studies image representations in 2D. While volumetric representations are more informative, the number of voxels scales cubically with the size of the region of interest. This raises challenges of learning more complex visual patterns as well as higher computational burden compared to the 2D cases. While developing compact and effective 3D network architectures is of significant interest, designing 3D CNNs remains a challenging problem.
The goal of this paper is to design a highresolution and compact network architecture for the segmentation of fine structures in volumetric images. For this purpose, we study the simple and flexible elements of modern convolutional networks, such as dilated convolution and residual connection. Most of the existing network architectures follow a fully convolutional downsampleupsample pathway [11, 4, 15, 3, 16, 13]. Lowlevel features with high spatial resolutions are first downsampled for higherlevel feature abstraction; then the feature maps are upsampled to achieve highresolution segmentation. In contrast to these, we propose a novel 3D architecture that incorporates high spatial resolution feature maps throughout the layers, and can be trained with a wide range of receptive fields. We validate our network with the challenging task of automated brain parcellation into structures from T1weighted MR images. We show that the proposed network, with twenty times fewer parameters, achieves competitive segmentation performance compared with stateoftheart architectures.
A welldesigned network could be trained with a largescale dataset and enables transfer learning to other image recognition tasks [9]
. In the field of computer vision, the wellknown AlexNet and VGG net were trained on the ImageNet dataset. They provide generalpurpose image representations that can be adapted for a wide range of computer vision problems. Given the large amount of data and the complex visual patterns of the brain parcellation problem, we consider it as a pretext task. Our trained network is the first step towards a generalpurpose volumetric image representation. It potentially provides an initial model for transfer learning of other volumetric image segmentation tasks.
The uncertainty of the segmentation is also important for indicating the confidence and reliability of one algorithm [5, 18, 19]. The high uncertainty of labelling can be a sign of an unreliable classification. In this work, we demonstrate the feasibility of voxellevel uncertainty estimation using Monte Carlo samples of the proposed network with dropout at test time. Compared to the existing volumetric segmentation networks, our compact network has fewer parameter interactions, and thus potentially achieves better uncertainty estimates with fewer samples.
2 On the elements of 3D convolutional networks
2.0.1 Convolutions and dilated convolutions.
To maintain a relatively low number of parameters, we choose to use small 3D convolutional kernels with only parameters for all convolutions. This is about the smallest kernel that can represent 3D features in all directions with respect to the central voxel. Although a convolutional kernel with voxels gives the same receptive field as stacking two layers of voxel convolution, the latter has approximately fewer parameters. Using smaller kernels implicitly imposes more regularisation on the network while achieving the same receptive field.
To further enlarge the receptive field to capture large image contexts, most of the existing volumetric segmentation networks downsample the intermediate feature maps. This significantly reduces the spatial resolution. For example, 3D Unet [3] heavily employs
voxel max pooling with strides of two voxels in each dimension. Each max pooling reduces the feature responses of the previous layer to only
of its spatial resolution. Upsampling layers, such as deconvolutions, are often used subsequently to partially recover the high resolution of the input. However, adding deconvolution layers also introduces additional computational costs.Recently, Chen et al. [2] used dilated convolutions with upsampled kernels for semantic image segmentation. The advantages of dilated convolutions are that the features can be computed with a high spatial resolution, and the size of the receptive field can be enlarged arbitrarily. Dilated convolutions can be used to produce accurate dense predictions and detailed segmentation maps along object boundaries.
In contrast to the downsampleupsample pathway, we propose to adopt dilated convolutions for volumetric image segmentation. More specifically, the convolutional kernels are upsampled with a dilation factor . For channels of input feature maps , the output feature channel generated with dilated convolutions are:
(1) 
where the index tuple runs through every spatial location in the volumes; the kernels consist of trainable parameters. The dilated convolution in Eq. (1) has the same number of trainable parameters as the standard convolution. It preserves the spatial resolution and provides a voxel receptive field. Setting to reduces the dilated convolution to the standard convolution. In practice, we implement 3D dilated convolutions with a splitandmerge strategy [2] to benefit from the existing GPU convolution routines.
2.0.2 Residual connections.
Residual connections were first introduced and later refined by He et al. [7, 8] for the effective training of deep networks. The key idea of residual connection is to create identity mapping connections to bypass the parameterised layers in a network. The input of a residual block is directly merged to the output by addition. The residual connections have been shown to make information propagation smooth and improve the training speed [7].
More specifically, let the input to the th layer of a residual block as , the output of the block has the form: ; where denotes the path with nonlinear functions in the block (shown in Fig. 1). If we stack the residual blocks, the last layer output can be expressed as: . The residual connections enables direct information propagation from any residual block to another in both forward pass and backpropagation.
2.0.3 Effective receptive field.
One interpretation of the residual network is that they behave like ensembles of relatively shallow networks. The unravelled view of the residual connections proposed by Veit et al. [20] suggests that the networks with residual blocks have a collection of unique paths.
Without residual connections, the receptive field of a network is generally considered fixed. However, when training with residual blocks, the networks utilise different paths and therefore features can be learned with a large range of different receptive fields. For example, the proposed network with residual blocks (see Section 3) has a maximum receptive field of voxels. Following the unravel view of the residual network, it consists of unique paths. Fig. 2 shows the distribution of the receptive field of these paths. The receptive fields range from to
, following a binomial distribution.
This differs from the existing 3D networks. For example, Deepmedic [11] model operates at two paths, with a fixed receptive field and respectively. 3D Unet [3] has a relatively large receptive field of about voxels. However, there are only eight unique paths and receptive fields.
Intuitively, given that the receptive field of a deep convolutional network is relatively large, the segmentation maps will suffer from distortions due to the border effects of convolution. That is, the segmentation results near the border of the output volume are less accurate due to the lack of input supporting window. We conduct experiments and demonstrate that the proposed networks generate only a small distortion near the borders (See Section 4.0.3). This suggests training the network with residual connections reduces the effective receptive field. The width of the distorted border is much smaller than the maximum receptive field. This phenomenon was also recently analysed by Luo et al. [14]
. In practice, at test time we pad each input volume with a border of zeros and discard the same amount of border in the segmentation output.
2.0.4 Loss function.
The last layer of the network is a softmax function that gives scores over all labels for each voxel. Typically, the endtoend training procedure minimises the cross entropy loss function using an
voxel image volume and the training data of class segmentation map where is:(2) 
where corresponds to the Dirac delta function, is the softmax classification score of over the th class. However, when the training data are severely unbalanced (which is typical in medical image segmentation problems), this formulation leads to a strongly biased estimation towards the majority class. Instead of directly reweighting each voxel by class frequencies, Milletari et al. [16] propose a solution by maximising the mean Dice coefficient directly, i.e.,
(3) 
We employ this formulation to handle the issue of training data imbalance.
2.0.5 Uncertainty estimation using dropout.
Gal and Ghahramani demonstrated that the deep network trained with dropout can be cast as a Bayesian approximation of the Gaussian process [5]. Given a set of training data and their labels , training a network with dropout has the effect of approximating the posterior distribution
by minimising the KullbackLeibler divergence term, i.e.
; where is an approximating distribution over the weight matriceswith their elements randomly set to zero according to Bernoulli random variables. After training the network, the predictive distribution of test data
can be expressed as . The prediction can be approximated using Monte Carlo samples of the trained network: , where is a set of samples from. The uncertainty of the prediction can be estimated using the sample variance of the
samples.With this theoretical insight, we are able to estimate the uncertainty of the segmentation map at the voxel level. We extend the segmentation network with a convolutional layer before the last convolutional layer. The extended network is trained with a dropout ratio of 0.5 applied to the newly inserted layer. At test time, we sample the network N times using dropout. The final segmentation is obtained by majority voting. The percentage of samples which disagrees with the voting results is calculated at each voxel as the uncertainty estimates.
3 The network architecture and its implementation
3.1 The proposed architecture
Our network consists of layers of convolutions. In the first seven convolutional layers, we adopt voxel convolutions. These layers are designed to capture lowlevel image features such as edges and corners. In the subsequent convolutional layers, the kernels are dilated by a factor of two or four. These deeper layers with dilated kernels encode mid and highlevel image features.
Residual connections are employed to group every two convolutional layers. Within each residual block, each convolutional layer is associated with an elementwise rectified linear unit (ReLU) layer and a batch normalisation layer
[10]. The ReLU, batch normalisation, and convolutional layers are arranged in the preactivation order [8].The network can be trained endtoend. In the training stage, the inputs to our network are
voxel images. The final softmax layer gives classification scores over the class labels for each of the
voxels. The architecture is illustrated in Fig. 3.3.2 Implementation details
In the training stage, the preprocessing step involved input data standardisation and augmentation at both image and subvolumelevel. At imagelevel, we adopted the histogrambased scale standardisation method [17]
to normalised the intensity histograms. As a data augmentation at imagelevel, randomisation was introduced in the normalisation process by randomly choosing a threshold of foreground between the volume minimum and mean intensity (at test time, the mean intensity of the test volume was used as the threshold). Each image was further normalised to have zero mean and unit standard deviation. Augmentations on the randomly sampled
subvolumes were employed on the fly. These included rotation with a random angle in the range of for each of the three orthogonal planes and spatial rescaling with a random scaling factor in the range of .All the parameters in the convolutional layers were initialised according to He et al. [6]. The scaling and shifting parameters in the batch normalisation layers were initialised to and respectively. The networks were trained with two Nvidia K80 GPUs. At each training iteration, each GPU processed one input volume; the average gradients computed over these two training volumes were used as the gradients update. To make a fair comparison, we employed the Adam optimisation method [12] for all the methods with fixed hyperparameters. The learning rate was set to , the step size hyperparameter was and was in all cases, except VNet for which we chose the largest that the training algorithm converges (). The models were trained until we observed a plateau in performance on the validation set. We do not employ additional spatial smoothing function (such as conditional random field) as a postprocessing step. Instead of aiming for better segmentation results by adding postprocessing steps, we focused on the dense segmentation maps generated by the networks. As we consider brain parcellation as a pretext task, networks without explicit spatial smoothing are potentially more reusable. We implemented all the methods (including a reimplementation of Deepmedic [11], Vnet [16], and 3D Unet [3]
architecture) with Tensorflow
^{1}^{1}1 The source code is available at https://github.com/giftsurg/HighRes3DNet.4 Experiments and results
4.0.1 Data.
To demonstrate the feasibility of learning complex 3D image representations from largescale data, the proposed network is learning a highly granular segmentation of T1weighted MR images of healthy controls from the ADNI dataset. The average number of voxels of each volume is about . The average voxel size is approximately . All volumes are biascorrected and reoriented to a standard RightAnteriorSuperior orientation. The bronze standard parcellation of brain structures and nonbrain outer tissues are obtained using the GIF framework [1]. Fig. 5(left) shows the label distribution of the dataset. We randomly choose 443, 50, and 50 volumes for training, test, and validation respectively.
4.0.2 Overall evaluation.
Architecture  Multilayer fusion  Num. param.  Loss type  DCS (%)  STD (%) 

HCdefault  Residual  0.81M  Dice loss  82.05  2.96 
HCdropout  Residual  0.82M  Dice loss  84.34  1.89 
NoResentropy  N/A  0.81M  Cross entr.  39.36  1.13 
NoResdice  N/A  0.81M  Dice loss  75.47  2.97 
Deepmedic[11]dice  Two pathways  0.68M  Dice loss  78.74  1.72 
3D Unet[3]dice  Feature forwarding  19.08M  Dice loss  80.18  6.18 
Vnet[16]  Feature forwarding  62.63M  Dice loss  74.58  1.86 
In this section, we compare the proposed highresolution compact network architecture (illustrated in Fig. 3; denoted as HCdefault) with three variants: (1) the HCdefault configuration without the residual connections, trained with crossentropy loss function (NoResentropy); (2) the HCdefault configuration without residual connections, trained with Dice loss function (NoResdice); and (3) the HCdefault configuration trained with an additional dropout layer, and makes predictions with a majority voting of Monte Carlo samples (HCdropout). For the dropout variant, our dropout layer employed before the last convolutional layer consists of kernels.
Additionally, three stateoftheart volumetric segmentation networks are evaluated. These include 3D Unet [3], Vnet [16], and Deepmedic [11]
. The last layer of each network architecture is replaced with a 160way softmax classifier.
We observe that training these networks with the cross entropy loss function (Eq. 2) leads to poor segmentation results. Since the crossentropy loss function treats all training voxels equally, the network may have difficulties in learning representations related to the minority classes. Training with the Dice loss function alleviates this issue by implicitly reweighting the voxels. Thus we train all networks using the Dice loss function for a fair comparison.
We use the mean Dice Coefficient Similarity (DCS) as the performance measure. Table LABEL:overall_cmp and Fig. 5(right) compare the performance on the test set. In terms of our network variants, the results show that the use of Dice loss function largely improves the segmentation performance. This suggests that the Dice loss function can handle the severely unbalanced segmentation problem well. The results also suggest that introducing the residual connections improved the segmentation performance measured in mean DCS. This indicates that the residual connections are important elements of the proposed network. By adopting the dropout method, the DCS can be further improved by 2% in DCS.
With a relatively small number of parameters, our HCdefault and HCdropout outperform the competing methods in terms of mean DCS. This suggests that our network is more effective for the brain parcellation problem. Note that Vnet has a similar architecture to 3D Unet and has more parameters, but does not employ the batch normalisation technique. The lower DCS produced by Vnet suggests that batch normalisation is important for training the networks for brain parcellation.
4.0.3 Receptive field and border effects.
We further compare the segmentation performance of a trained network by discarding the borders in each dimension of the segmentation map. That is, given a voxel input, at border size we only preserve the voxel output volume centred within the predicted map.
Fig. 7
plots the DCS and standard errors of segmentation according to the size of the segmentation borders in each dimension. The results show that the distorted border is around
voxels in each dimension. The border effects do not severely decrease the segmentation performance. In practice, we pad the volume images with zeros in each dimension, and remove the same amount of borders in the segmentation output.4.0.4 The effect of number of samples in uncertainty estimations.
This section investigates the number of Monte Carlo samples and the segmentation performance of the proposed network. Fig. 8(a) suggests that using samples is enough to achieve good segmentation. Further increasing the number of samples has relatively small effects on the DCS. Fig. 8(b) plots the voxelwise segmentation accuracy computed using only the voxels with an uncertainty less than a threshold. The voxelwise accuracy is high when the threshold is small. This indicates that the uncertainty estimation reflects the confidence of the network. Fig. 9 shows an uncertainty map generated by the proposed network. The uncertainties near the boundaries of different structures are relatively higher than the other regions.
Currently, our method takes about seconds to predict a typical volume with voxels. To achieve better segmentation results and measure uncertainty, Monte Carlo samples of our dropout model are required. The entire process takes slightly more than
minutes in total. However, during the Monte Carlo sampling at test time, only the dropout layer and the final prediction layer are randomised. To further reduce the computational time, the future software could reuse the features extracted from the layers before dropout, resulting in only a marginal increase in runtime when compared to a single prediction.
5 Conclusion
In this paper, we propose a highresolution, 3D convolutional network architecture that incorporates large volumetric context using dilated convolutions and residual connections. Our network is conceptually simpler and more compact than the stateoftheart volumetric segmentation networks. We validate the proposed network using the challenging task of brain parcellation in MR images. We show that the segmentation performance of our network compares favourably with the competing methods. Additionally, we demonstrate that Monte Carlo sampling of dropout technique can be used to generate voxellevel uncertainty estimation for our brain parcellation network. Moreover, we consider the brain parcellation task as a pretext task for volumetric image segmentation. Our trained network potentially provides a good starting point for transfer learning of other segmentation tasks.
In the future, we will extensively test the generalisation ability of the network to brain MR scans obtained with various scanning protocols from different data centres. Furthermore, we note that the uncertainty estimations are not probabilities. We will investigate the calibration of the uncertainty scores to provide reliable probability estimations.
Acknowledgements.
This work was supported through an Innovative Engineering for Health award by the Wellcome Trust [WT101957, 203145Z/16/Z], Engineering and Physical Sciences Research Council (EPSRC) [NS/A000027/1, NS/A000050/1], the National Institute for Health Research University College London Hospitals Biomedical Research Centre (NIHR BRC UCLH/UCL High Impact Initiative), UCL EPSRC CDT Scholarship Award [EP/L016478/1], a UCL Overseas Research Scholarship, a UCL Graduate Research Scholarship, and the Health Innovation Challenge Fund [HICFT4275, WT 97914], a parallel funding partnership between the Department of Health and Wellcome Trust. The authors would also like to acknowledge that the work presented here made use of Emerald, a GPUaccelerated High Performance Computer, made available by the Science & Engineering South Consortium operated in partnership with the STFC RutherfordAppleton Laboratory.
References
 [1] Cardoso, M.J., Modat, M., Wolz, R., Melbourne, A., Cash, D., Rueckert, D., Ourselin, S.: Geodesic information flows: Spatiallyvariant graphs and their application to segmentation and fusion. IEEE Transactions on Medical Imaging 34(9) (2015)
 [2] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. arXiv:1606.00915 (2016)
 [3] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D UNet: Learning dense volumetric segmentation from sparse annotation. In: MICCAI (2016)
 [4] Dou, Q., Chen, H., Yu, L., Zhao, L., Qin, J., Wang, D., Mok, V.C., Shi, L., Heng, P.A.: Automatic detection of cerebral microbleeds from MR images via 3D convolutional neural networks. IEEE Transactions on Medical Imaging 35(5) (2016)

[5]
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: ICML (2016)
 [6] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. In: ICCV (2015)
 [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
 [8] He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: ECCV (2016)
 [9] Huh, M., Agrawal, P., Efros, A.A.: What makes ImageNet good for transfer learning? arXiv:1608.08614 (2016)

[10]
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)
 [11] Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multiscale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36 (2017)
 [12] Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 (2014)
 [13] Kleesiek, J., Urban, G., Hubert, A., Schwarz, D., MaierHein, K., Bendszus, M., Biller, A.: Deep MRI brain extraction: A 3D convolutional neural network for skull stripping. NeuroImage 129 (2016)
 [14] Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the effective receptive field in deep convolutional neural networks. In: NIPS (2016)
 [15] Merkow, J., Kriegman, D., Marsden, A., Tu, Z.: Dense volumetovolume vascular boundary detection. In: MICCAI (2016)
 [16] Milletari, F., Navab, N., Ahmadi, S.A.: VNet: Fully convolutional neural networks for volumetric medical image segmentation. In: International Conference on 3D Vision (2016)
 [17] Nyúl, L.G., Udupa, J.K., Zhang, X.: New variants of a method of MRI scale standardization. IEEE Transactions on Medical Imaging 19(2) (2000)
 [18] Sankaran, S., Grady, L.J., Taylor, C.A.: Realtime sensitivity analysis of blood flow simulations to lumen segmentation uncertainty. In: MICCAI (2014)
 [19] Shi, W., Zhuang, X., Wolz, R., Simon, D., Tung, K., Wang, H., Ourselin, S., Edwards, P., Razavi, R., Rueckert, D.: A multiimage graph cut approach for cardiac image segmentation and uncertainty estimation. In: International Workshop on Statistical Atlases and Computational Models of the Heart (2011)
 [20] Veit, A., Wilber, M., Belongie, S.: Residual networks are exponential ensembles of relatively shallow networks. In: NIPS (2016)
Comments
There are no comments yet.