Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review

12/11/2017 ∙ by Jose Bernal, et al. ∙ University of Girona 0

In recent years, deep convolutional neural networks (CNNs) have shown record-shattering performance in a variety of computer vision problems, such as visual object recognition, detection and segmentation. These methods have also been utilized in medical image analysis domain for lesion segmentation, anatomical segmentation and classification. We present an extensive literature review of CNN techniques applied in brain magnetic resonance imaging (MRI) analysis, focusing on the architectures, pre-processing, data-preparation and post-processing strategies available in these works. The aim of this study is three-fold. Our primary goal is to report how different CNN architectures have evolved, now entailing state-of-the-art methods by extensive discussion of the architectures and examining the pros and cons of the models when evaluating their performance using public datasets. Second, this paper is intended to be a detailed reference of the research activity in deep CNN for brain MRI analysis. Finally, our goal is to present a perspective on the future of CNNs, which we believe will be among the growing approaches in brain image analysis in subsequent years.



There are no comments yet.


page 12

page 16

page 24

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Convolutional neural networks (CNNs), an outstanding branch of deep learning applications to visual purposes, have earned major attention in the last years due to its breakthrough performances in varied computer vision applications, such as in object recognition, detection and segmentation challenges 

russakovsky2015imagenet ; lin2014microsoft ; everingham2015pascal , in which they have achieved astonishing performances krizhevsky2012imagenet ; he2015deep ; szegedy2016inception ; noh2015learning ; chen14semantic . Their success has not been limited to reach top positions in different tasks, but also to achieve competent personnel levels in visual object recognition tasks ioffe2015batch and, perhaps more importantly, sensitive medical applications esteva2017dermatologist .

CNNs have been used in medical imaging applications since the 1990s in areas such as lung structure and nodule detection hasegawa1994convolution ; lo1995artificial and breast tissue classification sahiner1996classification . However, due to the lack of labelled training data and computational power limitations by that time, it was not possible to train deep CNNs without over-fitting. As a result, proposals in this regard were discontinued in the field for some years. With the time, large annotated training datasets and more powerful graphics processing units (GPUs) have been created, enabling researchers to continue working in the area. This trend can be observed in Fig. 1 in which the number of papers using these strategies has increased year after year from 2010. In brain image analysis, the pioneering work appeared in multiple sclerosis (MS) by Maleki et al. maleki2012diagnosis , after Krizhevsky et al. krizhevsky2012imagenet rekindled research on CNNs in 2012. Nowadays, deep CNN architectures are widely used in brain MRI for preprocessing data kleesiek2016deep , detecting and segmenting lesions dou2016automatic ; roth2014new ; sirinukunwattana2016locality ; anthimopoulos2016lung ; albarqouni2016aggnet ; brosch2016deep and segmenting tumours pereira2016brain ; kamnitsas2016efficient ; havaei2016brain , whole tissue moeskops2016automatic ; zhang2015deep ; chen2016voxresnet and sub-cortical structures Dolz2017 ; Wachinger2017 .

Although in medical image classification tasks, there are usually fewer classes compared to large-scale semantic image recognition, two significant difficulties hindering achieving similar accuracy to human raters. First, there is a lack of sufficiently labelled training data. Essentially, generating highly accurate labels and finding sufficient pre-processed and representative data require a considerable amount of time. Second, medical image annotation is carried out by experts which is subjective and error-prone llado2012automated ; llado2012segmentation . Learning a model from a less accurate representation of training samples degenerates the algorithm accuracy.

Brain MR image analysis has traditionally been an important area of research, attracting researchers to work on different tasks, such as lesion detection and segmentation, tissue segmentation and brain parcellation on neonatal, infant and adult subjects gonzalez2016review ; mitra2014lesion ; roura2015toolbox ; shen2012detecting . Several public brain MR image datasets are available to the community, especially those organised by the Medical Image Computing and Computer-Assisted Intervention (MICCAI111 society, actively encouraging research and publications in the field. These standard evaluation frameworks have been proposed for quantitatively comparing brain segmentation algorithms under the same directives and conditions. Consequently, the number of publications using CNNs in brain MRI has been increasing: of 23 works collected from PubMed, Scholar, IEEE Xplore and Scopus databases for this review work, 20 of them were published in the period from 2015 to May 2017.

Deep learning methods have been extensively reviewed in recent years bengio2013deep ; deng2014deep ; schmidhuber2015deep ; bengio2013representation ; lecun2015deep and discussed for particular applications deng2016deep ; guo2016deep ; angermueller2016deep ; mamoshina2016applications . Developments in the field of medical image analysis have been covered in shen2017deep ; litjens2017survey . To the best of our knowledge, this work is the first to present a detailed review of deep CNN techniques applied to brain MRI analysis. We intend to comprise all publications in this field published from 2012 to June 2017. Along with shen2017deep ; litjens2017survey , we aim to provide a good reading basis for newcomers to the topic.

Figure 1: Number of publications per year in IEEE-Xplore containing “deep learning” and “medical imaging” keywords from 2010 to 2017. (Queried: June 6th, 2017).

The rest of this review is structured as follows. The general concepts in CNNs are given in Section 2. Note that an experienced reader could skip this section. Afterwards, the different methods published in the literature on brain MRI are surveyed and analysed regarding advantages and disadvantages in Section 3. In Section 4, evaluations and comparisons of the works are performed, based on the reported numerical results. We conclude this work with a discussion, indicating future trends in the field.

2 Deep convolutional neural networks

For years, conventional supervised machine-learning techniques were built using automatic learning techniques and well-engineered algorithms. The approach consisted in taking the raw data, describing its content with low-dimensional feature vectors – using specific prior knowledge of the addressed scenario – and inputting the vectors to a trainable classifier. While the classifier was indeed useful for other purposes, the features were not necessarily generic. Indeed, the overall accuracy of the method would depend on how appropriately designed were the heuristics 

lecun1998gradient .

Representation learning appears as an alternative to this drawback: discover automatically suitable detection and classification representations from the input data. One of the first successful attempts using this strategy took place in 1998 when LeCun et al. lecun1990handwritten

presented a five-layer fully-adaptive architecture for addressing handwritten digit recognition. Despite its accuracy results (1% error rate and 9% reject rate from a dataset of around 1000 samples), the authors were able to apply neural networks to a real-world task. However, these authors were not able to scale a large number of hidden layers and larger images, mainly due to computational resource constraints and the “vanishing gradient” problem. This latter situation being the case when the gradients of the network’s output with regard to the parameters in early layers becomes negligible

279181 .

Two initial solutions were proposed in the literature to address the “vanishing gradient” problem. The first one was introduced by Hinton et al. hinton2006fast

, in which training was conducted in two stages. The first stage was training each layer in an unsupervised, greedy manner, essentially initialising the parameters with better values than random or uniform values. In the second stage, the network was fine-tuned using labelled data. This work rekindled interest in current deep learning strategies. The second solution appeared with the introduction of the ReLU activation function 

glorot2011deep , which showed successful results in large-scale image recognition tasks krizhevsky2012imagenet . Along with the availability of faster GPUs, these strategies have dramatically increased the research in deep neural networks.

One of the most widely adopted approaches of deep neural networks is the convolutional neural networks which can process array-like data lecun2015deep , such as images or video sequences. From a high-level perspective, the idea behind CNN is to identify compositional hierarchy features which object from the real world exhibit: low-level features (e.g. edges) form patterns, and these specific patterns form more intrinsic structures (e.g. shapes, textures). Further information about the building blocks of CNN is provided in following sections.

Figure 2: Generic architecture of convolutional neural networks. The output of each convolution operation at each layer is activated using activation functions before applying pooling operations. The convolution operation produces different numbers of feature maps, depending on the numbers of filters used. The pooling operations reduce the spatial dimensions of each feature map. After convolution and pooling the layers, the future maps are flattened in the fully connected layer before a prediction is made using linear classifiers.

2.1 Building blocks of CNN

2.1.1 Convolutional layer

CNNs learn the relationships among the pixels of input images by extracting representative features using convolution and pooling operations. The features detected at each layer using learnt kernels vary concerning their complexity, with the first layer extracting simple features, such as edges, and the later layers extracting more complex and high-level features. The convolution operation in CNNs has three main advantages. First, the weight-sharing mechanism helps to deal with high dimensional data, either 2D images or 3D data, such as videos and volumetric images. Second, local connectivity of the input topology can be exploited using 2D or 3D kernels. Finally, slight shift invariance is achieved using pooling layer. The typical architecture of a 2D CNN is shown in Fig. 


Very deep CNN architectures were recently proposed to replace the conventional convolutional layer with modules with more powerful representation while using less computational resources. For instance, Szegedy et al. szegedy2015going

introduced inception modules that could extract multi-scale features from the input feature maps and could efficiently reduce the number of parameters. This module was further improved in

ioffe2015batch and szegedy2015rethinking . The recent version of the inception module in szegedy2016inception was created to have a more uniform simplified architecture than previous versions and thus could achieve top performance in large-scale image classification tasks.

2.1.2 Non-linearity layer

The above convolutional layer is usually followed by non-linearity operations. Non-linearity is achieved using a specific family of functions called activation functions. These activation functions ensure that the representation in the input space is mapped to a sparse one and, hence achieving (i) a certain invariance to data variability and (ii) a computationally efficient representation glorot2011deep . The former situation refers to the fact that sparse representations are more resilient to slight modifications than dense ones. In the past, sigmoid and hyperbolic tangent functions were commonly used for this purpose. However, for large-scale image recognition, novel activation functions are being continuously proposed. We categorise the commonly used activation functions into three broad families.

Rectified linear units (ReLUs) and variants: which are expressed in a general form as


where is the input value at position on the feature map at the layer and is the slope of the negative linear function. There are five special cases distinguished depending on the conditions over . First, if , the expression results in the so-called ReLU krizhevsky2012imagenet ; nair2010rectified which is one of the most commonly used activation functions lecun2015deep . Despite its computationally efficient implementation, this method presents some drawbacks due to its gradient discontinuity at the origin in terms of gradient update and empirical performance szegedy2016inception ; he2015delving . Second, if is a small constant, the variant is referred as Leaky ReLU (LReLU) maas2013rectifier . This approximation enables to cope with the problem of zero gradient. Third, if is tuned up in the training process along with other parameters using back-propagation, the approach is referred as Parametric ReLU (PReLU) he2015delving . Fourth, in xu2015empirical , the parameter

is sampled from a uniform distribution for each example, and this approach is called Randomised ReLU (RReLU). Although using a slope parameter for the negative part in ReLU showed improvements in performance, RReLU showed better performance than the other ReLU variants, based on the evaluation in 

xu2015empirical on image classification tasks. Fifth, recently, Jin et al. jin2016deep proposed a new type of activation function, called S-shaped ReLU (SReLU), in which the essential idea is to consider a piecewise function composed of three linear functions, i.e.


where , , and are learnable parameters. According to the experiments in the paper, the SReLU is able to learn both convex and non-convex functions. This activation improved the performance of well-known CNN architectures in MNIST lecun1998mnist

and ImageNet 

deng2009imagenet datasets, compared to ReLU, LReLU and PReLU.

Maxout and variants: Maxout goodfellow2013maxout was proposed in particular to improve the optimisation and model averaging performance of dropout training. This activation function is also a generalisation of ReLU. It is computed by calculating the maximum across K affine feature maps, i.e.


One major drawback of this technique is that the number of weights to be learned in each layer is increased by a factor of . A workaround to this situation was proposed by Springenberg and Reidmiller springenberg2013improving in which a probabilistic sampling procedure to compute the maximum across feature maps, generalising maxout was considered. This activation function called Probout empirically matched or improved the performance of maxout springenberg2013improving .

Exponential Linear Units (ELU) and variants: ELUchen2015automatic are similar to the extensions of ReLU as they employ an identity for positive inputs. Unlike ReLU variants, they provide saturated output for negative inputs. The saturation in the negative regions of the function is reported to be beneficial for expediting the learning and improving the performance of very deep CNNs. It is defined as


Trottier et al. trottier2016parametric defined parameters controlling different aspects of the ELU function and proposed learning them with gradient descent during training. This parametric ELU (PELU) further improved the speed and performance of training deep networks. Using off-the-shelf ResNet he2015deep , PELU performed better than ELU and ReLU in image classification tasks on MNIST, CIFAR-10/100 and ImageNet datasets trottier2016parametric .

2.1.3 Pooling and sub-sampling layer

Typical convolutional layers consist of three steps. First, the layer performs several convolutions to produce feature maps. Second, non-linear activation functions are used on the resulting maps. Third, the output is modified by the pooling layer before reaching the next convolutional layer. The idea of a pooling function is to extract a summary statistics of non-overlapping neighbourhoods – usually – to (i) reduce the number of parameters in the following layers, (ii) control over-fitting, and (iii) achieve slight translation invariance Goodfellow-et-al-2016-Book .

Among several pooling methods, max pooling and average pooling are widely used types. Their operations are similar except that the former uses the maximum of the activations, and the latter uses the average of them. Max pooling is the most common due to its empirical performance

boureau2010learning ; boureau2010theoretical . Apart from these two approaches, other pooling methods are proposed to achieve higher performance.

Stochastic Pooling: Zeiler et al. 14d852e0427f418c8cba34d967623a3e

proposed stochastic pooling to regularise the convolutional layers and, hence, overcome the overfitting problem of average pooling and max pooling. In this method, the output of the activation from each pooling region is selected first by computing the probability distribution

by normalising the activations; then, the output activation is chosen by sampling from the multinomial distribution based on . The authors showed that this pooling strategy improved the accuracy on MNIST, CIFAR-10/100 and street view house numbers image recognition (SVHN) netzer2011reading datasets.

Spatial pyramid pooling: Another pooling method called spatial pyramid pooling he2014spatial , is proposed to work with any input image size. It is added after the last pooling layer, immediately before the fully connected layer, to generate fixed length representation regardless of the image size or scale. By simple modification of the off-the-shelf CNN architectures with a spatial pyramid pooling layer, this pooling scheme was reported to improve the performance of CNN models.

Deformation pooling: Despite max pooling and average pooling being useful in handling deformations, they are incapable of learning deformation constraints or geometric models from object parts. Deformation pooling (referred as def-pooling) was explicitly designed to overcome this drawback for object detection applications ouyang2015deepid .

Combination of max and average pooling: Lee et al. lee2016generalizing proposed a generalisation of max and average pooling operations that allowed them to be combined and adjusted in the training process. The combination operation was performed using mixed max-average and gated max-average pooling. The latter approach improved the characteristics of the region being pooled. The learning is performed during a combined pooling operation using a binary tree in which each leaf of the tree is associated with a learnt pooling filter. This strategy is called tree pooling. An experiment in lee2016generalizing reported that this method boosted the performance of AlexNet krizhevsky2012imagenet and GoogLeNet szegedy2015going models on MNIST, CIFAR-10 and SVHN datasets.

2.1.4 Fully connected layer

Unlike convolutional layer, the Fully Connected (FC) layer has a full connection to all of the units in the previous layer, as shown in Figure 2. This layer changes the previous layers 2D structure features into a predefined one-dimensional feature vector. Essentially, the main task of the FC layer is to mine the incoming features to extract information about the content of the input image. The process usually consists in flattening the feature maps coming from convolutional layers, to achieve a one dimension feature vector representation, and, then, inputting it into the FC layer. The output of this layer could either be the predict class labels krizhevsky2012imagenet or an intermediate layer girshick2014rich (consecutive FC layers can be stacked together).

Implementing FC layers usually require a large number of parameters – compared to other layers – as each cell of a feature map is fully connected to all elements in the previous layer. Besides, there are two drawbacks of these kinds of layers: (i) a single output is produced – if it is used as output layer – and (ii) accepts fixed-size inputs. The former issue means that a single input image receives a single output label. This situation is computationally inconvenient if the CNN is intended to be used in segmentation tasks rather than classification ones. The second issue relates to problems on extending the network: either the input images are scaled to fit the requirements of the network or the network is re-factorised to be able to process the new data. A solution to these problems lies in the fact that FC layers can be converted to convolutional layers of kernels long2015fully ( kernels in case of 3D). In this way, the model keeps the fully connected functionality while accepting arbitrary input size image and making dense predictions. These types of architectures, called fully convolutional networks (FCNN), have been gradually introduced in the literature dou2016automatic ; brosch2016deep ; kamnitsas2016efficient ; havaei2016brain ; Dolz2017 ; long2015fully .

2.1.5 Loss function

CNNs are well-known for their ability to extract discriminative features using learned weights in each layer. The learning process is reinforced by employing appropriate loss functions. Loss functions are designed to encourage intra-class similarity and inter-class separability.

In image classification tasks, most CNNs employ softmax loss, which is a combination of the softmax function and cross-entropy loss, mainly because of simplicity and the probabilistic interpretation of softmax classifiers. Hinge loss is another type of loss function, which in conjunction with margin-based classifiers was reported to perform better than cross-entropy loss in a standard image classification challenge tang2013deep . More recently, the research in liu2016large proposed a Large-Margin Softmax Loss (L-softmax), which is a modification of the softmax loss with a distance margin constraint. This loss was reported to boost the performance of deeply learned features in visual classification and verification tasks.

One of the main issues during the training phase is the disproportion among class observations (i.e. the number of available samples per class varies dramatically) as the resulting classifier may be biased towards the majority class. For instance, the region of interest to be analysed occupies only a small part of the scan. This problem can be addressed in the training phase by (i) undersampling the majority class kamnitsas2016efficient ; moeskops2016automatic , (ii) merging or subdividing classes chen2016voxresnet and (iii) penalising sample misclassification based on the reciprocal frequency cruz2016tackling ; or throughout the loss function. Brosch et al. brosch2015deep considered an approach for brain lesion segmentation in which the weighted sum of the mean squared difference of the lesion voxels (sensitivity) and non-lesion voxels (specificity) was used. Also, Milletari et al. milletari2016v proposed an objective function based on Dice Similarity Coefficient (DSC) for a two-class problem. According to the authors, this loss function yielded higher performance than the same architecture using softmax.

Other loss functions are task-specific, such as contrastive losses, which are mostly used to measure similarities between two data points. Su et al. sun2014deep

used a combination of softmax and contrastive losses for face recognition. The purpose of the softmax loss was to classify faces into different classes, thus encouraging inter-class separation. Then, the contrastive losses were applied for face verification to enforce intra-class similarity. A distance constraint was introduced to the contrastive loss in 

schroff2015facenet to obtain more discrimination of features.

2.1.6 Regularisations

We observed the appealing performance of deep CNN methods in different domains, although they use enormous numbers of parameters. Unless trained on a large, labelled training dataset, proper regularisation should be employed to mitigate over-fitting. There are several regularisation methods widely used in the community, such as or regularisation approaches encouraging sparsity and small weight magnitude; early stopping Goodfellow-et-al-2016-Book forcing the training to stop when there is a sign of over-fitting (also widely used to select hyper-parameters of the model); batch normalisation ioffe2015batch

in which each batch is preprocessed to achieve mean equal to zero and standard deviation equal to one; and dropout 

wager2013dropout ; srivastava14a in which some feature map units are skipped. This last approach being the dominant as (i) it is computationally inexpensive and (ii) prevents co-adaptation among feature map units by encouraging independent contributions of each of the units to the final prediction.

Different improvements in dropout have been proposed. For instance, Wang et al.  wang2013fast proposed a method to expedite training using dropout by sampling from a Gaussian approximation, instead of repeatedly sampling a random subset of input features. Another research in wan2013regularization proposed Dropconnect, which drops a subset of weights within the network instead of subset of activation within each layer. This approach showed better generalisation than the original dropout in standard image recognition tasks.

Data augmentation is another approach to improve the generalisation of CNNs by increasing the training dataset by artificially generating data. Although different image transformations krizhevsky2012imagenet ; szegedy2015going ; he2015delving ; howard2013some and colour perturbation chatfield2014return have been used and proved useful in recognition tasks, one limitation with these methods is that no theoretical background indicates the group of transformations helping to improve the generalisation capability of a model.

2.1.7 Optimisation

Deep CNNs are learnt by searching appropriate values for model parameters optimising the loss function. Gradient descent methods, in which the parameter update is performed using a back-propagation algorithm, are widely used for minimisation lecun1989backpropagation . This parameter update is undertaken by computing the loss function for a single, small subset, or the whole training set. Each of these cases is referred in the literature as stochastic, mini-batch and batch gradient descent, respectively. Updating values using the whole training set can be computationally expensive and, hence, mini-batch gradient descent is ubiquitously used in the community, leading to smoother parameter updating and more stable convergence.

The primary difficulty when optimising deep CNNs using gradient descent-based algorithms is the non-convex nature of the loss function. Non-convex functions have several local minima, in which gradient descent methods could get easily trapped sutton1986two . Dauphin et al. pascanu2014saddle

argued that saddle points, where the gradient vanishes at non-local optimum places, are much more of an issue than local optima in optimising non-convex loss functions. The momentum variable was introduced to the stochastic gradient descent (SGD) update method to avoid oscillation in local optima. Further, momentum-based SGD was improved to have some prescience about the next update direction, referred to as Nesterov accelerated gradient descent 

nesterov1983method ; sutskever2013importance . Recent works have also been proposed to escape saddle points during the optimisation process ge2015escaping ; anandkumar2016efficient .

One major drawback of the momentum-based SGD methods is to select an adequate learning rate, a parameter determining how substantial a change in the update should be made. This parameter is commonly set globally to be equal for all settings. Much work has been carried out on tuning the global learning rate adaptively based on the gradient of each parameter. One method in duchi2011adaptive , called adaptive gradient algorithm (Adagrad), scales the learning rate of each parameter according to the sum of the previous gradients. The amount of updating differs based on the sparsity of the parameters in each gradient update. Adadelta zeiler2012adadelta

and Root Mean Square Propagation (RMSProp

tieleman2012lecture improved the drawbacks of Adagrad unstable gradient updating by adopting a more stable learning rate scheme. Kingma and Ba kingma2014adam proposed Adam, which improved the gradient computation of both Adadelta and RMSProp. They also empirically showed improved performance in non-convex optimisation problems in different machine learning tasks.

2.1.8 Weight initialisation

Weight initialisation is crucial for non-convex optimisation algorithms. Initialising all the parameters to the same small value results in the undesired case in which all weights are updated the same during back-propagation. Another option that could break the symmetry is to initialise the weights with small values taken from a Gaussian distribution 

krizhevsky2012imagenet . Glorot and Bengio  glorot2010understanding

proposed a method known as “Xavier” initialisation that normalises the variance of each neuron’s output to one. There are two beneficial outcomes from this approach: (i) this avoids the variance in each layer and (ii) keeps the weights from exploding to large values or vanishing. This approach was later improved in 

he2015delving to account for the non-linearity of ReLU. This allows training very deep networks with better convergence than the “Xavier” initialisation. Another recent work from He et al. he2015delving proposed an initialisation by explicitly modelling the non-linearity of rectifiers (ReLU/PReLU), which helped the convergence of extremely deep models.

2.1.9 Normalisation

It is a common practice to use mean-centred training datasets by applying contrast normalisation to train deep CNNs krizhevsky2012imagenet ; pereira2016brain ; kamnitsas2016efficient ; havaei2016brain ; lin2013network . This simple preprocessing technique improves the convergence speed of SGD algorithms. Ioffe et al. ioffe2015batch argued that training deep networks can be slow since the distribution of parameters across hidden units changes dynamically during training, which is a phenomenon called the internal covariate shift. Their proposal consisted in normalising the data input in each layer, with a technique known as batch normalisation. This method provides any layer in a network with inputs that come from a unit Gaussian distribution and also enables the use of a high learning rate with a less careful weight initialisation choice. More recently, in salimans2016weight , the authors introduced weight normalisation. With simple weight re-parametrising at each layer, they were able to show improved conditioning of the optimisation problem and faster convergence of SGD methods.

3 CNN methods for brain image analysis

Automatic segmentation in brain MR has been widely investigated throughout the years to support medical tasks, such as diagnosis and patient monitoring. In the literature, most of the deep learning-based papers for brain MRI analysis have focused on lesion, tumour, tissue and whole brain and sub-cortical structure segmentation. This trend is primarily attributed to the different MICCAI challenges. Each year, the number of participants in these challenges employing deep learning approaches has been increasing. In this section, the proposed methods in this domain are discussed in more detail, based on their CNN architectures. Additionally, the description comprises the considered pipeline steps: pre-processing, data preparation, classification and post-processing techniques. There is no specific CNN architecture that is only suitable for a particular application; rather, a CNN model that is proposed for tumour segmentation could work for structure segmentation, and vice versa, with little or no modification.

The segmentation methods proposed in the literature could be seen from a top-level perspective as presented in Fig. 3. The overall pipeline consists of four stages: preprocessing, data preparation, classification, and post-processing. In the preprocessing stage the different pipelines consider noise filtering techniques, inter and intra-patient normalisation and skull-stripping methods – when necessary. Then, the data is prepared to be processed by the classifier. For instance, data preparation could contemplate augmenting the data or, in patch-based strategies, extracting patches from the input volumes. After that, classification takes place. Finally, once the segmentation results are obtained, they could be refined by removing small isolated areas by selecting the biggest groups only or smoothing regions. It is important to note that some works do not specify any preprocessing or post-processing methods. In the following sections, we will discuss each of these blocks emphasising on the CNN strategies.

Figure 3: A general block diagram of the CNN-based image analysis pipeline.

3.1 Pre-processing

Pre-processing in MRI is an essential step for subsequent segmentation task. In training supervised models such as CNNs, the input training data hugely influences the performance of the model, so having preprocessed and well-annotated data is a crucial step in achieving good performance.

Acquired brain MRI volumes incorporate non-brain tissue parts of the head, such as eyes, fat, spinal cord or skull. The process of extracting the brain tissue from non-brain one is referred in the literature as skull stripping. An example of an original volume and its corresponding skull stripped output is presented in Fig. 3(a) and  3(b), respectively. This step has direct consequences on the performance of automated methods, as the inclusion of skull or eyes as brain tissue may lead to unexpected results in classification acosta2008impact ; popescu2012optimizing , while unintended removal of the cortical surface may result in underestimation of the cortical thickness sadananthan2010skull . Among the different methods proposed in the literature for skull-stripping lee2003evaluation ; acosta2008impact ; roura2014marga , methods such as BET BET2002 ; BET22005 , BSE shattuck2001magnetic , ROBEX iglesias2011robust and BEaST Eskildsen20122362 are commonly used. In the literature, the methods used in clinical trail datasets employed BET brosch2016deep ; moeskops2016automatic ; yoo2014deep and ROBEX kamnitsas2016efficient . Zhang et al. zhang2015deep applied a paediatric brain skull stripping algorithm known as LABEL shi2012label . The public dataset images from the Brain Tumor Image Segmentation Challenge (BRATS) 2013222, 2014333 and 2015444 are preprocessed in this regard beforehand.

(a) T1-w image
(b) Skull stripped
(c) Bias field
(d) Preprocessed T1-w
Figure 4: An example of two preprocessing methods: skull stripping and bias field correction despotovic2015mri

. In the figure, a T1-w slice is displayed in (a), brain tissue after removing non-brain areas in (b), estimated bias field in (c), and preprocessed brain tissue in (d).

Inherent characteristics of the MRI acquisition process such as differences in the magnetic field, bandwidth filtering of the data or eddy currents driven by field gradients usually result in image artefacts that may also have a negative impact on the performance of the methods simmons1994sources . There is the need to remove spurious intensity variations caused by inhomogeneity of the magnetic fields and coils. In these cases, intensity correction of the MRI images is performed either before tissue segmentation, or as an integrated part of the tissue segmentation pipeline. A common technique to address this problem is to use bias-field correction juntu2005bias . The estimated bias field and the corrected version of Fig. 3(b) are depicted in Fig. 3(c) and 3(d), respectively. Among the available strategies arnold2001qualitative ; ge2002age , the non-parametric non-uniform intensity normalisation (N3) sled1998nonparametric and N4ITK tustison2010n4itk methods are currently the most widely used. Zhang et al. zhang2015deep and Yoo et al. yoo2014deep employed N3 algorithms on their clinical dataset. Similarly, Pereira et al. pereira2016brain used them in both BRATS 2013 and 2015 Challenges, Lyksborg et al. lyksborg2015ensemble in BRATS 2014, and Zikic et al. zikic2014segmentation in BRATS 2013.

Brain MRI datasets might have volumes acquired from different scanner vendors and also from the same scanner but with different protocols. As a result, the volumes may exhibit non-uniform intensity representation for the same tissue types, i.e. interclass variability. To correct this problem, image normalisation algorithms are utilised. According to the literature, this intensity normalisation can be driven in two ways: (i) histogram matching birenbaum2016longitudinal ; urban2014multi ; vaidhya2015multi ; kleesiek2016deep ; pereira2016brain and (ii) normalise data to achieve zero mean and unit variance pereira2016brain ; havaei2016brain ; chen2016voxresnet ; Dolz2017 ; valverde2017improving . In the former case, Urban et al. urban2014multi and Kleesiek et al. kleesiek2016deep considered matching the histogram of all volumes to a subject in the training set, which may result in fused grey levels, while Pereira et al. pereira2016brain – based on the normalisation method proposed by Nyul et al. nyul2000new – considered mapping to a virtual grey scale learnt directly from the data, so the undesired fusion of grey levels is avoided. Naturally, both normalisation strategies can be used one after the other one to improve the segmentation results. According to the results reported by Pereira et al. pereira2016brain , the preprocessing step improved their result, obtaining a mean gain of 4.6%.

In addition to the above discussed pre-processing methods, image registration between different MRI modalities is important depending on the dataset analysed. Image registration transforms different modalities of MRI into a common coordinate space. The authors of maleki2012diagnosis ; brosch2016deep ; kamnitsas2016efficient ; brosch2015deep 555The authors in maleki2012diagnosis and brosch2015deep reported using registration during their pre-processing stage but details were scarce in this regard. applied image registration algorithms on their clinic trial dataset. For instance, Brosch et al. brosch2016deep

applied a six degree-of-freedom intra-subject registration using one of the 3 mm scans as the target image to align the different modalities. Additionally, Kamnitsas et al. 

kamnitsas2016efficient applied affine atlas-based registration.

3.2 Data preparation

By data preparation, we refer to all of the operations performed before feeding the data into the network, such as data augmentation and patch extraction. Although this stage in most pipelines is considered to be pre-processing, we include it as a separate step since (i) the preprocessing steps are generic while these are particular for the CNN approaches and also (i) to provide more details.

Data augmentation is mainly employed to increase the training samples to mitigate over-fitting as discussed in Section 2.1.6. It is a common practice to use data augmentation in computer vision tasks in which (i) the CNN architectures are very deep, and (ii) obtaining enormous amounts of labelled training data is difficult. Moreover, unless the dataset is large enough to correctly train the network (which is not the common case havaei2016brain ; lyksborg2015ensemble ; vaidhya2015multi ), the high disproportion between the cardinality of the majority and the minority classes may lead to biased classifiers. Data augmentation could be used in these cases to alleviate and improve the overall performance of the model. Pereira et al. pereira2016brain used data augmentation for dual purposes: (i) to increase the training data by applying rotation at different angles and (ii) to introduce class balance by adding more data from the minority class. According to their experiments, by augmenting using rotation, they achieved better delineation of the complete tumour types, as well as of the intra-tumoural structures, reporting a mean gain of up to 2.6%. In addition, by oversampling the minority class, they stated a mean gain of 1.9%. In contrast, Havaei et al. havaei2016brain employed data augmentation by flipping the training data but had no success in improving the accuracy of the model. Furthermore, Chen et al. chen2016voxresnet used data augmentation as input, in addition to the given multi-modal images. The augmentation was carried out by subtracting a Gaussian smoothed version and applying histogram equalisation using the Contrast-Limited Adaptive Histogram Equalisation (CLAHE) pizer1987adaptive for enhancing local contrast chen2016voxresnet ; stollenga2015parallel . Afterwards, they used the generated and original volumes to train their network. Nevertheless, the actual effect of this type of augmentation was not reported in their work.

The patch extraction can be performed on a single plane (referred as 2D architectures), from three anatomical planes (referred as 2.5D architectures) or directly from 3D. Also, the patches can be acquired from different imaging modalities – if available. The advantages of one approach or the other one are discussed in the following section. After extracting the patches of training images, the data is normalised by subtracting the mean intensity and dividing by the standard deviation brosch2016deep ; pereira2016brain ; zhang2015deep ; chen2015automatic ; yoo2014deep ; dvorak2015structured . The resulting zero mean and unit variance training data helps to expedite the convergence of SGD. It is also a recommended practice to shuffle the samples randomly, especially when training using a mini-batch gradient decent algorithm bengio2012practical .

3.3 CNN architectures

The classification of CNN architectures in medical imaging can be grouped around five aspects: (i) number of interconnected operating modules, (ii) number of input modalities, (iii) input patch dimension, (iv) number of predictions at a time and (v) implicit and explicit contextual information. All these variants, their advantages and disadvantages are described in the following sections. A summary of the reviewed methods in the literature concerning pre-processing, post-processing and target of the segmentation are described in Table 1.

Architecture Article Pre-Processing Post-processing Target

Single CNN

2D CNN Maleki et al. 2012 maleki2012diagnosis Normalisation
- MS lesions
2D CNN Zikic et al. 2014 zikic2014segmentation N4ITK - Tumour
2D CNN Dvorak & Menze 2015 dvorak2015structured Normalisation
- Tumour
2D CNN Chen et al. 2015 chen2015automatic Normalisation - Cerebral micro-bleed
3D CNN Lai 2015 lai2015deep Masking in normalised coordinates Connected component analysis Hippocampus
2D CNN Zhang et al. 2015 zhang2015deep N3
Skull stripping (LABEL)
- Tissue
3D CNN Milletari et al. 2016 milletari2016hough - - Deep brain structure
3D CNN Dou et al. 2016 dou2016automatic Normalisation Connected component analysis Cerebral micro-bleed
3D CNN Urban et al. 2016 urban2014multi Normalisation Connected component analysis Tumour
3D CNN Brosch et al. 2016 brosch2016deep Normalisation
- MS lesions
2D CNN Pereira et al. 2016 pereira2016brain Normalisation
Connected component analysis Tumour
3D CNN Kleesiek et al. 2016 kleesiek2016deep Bias field correction
Intensity scaling
Connected component analysis Skull stripping
3D CNN Wachinger et al. 2017 Wachinger2017 3D CRF Sub-cortical structure
3D CNN Li et al. 2017 li2017compactness Normalisation Deep brain structure

Multi-path CNN

2.5D CNN Lyksborg et al. 2015 lyksborg2015ensemble N4ITK
Intensity scaling
Grow cut Tumour
2.5D CNN Birenbaum and Greenspan et al. 2016 birenbaum2016longitudinal Histogram matching MS lesions
2D & 3D CNN Brebisson et al. 2015 de2015deep - - Deep brain structure
3D RNN Stollenga et al. stollenga2015parallel Normalisation Tissue
2D CNN Havaei et al. 2016 havaei2016brain Outlier removal
Connected component analysis Tumour
2D CNN Zhao and Jia 2016 zhao2016multiscale - - Tumour
2D CNN Moeskops et al. 2016 moeskops2016automatic Intensity scaling
Bias field correction (likar2001retrospective )
Skull stripping (BET)
- Tissue
3D FCNN Kamnitsas et al. 2016 kamnitsas2016efficient Normalisation
Fully connected 3D CRF Tumour
2D FCNN Shakeri et al. 2016 shakeri2016sub 2D CRF Sub-cortical structure
3D CNN Chen et al. 2017 chen2016voxresnet Intensity normalisation - Tissue
2D & 3D CNN Mehta et al. 2017 mehta2017brainsegnet N4ITK
Skull stripping (BET)
Sub-cortical structure
3D FCNN Dolz et al. 2017 Dolz2017 Bias field correction
Skull stripping (BET)
Connected component analysis Sub-cortical structure
3D CNN Valverde et al. 2017 valverde2017improving Normalisation MS lesions
Table 1: Summary of deep learning approaches for MRI brain analysis with regard to the pre-processing, architecture, post-processing and target application.

3.3.1 Interconnected operating modules

According to the number of interconnected operating modules, the strategies in the literature can be classified into single-path and multi-path architectures. The single-path architectures correspond to the cases in which there is a unique flow of information: the input data is processed by convolutional, pooling and non-linear rectifier layers; the feature maps are then mined in the FC layers, and afterwards used for predicting the label in the output layer. An example of 2D single-path architecture is shown in Fig. 2. This category was popular in the literature maleki2012diagnosis ; dou2016automatic ; pereira2016brain ; kleesiek2016deep ; Wachinger2017 ; brosch2015deep ; zikic2014segmentation ; dvorak2015structured perhaps due to its fast computation and simplicity compared to multi-path architectures. In contrary to single-path networks, the multi-path architectures are a type of CNN in which independent operative networks are integrated into a single model to capture a more varied set of features. The modules can be arranged either in parallel or series. In the former approach, the idea is to achieve a consented label between different sources of information while in the latter approach the key is to reprocess the output of one network using another one.

The parallel multi-path architectures are composed of different CNNs designed to operate in parallel to capture more comprehensive features. Each network uses different versions of the same target area. There might be two possible variants of this type of architecture. The first variant aggregates features extracted from separate networks before making predictions, which can be understood as a process of information fusion that could occur at two levels: first, at a data level, where feature maps are merged to form a larger feature map; and second, at a feature level, where the extracted final features are concatenated before making the prediction. Fig. 

4(a) graphically depicts a multi-path architecture in which different features are processed using separate networks and are concatenated before making the prediction. This concatenation could also be performed at the convolutional layers, where feature maps from different models form a larger feature map in its third dimension. The second variant of multi-path architectures consists of learning ensembles of more than two networks to improve prediction. Predictions made by each module for a given target voxel are merged probabilistically.

There are several variants of the parallel multi-path category within the literature (i) multi-scale/multi-resolution kamnitsas2016efficient ; moeskops2016automatic ; Dolz2017 ; urban2014multi ; zhao2016multiscale – in which different views of the same ROI are used as input – (ii) 2.5D lyksborg2015ensemble – in which the data is extracted from the three anatomical planes – and (iii) 2.5D+3D de2015deep ; mehta2017brainsegnet – in which the different modules process 2.5D and 3D patches and operate simultaneously to provide consented labels. The fusion of various information sources allows the network to obtain contrast, local detailed and implicit contextual information.

(a) Multi-path architecture
(b) Cascade architecture
Figure 5: Architecture of multi-path CNNs. The diagram in (a) represents a multi-path CNN architecture: different operating CNNs process in parallel input data and output a consented label. The diagram in (b) corresponds to a cascade architecture of CNN networks: the output of one network is provided as input to another one to refine the segmentation.

The authors in zhao2016multiscale and moeskops2016automatic used multi-scale (i.e., different views of the same target area) 2D CNNs using three separate networks and combining the features immediately before the prediction step. According to the authors, the multi-scale design allowed the network to obtain fine-grained local and general information – presumably spatial context as well – to produce the classification of the target pixel. Similarly, Lyksborg et al. lyksborg2015ensemble trained three networks separately, using 2D slices from axial, sagittal and coronal planes. Unlike in zhao2016multiscale and moeskops2016automatic , the ensemble of the three networks was performed after the prediction step, and it was based on majority vote. The ensemble, in this case, was used twice: to segment the tumour area and to perform a within-tumour segmentation.

Another work, presented by Kamnitsas et al. kamnitsas2016efficient , proposed a network called DeepMedic that used a small kernel to increase the depth of the network to 11 layers, and they used two scales to train two separate networks for brain lesion segmentation. The output of each network was fused on the fully connected layer. The 11 layer 3D CNN produced a soft segmentation map. They extended a conditional random field krahenbuhl2011efficient to a fully connected 3D CRF and used it as post-processing to impose a regularisation constraint and produce the final hard segmentation labels.

Brebisson and Montana de2015deep were able to train eight networks arranged in parallel, with each network having a seven-layer CNN architecture for whole brain anatomical segmentation. The inputs of the network captured information at different scales around the voxel of interest: 3D and orthogonal 2D intensity patches captured a local spatial context, while downscaled, large 2D orthogonal patches and the distances to the regional centroids enforced global spatial consistency. They reported over-fitting, mainly due to the enormous number of parameters of the overall model and the few training images provided in the MICCAI 2012 challenge, which was a multi-atlas labelling challenge.

The second variant of multi-path architectures is the series arrangement. In this variant, different CNN networks are cascaded in series, with information from the previous network used as input for the latter, as shown in Fig. 4(b). It also renders the overall CNN deeper and, as a result, increases the expressing power of the model.

Havaei et al. havaei2016brain trained a cascaded architecture, starting by training a two-path CNN. Then, they fixed the parameters of these two-path networks and used the already trained models in the cascaded architecture. They evaluated the performance of concatenating feature maps from the last layer of the first network as feature maps for the second network at three different layers. From their three architectures submitted to the BRATS 2013 brain tumour challenge, they reported the best performance by concatenating the last layer feature maps of the first network, with the actual inputs used as input for the second network.

Valverde et al. valverde2017improving presented a cascaded 3D CNN approach for addressing MS lesion segmentation. Due to the conditions of the scenario (i.e. lack of a large number of samples and data imbalance), the authors opted for performing the segmentation in two steps. In the first step, the network is trained using all the positive samples and the same number of negative samples (i.e. the negative class is under-sampled). Once the classification is performed, the probabilities for each voxel to belong to the positive class are obtained. In the second step, the network is trained with all the positive voxels and the same number of misclassified negative samples. In the paper, no details regarding the improvement of the cascade approach over the single-path architecture are quantitatively discussed. Nevertheless, with this cascaded architecture, the authors were able to outperform the rest of participants on the MICCAI 2008 challenge.

Chen et al. chen2016voxresnet proposed a 3D CNN, VoxResNet, for brain tissue segmentation. The network is a 3D extension of the ResNet network he2015deep with a shallower depth. The authors also integrated segmentation maps from VoxResNet (used as the context information) together with the original volumes (i.e. appearance information) to train a new classifier called Auto-context VoxResNet. In this sense, possibly misclassified voxels are reprocessed and probably refined. It is important to understand that more than an “auto-context” technique, the procedure corresponds to a cascaded CNN similar to the ones described in havaei2016brain ; valverde2017improving . Although this classifier was intended to refine the segmentation iteratively, according to the authors, the improvement was marginal and, hence, a single iteration was considered.

In general, training multiple networks, rather than a single network, could lead to better performance. The networks can be integrated in parallel or series. The former approach permits the incorporation of different pieces of information, which help in labelling the central pixel. Different networks, which extract various useful features, could be aggregated to perform the desired segmentation. Each network arranged in parallel might need different input data. Training all of the networks arranged in parallel at the same time might demand much memory. Also, managing individual networks, training processes, selecting the best hyper-parameters and avoiding over-fitting also require a significant amount of time. The latter approximation, models arranged in series, allows refining the output of one network using another one, which needs a careful design and training process. This type of architecture performs better than single networks havaei2016brain .

3.3.2 Input modalities

The strategies in the literature can also be grouped according to the number of modalities that are processed at the same time. The categories are two: single- and multi-modality. The former case is certainly more adaptable to different scenarios as a single modality is commonly provided in datasets for tissue and sub-cortical structure segmentation (mainly T1). The latter case contemplates processing different sources of information. As stated by Zhang et al. zhang2015deep , it is important to consider fusing information from different imaging modalities – if possible – since it has been proven beneficial to discriminate when lacking contrast among tissues. The imaging sequences can be processed in parallel using multi-path configurations (e.g. one branch processes T1 and another one, FLAIR) or can be appended altogether to have multiple channels zhang2015deep ; chen2016voxresnet ; lyksborg2015ensemble . The difference between these two approaches is that specific kernels process each modality in the former case but, at the same time, the number of parameters increases.

3.3.3 Patch dimension

As mentioned previously, the different architectures can also be classified according to the dimension of the input patches. The categories, in this case, are three: 2D, 2.5D and 3D.

Two-dimensional architectures consider patches from a single plane (i.e. axial, sagittal or coronal). This architecture is widely used in the literature since it is adaptable to different image domains and segmentation tasks. In semantic computer vision, these types of networks are widespread. Compared to the earlier proposal from AlexNet krizhevsky2012imagenet , the depth of the recently proposed networks has shown a significant increase he2015deep ; szegedy2016inception ; szegedy2015going ; simonyan2014very . In MRI brain analysis, the architecture remains very similar despite being shallower, mainly because of limited available training data. It also seems unnecessary to design a very deep network if it does not improve the performance and, perhaps more importantly, it can be detrimental, due to potential over-fitting if too many network weights are required to be adjusted.

Diagnosing abnormalities employing a binary classifier is mostly the first step toward the localisation and segmentation of regions of interest. To the best of our knowledge, pioneering work using deep learning for brain image analysis was first reported in the detection of MS lesions by Maleki et al. maleki2012diagnosis . They used a conventional CNN architecture to extract features from 2D MR FLAIR images and a three-layer neural network as a classifier. A similar classification algorithm principle could be extended to patch-wise image segmentation algorithms. Usually incorporating multi-modal images during segmentation as a stack of 2D images improves performance. Zikic et al. zikic2014segmentation used 2D slices from T1, T1c, T2 and FLAIR images to classify small patches into five classes in the BRATS 2013 dataset. Their simplified CNN architecture consisted of 2D convolution, ReLU, max-pooling and softmax classifier layers.

Pixel-based labelling schemes that predict the label for a single voxel at a time have been a popular choice for training MR images using CNNs. Despite the success of this approach, scanning each voxel using a sliding window renders it computationally slow. To overcome this limitation, Dvorak and Menze dvorak2015structured fed their network several versions of the whole 2D slice input image, shifted on the X and Y axes, and they merged the outputs accordingly. They used images from multi-modal volumes to train a CNN for the prediction of extended label patches.

As mentioned earlier, in the context of brain MRI, building a very deep network is a rare practice because of the difficulty of obtaining sufficient representative and accurate data for training. In spite of the performance advantages, deeper networks usually have a larger number of parameters, and they may tend to suffer from over-fitting. Training on large datasets with proper regularisation algorithms is a common practice in the community. Pereira et al. pereira2016brain proposed a deep architecture, eleven layers, with 2D patches extracted from T1, T1c, T2 and FLAIR for gliomas segmentation. They made use of small kernels which allowed designing a deeper architecture simonyan2014very . A CNN built using those small kernels has fewer parameters, and therefore reduces the over-fitting problem. Moreover, they also took advantage of data augmentation using rotation to increase their original training dataset. Their 2D CNN consisted of multiple 2D convolutions, followed by LeakyReLU and max-pooling layers.

In the reviewed literature, we observed that the conventional CNN models are popular choices of MRI analysis researchers, perhaps because of the authors’ interest in utilising the already developed 2D CNN architectures, in addition to reducing computational complexity. This approach is in contrast to the actual 3D nature of the volume.

Tri-planar or 2.5D architectures are provided with patches from the three anatomical planes (i.e. axial, sagittal and coronal), commonly using a multi-path design. As explained by Lyksborg et al. lyksborg2015ensemble , the 2.5D information provides a better understanding of the 3D scenario than 2D-based networks since it exploits the 3D nature of MR images and, consequently, brings up contextual information. Moreover, although 2.5D architectures are more computationally expensive than 2D variants, these networks still consider 2D convolutions and, hence, are expected to be less costly than 3D alternatives dvorak2015structured ; lai2015deep .

Lyksborg et al. lyksborg2015ensemble presented a 2.5D multi-path approach in which the patches from different planes were processed independently by similar operative modules. By using probabilistic methods, the knowledge coming from the three branches was combined after the prediction layer. Although the approach is reliable, it somehow assumes that the contribution among the three branches is the same, which may not be the case at some point (i.e. due to the anisotropic nature of the data and/or to the fact that regions of interest may be more visible in specific planes). A workaround to this situation is to perform the merging step before the prediction layer and, hence, the weight of the features to be merged are specifically fixed for the case of study since the network determines the best configuration based on their contribution to the final segmentation.

Three-dimension architectures consist in taking 3D segments directly from the MRI volume. This architecture utilises 3D convolution kernels, which seems to be a more appropriate solution for fully exploiting the spatial contextual information in volumetric data. The main constraint of a 3D CNN approach lies in its expensive computational cost, memory requirements and computational time. Utilising the whole volume at a time is computationally very expensive; therefore, small, three-dimensional patches are extracted for training and testing. For instance, Urban et al. urban2014multi used four (multi-modal) voxels as input to segment a tumour in the BRATS 2013 dataset. Figure 6 shows a general 3D CNN architecture, in which the input is volumetric data, a 3D brain MR image in this case, and 3D kernels are used to perform the convolution.

Figure 6: Architecture of 3D CNNs. The convolution operation is performed by employing 3D filters. The pooling operation is also performed in three dimensions.

Lai and Rueckert lai2015deep investigated the use of three different convolutional network architectures for patch-based segmentation in ADNI hippocampus MRI dataset. They evaluated stacked 2D patches (one around the voxel in question, one in parallel and above it, and one in parallel and below it), tri-planar patches and 3D patches. According to their experiments, 3D convolutional architectures performed the best despite being the most computationally expensive.

To summarise, most of the algorithms, 10 of the 27, reviewed in the literature used a 2D CNN architecture. They were based on the two-dimensional convolutions used extensively in computer vision applications on natural images. This fact eases the model development process for practitioners by customising the popular CNN architectures and exploiting the available frameworks. Although patch-wise implementation of these types of architectures is very slow, converting the last output layer into a convolutional layer helps to expedite the inference time havaei2016brain . An implementation using a single network might not result in good performance, demonstrating that deeper networks that use smaller kernels achieve better performance, compared to using larger kernels and shallower architecture pereira2016brain . Two-dimensional CNNs disallow for exploiting of the actual 3D nature of the MRI data. In this sense, 2.5D and 3D CNNs were introduced mainly to overcome this limitation. Although both 2.5D and 3D architectures require more processing than 2D ones, the former type is less computationally expensive than the second one: 2D convolutional layers are used in 2.5D while 3D convolutional are used in 3D. Indeed, the main impediments to working with 3D CNNs are the number of parameters, memory requirements, expensive computational costs and slow speed. However, with the rapid development of GPUs, this situation may not represent a drawback in the coming years.

3.3.4 Predictions at a time

The approaches can also be grouped according to the number of predictions they perform at a time into CNN and FCNN Dolz2017 ; kamnitsas2016efficient ; shakeri2016sub . The former case corresponds to the traditional approach in which a single patch is processed by the network, and a single output is returned. Since the idea at the end is to obtain segmented areas, this approach can be slow in practice. The latter case corresponds to the architectures for which the fully connected layers are replaced by -kernel (or -kernel) convolutional layers to obtain a dense prediction. Additionally, some approaches consider upsampling layers to increase the number of predictions at a time. Naturally, the main drawback of these approaches is that a larger number of parameters are required to be set up (hence, more training samples to train the network properly are needed) in comparison to CNN strategies.

Dolz et al. Dolz2017 implemented a 3D FCNN architecture which was used to segment sub-cortical structures. Apart from the FCNN implementation, the authors considered multi-resolution information by extracting feature maps from high-resolution layers and merging them with low-resolution information, coming from the main information flow, before the -kernel convolutional layers. As explained by Bharath hariharan2015hypercolumns , this kind of connections allowed the network to learn from semantic – coming from deeper layers – as well as fine-grained localisation information – coming from shallow layers.

Ronneberger et al. ronneberger2015u proposed a multi-path 2D FCNN architecture called the U-Net. The network was successfully tested on the 2D electron microscopy em2015 and ISBI 2015 cell tracking666The challenge web page is challenges, achieving the first position in both of them. There were two key components on this FCNN. First, up-sampling layers were used to increase the resolution of the output. Thus, the volumes could be rapidly processed. In the paper, the authors were able to produce an output map of from an input image of size . Second, localisation and context were retained in the network by merging features maps from high-resolution layers with the up-sampled ones. On the other hand, it is important to highlight that this type of network, as stated by its authors, requires setting up a large number of parameters. Indeed, the original implementation required data augmentation techniques (for example, non-linear deformations generated on-the-fly during training) to achieve desired segmentation results.

Several approaches in different scenarios have been inspired by the U-Net architecture. Milletari et al. milletari2016v and Yu et al. yu2017

considered a 3D extension of the U-Net along with skip and/or residual connections for addressing the MICCAI Prostate MR Image Segmentation (PROMISE12). According to their experiments, the use of links improved the accuracy of the methods and also proved the effectiveness of information propagation. Also, the 3D variant has been implemented for confocal microscopic data 

cciccek20163d and liver and lesion segmentation in CT images christ2016automatic . Although none of these papers is directly applied to brain MRI, we consider that they could achieve relevance for brain MRI segmentation as well.

Brosch et al. brosch2016deep used 3D convolutional encoder networks for MS lesion segmentation. In addition to learning deconvolution and un-pooling, they introduced short-cut connections from the first to the last deconvolution layers inspired by the U-Net architecture ronneberger2015u

. The network in the convolutional encoder path was initialised using weights pre-trained on a stack of convolutional restricted Boltzmann machines. With this approach, they achieved top performance in public MS lesion segmentation challenge datasets.

3.3.5 Contextual information

The fact that the discussed architectures process patches separately and not the whole volume has some implications. For instance, the spatial distribution of brain structures is not directly encoded. Although some implicit contextual information is encoded in 2D, 2.5D and 3D patches their information is limited to the size of the patches. The bigger the size of the patch, the more information the network can take into account to produce the prediction; but the more the parameters to be trained. When considering the architecture of Havaei et al. havaei2016brain and the work of Kamnitsas et al. kamnitsas2016efficient , it seems to show that the context is important, but to preserve spatial correspondence of the multi-scale features is crucial. This could explain why a simple approach as pereira2016brain could outperform havaei2016brain .

Recently, Wachinger et al. Wachinger2017 introduced explicit within-brain location information through Cartesian and spectral coordinates Spectral2013 ; Cartesian2014

to help the classifier to discriminate one class from another. The Cartesian coordinates were obtained by taking the XYZ-coordinates of each of the voxels while the spectral coordinates came from the eigenvectors of the Laplacian matrix of the graph representing the brain mask. As explained by the authors in 

Wachinger2016 , the explicit spatial information is useful to (i) discriminate between patches from different brain hemispheres since patch-based strategies loss spatial context and (ii) help in the segmentation of tissues exhibiting low contrast. In this latter approach, it is essential to have in mind that (i) considering XYZ-coordinates on the image plane makes sense only if the volumes are registered and (ii) the distribution of the eigenvectors depends highly on the shape of the brain mask, i.e. these features may not be reliable when the shape of the region of interest differs considerably from one volume to another.

3.3.6 Summary of pros and cons of the different strategies

Table 2 summarises the main advantages and disadvantages of the approaches discussed previously in this section.

Strategy Advantage Disadvantage
Interconnected operating modules
Single path In principle, fast computation Single flow of information
Multi-path - parallel Extract more varied features
Verdict is consented among interconnected modules
Incorporated information may provide the network with contextual information (e.g. multi-resolution architectures)
More parameters than single-path architectures
More computationally demanding (data preparation and processing)
Multi-path - series Extract more varied features
Allows refining information at different stages
Requires careful design of the network
May require training different networks
The improvement could be marginal
Input modalities
Single modality Adaptable to different scenarios
Easily extensible to multiple modalities (channels or multi-path)
Single source of information
Multi-modality Gain valuable contrast information Requires more parameters than single modality
Patch dimension
2D Easily scalable to complex network architectures
Flexibility and adaptability
Fast computation
Heavily dependent on the network design tricks to obtain good results
Disallow 3D nature of MRI
2.5D Faster computation than 3D
Exploits the 3D nature of MRI
Acquires implicit contextual information
Computationally more expensive than 2D
3D CNN Exploits 3D nature of the MRI volume directly
Usually leads to better performance than 2D
Acquires implicit contextual information
Computationally expensive
Scaling to larger patch size may be computationally demanding
Due to the number of parameters, it may require large training data
Predictions at a time
CNN In principle, less number of parameters than FCNN Single classification for a single patch
FCNN Faster segmentation (some architectures can classify a single volume in one shot) Larger number of parameters to be set up
Generally, requires more training samples (data augmentation is commonly adopted in this sense)
Contextual information
Implicit No additional calculations are required Context depends on the size of the patch
Explicit Discriminative features are provided to the network
Sense of within-brain positioning
Subject to registration accuracy or shape of the region of interest
Table 2: Pros and cons of the segmentation strategies presented.

3.4 Post-processing

Post-processing is the step to refine or improve the results of the segmentation. The purpose of post-processing could be either to create hard segmentation labels kamnitsas2016efficient or to remove false positives dou2016automatic ; zhang2015deep ; pereira2016brain ; havaei2016brain ; urban2014multi ; lai2015deep from segmentation results, using different algorithms such as connected component analysis and Conditional Random Fields (CRF). The former strategy consists in keeping only the largest areas from the segmentation to remove spurious outputs. Nevertheless, in sub-cortical structure segmentation these spurious areas may have a larger volume than the small structures of interest and, hence, it becomes impractical. The latter strategy, CRF model, considers not only the given voxel but also its neighbours to produce a refined label, especially around the edges. Although CRF-based methods require learning parameters, they could bring regularisation properties to the segmentation since they are based on the minimisation of an energy equation. This stage helps to improve the overall performance of the model at the expense of adding computational complexity.

Contrast-driven post-processing techniques may lead to unexpected/unsatisfactory smoothing results. For instance, sub-cortical structures and some parts of the brainstem could be shrunk in this step due to low contrast with surrounding areas as shown in Fig. 7

. However, this does not signify that the method is not capable of refining the segmentation since both the intensity features and the prior probabilities take part in the process.

Figure 7: Two input slices in which low contrast between the region of interest and the surrounding areas is exhibited. In the image (a) the focus is on the mesencephalon which is highlighted by the purple area in (b). In (c) the focus is on caudate, putamen, pallidum and thalamus structures presented in green in (d).

In addition to CRF and connected component analysis, Lyksborg et al. lyksborg2015ensemble utilised an algorithm called Grow cut vezhnevets2005growcut . It was initially proposed as a continuous state cellular automata method for automated segmentation based on user labelled seed voxels. The authors used it to refine the initial tumour segmentation results. The post-processing algorithm was applied iteratively until a stable segmentation was obtained.

4 Evaluation

Brain MRI volume segmentation approaches are usually evaluated using different quantitative measurements. The described CNN models were tested in public and clinical research trial datasets. Regarding the works reviewed, both the most common datasets and the standard measurements used for the evaluation are described in this section. Moreover, the results obtained by the analysed methods on the standard datasets are presented.

4.1 Public evaluation frameworks and datasets

Several tissue segmentation algorithms have been proposed during the last years in the literature. However, determining which of them achieves the best performance can be complicated in principle since not all the algorithms are publicly available. A way to address this situation is by creating standard evaluation frameworks, such as the grand challenges taking place on relevant international conferences. Thus, before delving into a detailed discussion about the methods, we describe some of the most widely used public datasets for brain lesions, structures and anatomical segmentation.

Name Number of scans Modality Scanner Segmentation tasks Reconstruction matrix Pixel spacing ()
Train Test
MICCAI 2008 styner20083d 20 24 T1, T2, and FLAIR 3T Siemens Allegra MS lesion
MICCAI 2012 15 20 T1 Whole brain, tissue and sub-cortical structure
NeoBrainS12 ivsgum2015evaluation 20 5 T1 and T2 3T Philips Achieva Tissue

IBSR18 18 T1 Tissue and sub-cortical structure

MRBrainS13 mendrik2015mrbrains 5 15 T1,T1_1mm, T1_IR and FLAIR 3T Philips Achieva Tissue

T1_1mm has
BRATS 2012 menze2015multimodal 80 30 T1, T1c, T2, and FLAIR Tumour
BRATS 2013 menze2015multimodal 30 25 T1, T1c, T2, and FLAIR Tumour
BRATS 2014 menze2015multimodal 309 T1, T1c, T2, and FLAIR Tumour
BRATS 2015 menze2015multimodal 274 110 T1, T1c, T2, and FLAIR Tumour
ISBI 2015 carass2017longitudinal 21 61 T1, T2, PD, and FLAIR 3T MRI Longitudinal MS lesion
ISLES 2015 maier2017isles 28 36 FLAIR, T2 TSE, T1 TFE/TSE, DWI 3T Philips Ischemic stroke lesion
30 20 T1c, T2, DWI, CBF, CBV, TTP, Tmax 1.5T Siemens Magnetom Avanto
3T Siemens Magnetom Trio
MSSEG 2016 commowick2016msseg 15 38 T1, T1 GADO, FLAIR, DP/T2 3T GE Discovery
3T Philips Ingenia
1.5T Siemens Aera
3T Siemens Verio
MS lesion

LBPA40 40 T1 1.5T GE Brain structure
Hammers95n30 30 T1 1T Philips HPQ Brain structure Varied
Hammers83n30 30 T1 1T Philips HPQ Brain structure Varied
Hammers67n20 20 T1 1T Philips HPQ Brain structure Varied
Table 3: Summary of commonly used public datasets in MS, tumour, tissue and structure segmentation. The table is structured as follows. The different datasets are listed in the first column and their corresponding information: number of scans, offered modalities, acquisition scanner, segmentation tasks in which it is used, and image information such as reconstruction matrix and pixel spacing are detailed from column two to seven.

Some of the most commonly used public datasets in brain image MRI analysis are summarised in Table 3. The reviewed works primarily used MICCAI datasets for MS lesions, tumours, tissue and structure segmentation. For MS lesion segmentation, the challenge was created in conjunction with MICCAI 2008777 The challenge was part of the 3D segmentation in the Clinic Grand Challenge II and consisted of 54 brain MRI images, 20 of which are available for training with their corresponding ground truth. For brain tumour segmentation, the series of multi-modal brain tumour segmentation (BRATS) challenges are commonly used. These challenges started in 2012 with 80 cases of real and synthetic data and, every year, the size of the training and testing data has been enlarged (up to 300+ cases in total). For sub-cortical structure segmentation, the MICCAI 2012 888 challenge in multi-atlas labelling and the Internet Brain Segmentation Repository (IBSR)999Available at are widely used. It is important to note that MICCAI 2012 challenge is initially intended to evaluate algorithms on whole-brain structure segmentation, but naturally, the labels can be ignored and merged to obtain the sub-cortical structures only. Additionally, the IBSR dataset can be used for both sub-cortical structures and brain tissue segmentation. Other datasets widely adopted to evaluate algorithms for tissue segmentation are two the MICCAI Grand Challenges MRBrainS13101010 and NeoBrainS12111111 The difference between the two datasets is that in the former case, the algorithms are evaluated using adults exhibiting WM lesions and, in the latter case, the data comes from neonatal subjects.

The main evaluation measures for the challenges mentioned previously are DSC, specificity, sensitivity, positive predictive value (precision), average surface distance (ASD), average volumetric difference (AVD) and modified Hausdorff distance (MHD).

4.2 Evaluation measurements

Regarding the application, we found that the CNN models were applied for MS lesion segmentation, brain tumour segmentation and structure segmentation. Although there are public datasets available for these applications, researchers still prefer to work on their databases. One main reason might be the limitation of the number of training samples available in the public datasets. As stated before, working with a small dataset affects the performance of deep CNN models enormously and imposes restrictions on practitioners from fully exploiting the capacity of those algorithms.

The evaluation measurements compare the output of segmentation algorithms with ground truth in either a pixel-wise or a volume-wise basis. For a more robust assessment, more than one expert might be involved in generating the ground truth volumes to avoid inter-observer variability. In the reviewed literature, the results were evaluated using different evaluation metrics. In cases of MS and tumour segmentation, the most common evaluation measurements are DSC, precision, recall, the true positive rate (TPR) and the positive predictive value (PPV). Also, absolute volume difference, lesion-wise true positive rate (LTPR) and the lesion-wise false positive rate (LFPR) are also employed. Similarly, for brain tumour and structure segmentation applications, the DSC score, precision, and recall are among the widely used measurements. One main reason for this choice is that the challenge organisers primarily rely on these results. Table 

4 summarises the types of databases, numbers of samples, modalities considered, evaluation measurements applied and corresponding results reported of the surveyed works.

Article Dataset Number of samples Modality Classes Evaluation measures Result
Train Val Test
Maleki et al. maleki2012diagnosis Clinical trial 152 - - FLAIR 2 Sensitivity
Brosch et al. brosch2016deep MICCAI 2008 20 - 23 T1, T2, PD and FLAIR 2 VD
UNC 63.5%, CHB 52.0%
UNC 47.1%, CHB 56.0%
UNC 52.7%, CHB 49.8%
ISBI 2015 20 1 61 T1, T2, PD and FLAIR 2 DSC
Clinical trial 250 50 77 T1, T2 and FLAIR 2 DSC
Chen et al. chen2015automatic Clinic data 55 - 55 - 2 Sensitivity
Average FP
Dou et al. dou2016automatic Clinical data 924 108 117 SWI 2 Sensitivity
Mean FP
Birenbaum and Greenspan birenbaum2016longitudinal ISBI 2015 21 - 61 T1, T2, PD and FLAIR 2 DSC 62.7%
Valverde et al. valverde2017improving MICCAI 2008 20 - 23 T1, T2, PD and FLAIR 2 VD
UNC 62.5%, CHB 48.8%
UNC 55.5%, CHB 68.7%
UNC 46.8%, CHB 46.0%
Clinical trial 1 35 T1, T2 and FLAIR 2 DSC
Clinical trial 2 25 T1, T2 and FLAIR 2 DSC

Zikic et al. zikic2014segmentation BRATS 2013 20 - - - 5 DSC Complete 83.7%, Core 73.6%, Enhanced 69.0%
Urban et al. urban2014multi BRATS 2013 30 - 15 T1, T1c, T2 and FLAIR 5 DSC Complete 87.0%, Core 77.0%, Active 73.0%
Dvorak & Menze dvorak2015structured BRATS 2014 166 25 66 T1,T1c,T2 and FLAIR DSC Complete 83% , Core 75%, Active 77%
Lyksborg et al. lyksborg2015ensemble BRATS 2014 91 - 187 T1, T2, PD, and FLAIR 2 DSC
Positive predictive
Complete 79.9%, Core 63.1%, Enhancing 62.5%
Complete 0.78, Core 0.63, Enhancing 0.58
Complete 0.86, Core 0.74 Enhancing 0.78
Pereira et al. pereira2016brain BRATS 2013 30 - 25 T1, T1c, T2 and FLAIR 5 DSC
Complete 88%, Core 83%, Enhanced 77%
Complete 0.85, Core 0.82, Enhanced 0.60
Complete 0.86, Core 0.76, Enhanced 0.68
BRATS 2015 30 - 25 T1, T1c, T2 and FLAIR DSC Complete 78.0%, Core 65.0%, Enhanced 75.0%
Kamnitsas et al. kamnitsas2016efficient Clinical data 46 - 15 MPRAGE, FLAIR, T2, PD and GE 2 DSC
BRATS 2015 274 - 110 T1, T1c, T2 and FLAIR 5 DSC
Complete 84.9%, Core 66.7%, Enhanced 63.4%
Complete 85.3%, Core 86.1%, Enhanced 63.4%
Complete 87.7%, Core 60.0%, Enhanced 67.4%
ISLES 2015 28 - 36 T1, T1c, FLAIR and DWI 2 DSC
Havaei et al. havaei2016brain BRATS 2013


- 25 T1, T1c,T2 and FLAIR 5 DSC
Complete 84.0%, Core 71.0%, Enhancing 57.0%
Complete 0.88, Core 0.79, Enhancing 0.54
Complete 0.84, Core 0.72, Enhancing 0.68
Zhao and Jia zhao2016multiscale BRATS 2013 80 - 30 T1, T1c,T2 and FLAIR 5 DSC 81.0% 9.5%

Zhang et al. zhang2015deep Clinic data 7 1 - T1, T2 and FA 3 DSC
85.03% 2.27%
0.32 0.12

Stollenga stollenga2015parallel
MRBrainS13 5 15 T1, FLAIR and T1_IR 5 DSC
WM 88.33%, GM 84.82%, CSF 83.72%
WM 2.07, GM 1.69, CSF 2.14
WM 7.05, GM 6.77, CSF 7.10

Moeskops et al. moeskops2016automatic
NeoBrainS12 - - - T2 8 DSC Cor. 30 wks 82.7%, Ax. 40 wks 80.5%, Cor. 40 wks 81.9%

MRBrainS13 5 - 15 T1 7 Mean DSC
Mean ASD
MICCAI 2012 15 - 20 T1 6 Mean DSC
Mean ASD
Chen et al. chen2016voxresnet MRBrainS13 5 - 15 T1, T1_IR and FLAIR 4 DSC
WM 89.39%, GM 86.12%, CSF 83.96%
WM 1.94, GM 1.47, CSF 2.28
WM 5.84, GM 6.42, CSF 7.44

Lai lai2015deep ADNI 120 40 40 - 3 Error rate 7.21%
Brebisson et al. de2015deep MICCAI 2012 15 - 20 T1 134 Mean DSC 72.5%
Milletari et al. milletari2016hough Clinical data 45 - 10 QSM 26 Mean DSC 77.0%
Shakeri et al. shakeri2016sub IBSR18 18 T1 5 Mean DSC 82.40%
Mehta et al. mehta2017brainsegnet MICCAI 2012 15 20 T1 134 Mean DSC 74.30%
IBSR18 9 9 T1 32 Mean DSC 84.4%
LONI-LPBA40 20 20 T1 54 Mean DSC 82.4%
Hammers67n20 10 10 T1 67 Mean DSC 84.0%
Hammers83n30 15 15 T1 83 Mean DSC 80.8%
Wachinger et al. Wachinger2017 MICCAI 2012 15 15 T1 25 Mean DSC 90.6%
Dolz et al. Dolz2017 IBSR18 12 3 3 T1 4 Mean DSC 90.0%
ABIDE 150 15 947 T1 4 Mean DSC 90.0%
Li et al. li2017compactness ADNI 443 50 50 T1 155 DSC

Skull stripping
Kleesiek et al. kleesiek2016deep IBSR 18 T1 2 DSC
LPBA40 40 T1 2 Sensitivity
OASIS 77 T1 2 Specificity
Clinical data 53 T1, T1c, T2, FLAIR 2 DSC

Table 4: Summary of results in the reviewed papers. The acronyms for the sequences stand for: susceptibility-weighted imaging (SWI), quantitative susceptibility mapping (QSM), proton density (PD), fractional anisotropy (FA), magnetisation prepared rapid gradient echo (MPRAGE) and gradient echo (GE). The acronyms in the measurements stand for: modified Hausdorff distance (MHD), positive predictive (PP) and average symmetric surface distance (ASSD). The acronyms in results are: grey matter (GM), white matter (WM), cerebrospinal fuild (CSF), dataset from University of North Carolina (UNC) and dataset from Children’s Hospital Boston (CHB).

5 Discussion and future directions

Deep CNNs have made a large impact in a broad range of application domains. Today, they are the first choice to solve many problems in computer vision, speech recognition and natural language processing. Taking computational advantage of working with small 2D and 3D patches, rather than the entire slice or volume, researchers in brain MR image analysis can train deep CNNs to obtain accurate segmentation algorithms. This success has received overwhelming acceptance by the community, in which shallow architectures were predominantly used. Currently, most CNN architectures have many layers, including additional normalisation layers, such as batch normalisation. Furthermore, each architecture is becoming increasingly more sophisticated, employing ideas from optimisation and probabilistic models.

About the architectures and their performances, the majority of the proposed works in Table 1 used 2D networks. Although working with each model has its computational advantages and drawbacks, obtaining good generalisation requires an architecture with optimised layers, considering the class imbalance and selecting the best hyper-parameters and advanced training procedures. From the results shown in Table 4, methods with 2D CNN architecture with sufficient depth pereira2016brain , cascade havaei2016brain , “short-cut connections” brosch2016deep and parallel networks moeskops2016automatic ; de2015deep showed top performance in their respective applications. After all, there is no universal architecture, and the ongoing research contributes to obtaining a model that can learn to provide a good representation of the underlying input image without suffering from significant over-fitting.

In the reviewed works, we observed that most of the proposed methods emphasised the shortcomings of working with deep CNN models. First, there is the computational requirement. Analysing, manipulating and processing each voxel in a volume is expensive computationally. The enormous amount of memory needed to store the extracted patches and a large amount of time required to process them constitute a difficulty. The deep learning software libraries used to implement layers of deep CNNs have either parallel or distributed frameworks, which help researchers to train their models in multi-core architectures or GPUs. Second, training CNN models for brain image analysis is hindered by the data imbalance problem, especially with small lesion or structure segmentation. For instance, in tumour or MS lesion segmentation, obtaining good generalisation is a challenge as most of the lesions are smaller than the entire volume. Two-phase training havaei2016brain , careful patch selection pereira2016brain ; moeskops2016automatic ; lai2015deep and loss functions brosch2016deep ; sudre2017generalised ; fidon2017wassertein are among the proposed strategies to overcome this problem.

Although we observed the success of CNNs; their full capacity has not yet been fully leveraged in brain MRI analysis. Networks can perform exceptionally when trained and tested on images with similar acquisition characteristics (e.g. resolution, intensity range), but when tested on sets with slight variations concerning the training set, they tend to fall behind traditional methods like FAST (FAST2001, ) and SPM (SPM5, ; SMP8, ). This is the leading cause why traditional approaches are still being used in medical centres due to their robustness and adaptability (valverde2017automated, ). Even though there is progress in domain adaptation techniques, more research in this sense is needed before being able to use resilient CNN-based software on medical centres.

It is becoming a widespread practice in the computer vision community to release source codes to the public. This practice helps to expedite the research in the field. The most commonly used deep learning libraries for MRI segmentation are Tensorflow


, Theano


, Caffe


, Keras


and PyTorch

161616 Also, open-source frameworks with implementation of major CNN approaches can be found online, such as NiftyNet171717 and DLTK181818 Another recommended practice is validating the model on different datasets. Few works kleesiek2016deep ; brosch2016deep ; pereira2016brain ; kamnitsas2016efficient ; moeskops2016automatic ; mehta2017brainsegnet have reported their results on three or more different public datasets. This practice opens the door to design a robust model that can be applied to datasets of similar applications but with different types of MRI scanners, imaging modalities and numbers of training cases.

With the lack of training data, the poor spatial resolution of MR images and the need for a short prediction time, it has been impossible to train considerably deep CNNs. To train such networks, a considerable effort is needed in designing faster methods to perform convolutions. FFT algorithms brosch2016deep and quicker matrix multiplication methods lavin2015fast have been used to improve the computation speed of CNNs, but there is yet room for improvement in the training algorithms of deep CNNs using variants of SGD paine2013gpu and their parallelised and distributed implementations. The new algorithms to come are expected not only to improve the performance of deep CNNs but also to be highly optimised, with less or no hyper-parameters, which constitute one of the major bottlenecks for most users to tune.


Jose Bernal and Kaisar Kushibar hold FI-DGR2017 grants from the Catalan Government with reference numbers 2017FI B00476 and 2017FI B00372, respectively. Daniel S. Asfaw holds an FI-DGR2016 grant from the Catalan Government. This work has been partially supported by La Fundació la Marató de TV3, by MPC UdG 2016/022 grant, by Retos de Investigació TIN2014-55710-R, TIN2015-73563-JIN and DPI2017-86696-R from the Ministerio de Ciencia y Tecnología. The authors gratefully acknowledge the support of the NVIDIA Corporation with their donation of the Tesla K40 and the Titan X PASCAL GPUs used in this research.


  • (1) O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition challenge, International Journal of Computer Vision 115 (3) (2015) 211–252.
  • (2) T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: European Conference on Computer Vision, Springer, 2014, pp. 740–755.
  • (3) M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, A. Zisserman, The pascal visual object classes challenge: A retrospective, International Journal of Computer Vision 111 (1) (2015) 98–136.
  • (4) A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097–1105.
  • (5)

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

  • (6) C. Szegedy, S. Ioffe, V. Vanhoucke, A. A. Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA., 2017, pp. 4278–4284.
  • (7) H. Noh, S. Hong, B. Han, Learning deconvolution network for semantic segmentation, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1520–1528.
  • (8) L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille, Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs, in: International conference on learning representations, 2015,
  • (9)

    S. Ioffe, C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, in: Proceedings of The 32nd International Conference on Machine Learning, 2015, pp. 448–456.

  • (10) A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, S. Thrun, Dermatologist-level classification of skin cancer with deep neural networks, Nature 542 (7639) (2017) 115–118.
  • (11) A. Hasegawa, S.-C. B. Lo, M. T. Freedman, S. K. Mun, Convolution neural-network-based detection of lung structures, in: Medical Imaging 1994, International Society for Optics and Photonics, 1994, pp. 654–662.
  • (12) S.-C. Lo, S.-L. Lou, J.-S. Lin, M. T. Freedman, M. V. Chien, S. K. Mun, Artificial convolution neural network techniques and applications for lung nodule detection, IEEE Transactions on Medical Imaging 14 (4) (1995) 711–718.
  • (13) B. Sahiner, H.-P. Chan, N. Petrick, D. Wei, M. A. Helvie, D. D. Adler, M. M. Goodsitt, Classification of mass and normal breast tissue: a convolution neural network classifier with spatial domain and texture images, IEEE transactions on Medical Imaging 15 (5) (1996) 598–610.
  • (14) M. Maleki, M. Teshnehlab, M. Nabavi, Diagnosis of Multiple Sclerosis (MS) Using Convolutional Neural Network (CNN) from MRIs, Global Journal of Medicinal Plant Research 1 (1) (2012) 50–54.
  • (15) J. Kleesiek, G. Urban, A. Hubert, D. Schwarz, K. Maier-Hein, M. Bendszus, A. Biller, Deep MRI brain extraction: a 3D convolutional neural network for skull stripping, Neuroimage 129 (2016) 460–469.
  • (16) Q. Dou, H. Chen, L. Yu, L. Zhao, J. Qin, D. Wang, V. C. Mok, L. Shi, P.-A. Heng, Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks, IEEE transactions on medical imaging 35 (5) (2016) 1182–1195.
  • (17) H. R. Roth, L. Lu, A. Seff, K. M. Cherry, J. Hoffman, S. Wang, J. Liu, E. Turkbey, R. M. Summers, A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2014, pp. 520–527.
  • (18) K. Sirinukunwattana, S. E. A. Raza, Y.-W. Tsang, D. R. Snead, I. A. Cree, N. M. Rajpoot, Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images, IEEE transactions on medical imaging 35 (5) (2016) 1196–1206.
  • (19) M. Anthimopoulos, S. Christodoulidis, L. Ebner, A. Christe, S. Mougiakakou, Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network, IEEE transactions on medical imaging 35 (5) (2016) 1207–1216.
  • (20) S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, N. Navab, AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images, IEEE transactions on medical imaging 35 (5) (2016) 1313–1321.
  • (21) T. Brosch, L. Y. Tang, Y. Yoo, D. K. Li, A. Traboulsee, R. Tam, Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation, IEEE transactions on medical imaging 35 (5) (2016) 1229–1239.
  • (22) S. Pereira, A. Pinto, V. Alves, C. A. Silva, Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images, IEEE transactions on medical imaging 35 (5) (2016) 1240–1251.
  • (23) K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, B. Glocker, Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation, Medical Image Analysis 36 (2017) 61–78.
  • (24) M. Havaei, A. Davy, D. Warde-Farley, A. Biard, A. Courville, Y. Bengio, C. Pal, P.-M. Jodoin, H. Larochelle, Brain tumor segmentation with deep neural networks, Medical image analysis 35 (2017) 18–31.
  • (25) P. Moeskops, M. A. Viergever, A. M. Mendrik, L. S. de Vries, M. J. Benders, I. Išgum, Automatic segmentation of MR brain images with a convolutional neural network, IEEE transactions on medical imaging 35 (5) (2016) 1252–1261.
  • (26) W. Zhang, R. Li, H. Deng, L. Wang, W. Lin, S. Ji, D. Shen, Deep convolutional neural networks for multi-modality isointense infant brain image segmentation, Neuroimage 108 (2015) 214–224.
  • (27) H. Chen, Q. Dou, L. Yu, J. Qin, P.-A. Heng, VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images, NeuroImage (2017) In press.
  • (28) J. Dolz, C. Desrosiers, I. B. Ayed, 3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study, Neuroimage (2017) In press.
  • (29) C. Wachinger, M. Reuter, T. Klein, DeepNAT: Deep convolutional neural network for segmenting neuroanatomy, Neuroimage (2017) In press.
  • (30) X. Lladó, O. Ganiler, A. Oliver, R. Martí, J. Freixenet, L. Valls, J. C. Vilanova, L. Ramió-Torrentà, À. Rovira, Automated detection of multiple sclerosis lesions in serial brain MRI, Neuroradiology 54 (8) (2012) 787–807.
  • (31) X. Lladó, A. Oliver, M. Cabezas, J. Freixenet, J. C. Vilanova, A. Quiles, L. Valls, L. Ramió-Torrentà, À. Rovira, Segmentation of multiple sclerosis lesions in brain MRI: a review of automated approaches, Information Sciences 186 (1) (2012) 164–185.
  • (32) S. González-Villà, A. Oliver, S. Valverde, L. Wang, R. Zwiggelaar, X. Lladó, A review on brain structures segmentation in magnetic resonance imaging, Artificial Intelligence in Medicine 73 (2016) 45–69.
  • (33)

    J. Mitra, P. Bourgeat, J. Fripp, S. Ghose, S. Rose, O. Salvado, A. Connelly, B. Campbell, S. Palmer, G. Sharma, et al., Lesion segmentation from multimodal MRI using random forest following ischemic stroke, Neuroimage 98 (2014) 324–335.

  • (34) E. Roura, A. Oliver, M. Cabezas, S. Valverde, D. Pareto, J. C. Vilanova, L. Ramió-Torrentà, À. Rovira, X. Lladó, A toolbox for multiple sclerosis lesion segmentation, Neuroradiology 57 (10) (2015) 1031–1043.
  • (35) K.-k. Shen, J. Fripp, F. Mériaudeau, G. Chételat, O. Salvado, P. Bourgeat, A. D. N. Initiative, et al., Detecting global and local hippocampal shape changes in Alzheimer’s disease using statistical shape models, Neuroimage 59 (3) (2012) 2155–2166.
  • (36) Y. Bengio, Deep learning of representations: Looking forward, in: International Conference on Statistical Language and Speech Processing, Springer, 2013, pp. 1–37.
  • (37) L. Deng, D. Yu, Deep Learning, Signal Processing 7 (2014) 3–4.
  • (38) J. Schmidhuber, Deep learning in neural networks: An overview, Neural Networks 61 (2015) 85–117.
  • (39) Y. Bengio, A. Courville, P. Vincent, Representation learning: A review and new perspectives, IEEE transactions on pattern analysis and machine intelligence 35 (8) (2013) 1798–1828.
  • (40) Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015) 436–444.
  • (41) L. Deng, Deep learning: from speech recognition to language and multimodal processing, APSIPA Transactions on Signal and Information Processing 5 (2016) e1.
  • (42) Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu, M. S. Lew, Deep learning for visual understanding: A review, Neurocomputing 187 (2016) 27–48.
  • (43) C. Angermueller, T. Pärnamaa, L. Parts, O. Stegle, Deep learning for computational biology, Molecular Systems Biology 12 (7) (2016) 878.
  • (44) P. Mamoshina, A. Vieira, E. Putin, A. Zhavoronkov, Applications of deep learning in biomedicine, Molecular pharmaceutics 13 (5) (2016) 1445–1454.
  • (45) D. Shen, G. Wu, H.-I. Suk, Deep learning in medical image analysis, Annual review of biomedical engineering 19 (2017) 221–248.
  • (46) G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, C. I. Sánchez, A survey on deep learning in medical image analysis, Medical image analysis 42 (2017) 60–88.
  • (47) Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.
  • (48) Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, L. D. Jackel, Handwritten digit recognition with a back-propagation network, in: Advances in neural information processing systems, 1990, pp. 396–404.
  • (49) Y. Bengio, P. Simard, P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE Transactions on Neural Networks 5 (2) (1994) 157–166.
  • (50) G. E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural computation 18 (7) (2006) 1527–1554.
  • (51) X. Glorot, A. Bordes, Y. Bengio, Deep Sparse Rectifier Neural Networks., in: Aistats, Vol. 15, 2011, p. 275.
  • (52) C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.
  • (53) C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818–2826.
  • (54) V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in: Proceedings of the 27th International Conference on Machine Learning (ICML-10), 2010, pp. 807–814.
  • (55) K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1026–1034.
  • (56) A. L. Maas, A. Y. Hannun, A. Y. Ng, Rectifier nonlinearities improve neural network acoustic models, in: Proc. International conference on machine learning , Vol. 30, 2013.
  • (57) B. Xu, N. Wang, T. Chen, M. Li, Empirical evaluation of rectified activations in convolutional network, CoRR abs/1505.00853,
  • (58) X. Jin, C. Xu, J. Feng, Y. Wei, J. Xiong, S. Yan, Deep Learning with S-Shaped Rectified Linear Activation Units, in: Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  • (59)

    Y. LeCun, C. Cortes, C. J. Burges, The MNIST database of handwritten digits (1998).

  • (60) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE, 2009, pp. 248–255.
  • (61) I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, Y. Bengio, Maxout networks., ICML (3) 28 (2013) 1319–1327.
  • (62) J. T. Springenberg, M. A. Riedmiller, Improving deep neural networks with probabilistic maxout units, CoRR abs/1312.6116,
  • (63) H. Chen, L. Yu, Q. Dou, L. Shi, V. C. Mok, P. A. Heng, Automatic detection of cerebral microbleeds via deep learning based 3D feature representation, in: 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), IEEE, 2015, pp. 764–767.
  • (64) L. Trottier, P. Giguére, B. Chaib-draa, Parametric exponential linear unit for deep convolutional neural networks, CoRR abs/1605.09332,
  • (65) I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016.
  • (66) Y.-L. Boureau, F. Bach, Y. LeCun, J. Ponce, Learning mid-level features for recognition, in: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 2010, pp. 2559–2566.
  • (67) Y.-L. Boureau, J. Ponce, Y. LeCun, A theoretical analysis of feature pooling in visual recognition, in: Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 111–118.
  • (68) M. D. Zeiler, R. Fergus, Stochastic pooling for regularization of deep convolutional neural networks, CoRR abs/1301.3557,
  • (69) Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, Reading digits in natural images with unsupervised feature learning, in: NIPS workshop on deep learning and unsupervised feature learning, Vol. 2011, 2011, p. 5.
  • (70) K. He, X. Zhang, S. Ren, J. Sun, Spatial pyramid pooling in deep convolutional networks for visual recognition, in: European Conference on Computer Vision, Springer, 2014, pp. 346–361.
  • (71) W. Ouyang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, S. Yang, Z. Wang, C.-C. Loy, et al., Deepid-net: Deformable deep convolutional neural networks for object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2403–2412.
  • (72) C.-Y. Lee, P. W. Gallagher, Z. Tu, Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree, in: International Conference on Artificial Intelligence and Statistics, 2016.
  • (73) R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587.
  • (74) J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.
  • (75) Y. Tang, Deep learning using support vector machines, CoRR abs/1306.0239,
  • (76) W. Liu, E. CN, Y. Wen, M. SCUT, Z. Yu, M. Yang, Large-Margin Softmax Loss for Convolutional Neural Networks, in: Proceedings of The 33rd International Conference on Machine Learning, 2016, pp. 507–516.
  • (77) R. Cruz, K. Fernandes, J. S. Cardoso, J. F. P. Costa, Tackling class imbalance with ranking, in: Neural Networks (IJCNN), 2016 International Joint Conference on, IEEE, 2016, pp. 2182–2187.
  • (78) T. Brosch, Y. Yoo, L. Y. Tang, D. K. Li, A. Traboulsee, R. Tam, Deep convolutional encoder networks for multiple sclerosis lesion segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 3–11.
  • (79) F. Milletari, N. Navab, S.-A. Ahmadi, V-net: Fully convolutional neural networks for volumetric medical image segmentation, in: 3D Vision (3DV), 2016 Fourth International Conference on, IEEE, 2016, pp. 565–571.
  • (80) Y. Sun, Y. Chen, X. Wang, X. Tang, Deep learning face representation by joint identification-verification, in: Advances in Neural Information Processing Systems, 2014, pp. 1988–1996.
  • (81) F. Schroff, D. Kalenichenko, J. Philbin, Facenet: A unified embedding for face recognition and clustering, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 815–823.
  • (82) S. Wager, S. Wang, P. S. Liang, Dropout training as adaptive regularization, in: Advances in neural information processing systems, 2013, pp. 351–359.
  • (83) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15 (2014) 1929–1958.
  • (84) S. I. Wang, C. D. Manning, Fast dropout training., in: ICML (2), 2013, pp. 118–126.
  • (85) L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, R. Fergus, Regularization of neural networks using dropconnect, in: Proceedings of the 30th International Conference on Machine Learning (ICML-13), 2013, pp. 1058–1066.
  • (86) A. G. Howard, Some improvements on deep convolutional neural network based image classification, CoRR abs/1312.5402,
  • (87) K. Chatfield, K. Simonyan, A. Vedaldi, A. Zisserman, Return of the devil in the details: Delving deep into convolutional nets, CoRR abs/1405.3531,
  • (88)

    Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel, Backpropagation applied to handwritten zip code recognition, Neural computation 1 (4) (1989) 541–551.

  • (89) R. S. Sutton, Two problems with backpropagation and other steepest-descent learning procedures for networks, in: Proc. 8th annual conf. cognitive science society, 1986, pp. 823–831.
  • (90) Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, Y. Bengio, Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, in: Advances in neural information processing systems, 2014, pp. 2933–2941.
  • (91) Y. Nesterov, A method for unconstrained convex minimization problem with the rate of convergence O (1/k2), in: Doklady an SSSR, Vol. 269, 1983, pp. 543–547.
  • (92) I. Sutskever, J. Martens, G. E. Dahl, G. E. Hinton, On the importance of initialization and momentum in deep learning., ICML (3) 28 (2013) 1139–1147.
  • (93)

    R. Ge, F. Huang, C. Jin, Y. Yuan, Escaping from saddle points—online stochastic gradient for tensor decomposition, in: Proceedings of The 28th Conference on Learning Theory, 2015, pp. 797–842.

  • (94) A. Anandkumar, R. Ge, Efficient approaches for escaping higher order saddle points in non-convex optimization, in: Conference on Learning Theory, 2016, pp. 81–102.
  • (95)

    J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, Journal of Machine Learning Research 12 (Jul) (2011) 2121–2159.

  • (96) M. D. Zeiler, ADADELTA: an adaptive learning rate method, CoRR abs/1212.5701,
  • (97) T. Tieleman, G. Hinton, Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural Networks for Machine Learning 4 (2).
  • (98) D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, CoRR abs/1412.6980,
  • (99) X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks., in: Aistats, Vol. 9, 2010, pp. 249–256.
  • (100) M. Lin, Q. Chen, S. Yan, Network in network, CoRR abs/1312.4400,
  • (101) T. Salimans, D. P. Kingma, Weight normalization: A simple reparameterization to accelerate training of deep neural networks, in: Advances in Neural Information Processing Systems, 2016, pp. 901–901.
  • (102) J. Acosta-Cabronero, G. B. Williams, J. M. S. Pereira, G. Pengas, P. J. Nestor, The impact of skull-stripping and radio-frequency bias correction on grey-matter segmentation for voxel-based morphometry, Neuroimage 39 (4) (2008) 1654–1665.
  • (103) V. Popescu, M. Battaglini, W. S. Hoogstrate, S. C. J. Verfaillie, I. C. Sluimer, R. A. van Schijndel, B. W. van Dijk, K. S. Cover, D. L. Knol, M. Jenkinson, F. Barkhof, N. de Stefano, H. Vrenken, Optimizing parameter choice for FSL-Brain Extraction Tool (BET) on 3D T1 images in multiple sclerosis, Neuroimage 61 (4) (2012) 1484–1494.
  • (104) S. A. Sadananthan, W. Zheng, M. W. L. Chee, V. Zagorodnov, Skull stripping using graph cuts, Neuroimage 49 (1) (2010) 225–239.
  • (105) J.-M. Lee, U. Yoon, S. H. Nam, J.-H. Kim, I.-Y. Kim, S. I. Kim, Evaluation of automated and semi-automated skull-stripping algorithms using similarity index and segmentation error, Computers in biology and medicine 33 (6) (2003) 495–507.
  • (106) E. Roura, A. Oliver, M. Cabezas, J. C. Vilanova, À. Rovira, L. Ramió-Torrentà, X. Lladó, MARGA: multispectral adaptive region growing algorithm for brain extraction on axial MRI, Computer methods and programs in biomedicine 113 (2) (2014) 655–673.
  • (107) S. M. Smith, Fast robust automated brain extraction, Human brain mapping 17 (3) (2002) 143–155.
  • (108) M. Jenkinson, M. Pechaud, S. Smith, BET2: MR-based estimation of brain, skull and scalp surfaces, in: Eleventh Annual Meeting of the Organization for Human Brain Mapping, 2005.
  • (109) D. W. Shattuck, S. R. Sandor-Leahy, K. A. Schaper, D. A. Rottenberg, R. M. Leahy, Magnetic resonance image tissue classification using a partial volume model, Neuroimage 13 (5) (2001) 856–876.
  • (110) J. E. Iglesias, C.-Y. Liu, P. M. Thompson, Z. Tu, Robust brain extraction across datasets and comparison with publicly available methods, IEEE transactions on medical imaging 30 (9) (2011) 1617–1634.
  • (111) S. F. Eskildsen, P. Coupé, V. Fonov, J. V. Manjón, K. K. Leung, N. Guizard, S. N. Wassef, L. R. Østergaard, D. L. Collins, BEaST: Brain extraction based on nonlocal segmentation technique, Neuroimage 59 (3) (2012) 2362 – 2373.
  • (112) Y. Yoo, T. Brosch, A. Traboulsee, D. K. Li, R. Tam, Deep learning of image features from unlabeled data for multiple sclerosis lesion segmentation, in: International Workshop on Machine Learning in Medical Imaging, Springer, 2014, pp. 117–124.
  • (113) F. Shi, L. Wang, Y. Dai, J. H. Gilmore, W. Lin, D. Shen, LABEL: pediatric brain extraction using learning-based meta-algorithm, Neuroimage 62 (3) (2012) 1975–1986.
  • (114) I. Despotović, B. Goossens, W. Philips, MRI segmentation of the human brain: challenges, methods, and applications, Computational and mathematical methods in medicine 2015.
  • (115) A. Simmons, P. S. Tofts, G. J. Barker, S. R. Arridge, Sources of intensity nonuniformity in spin echo images at 1.5 T, Magnetic Resonance in Medicine 32 (1) (1994) 121–128.
  • (116) J. Juntu, J. Sijbers, D. Van Dyck, J. Gielen, Bias field correction for MRI images, in: Computer Recognition Systems, Springer, 2005, pp. 543–551.
  • (117) J. B. Arnold, J.-S. Liow, K. A. Schaper, J. J. Stern, J. G. Sled, D. W. Shattuck, A. J. Worth, M. S. Cohen, R. M. Leahy, J. C. Mazziotta, D. A. Rottenberg, Qualitative and quantitative evaluation of six algorithms for correcting intensity nonuniformity effects, Neuroimage 13 (5) (2001) 931–943.
  • (118) Y. Ge, R. I. Grossman, J. S. Babb, M. L. Rabin, L. J. Mannon, D. L. Kolson, Age-related total gray matter and white matter changes in normal adult brain. Part I: volumetric MR imaging analysis, American journal of neuroradiology 23 (8) (2002) 1327–1333.
  • (119) J. G. Sled, A. P. Zijdenbos, A. C. Evans, A nonparametric method for automatic correction of intensity nonuniformity in MRI data, IEEE transactions on medical imaging 17 (1) (1998) 87–97.
  • (120) N. J. Tustison, B. B. Avants, P. A. Cook, Y. Zheng, A. Egan, P. A. Yushkevich, J. C. Gee, N4ITK: improved N3 bias correction, IEEE transactions on medical imaging 29 (6) (2010) 1310–1320.
  • (121) M. Lyksborg, O. Puonti, M. Agn, R. Larsen, An ensemble of 2D convolutional neural networks for tumor segmentation, in: Scandinavian Conference on Image Analysis, Springer, 2015, pp. 201–211.
  • (122) D. Zikic, Y. Ioannou, M. Brown, A. Criminisi, Segmentation of brain tumor tissues with convolutional neural networks, Proceedings MICCAI-BRATS (2014) 36–39.
  • (123) A. Birenbaum, H. Greenspan, Longitudinal multiple sclerosis lesion segmentation using multi-view convolutional neural networks, in: International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, Springer, 2016, pp. 58–67.
  • (124) G. Urban, M. Bendszus, F. Hamprecht, J. Kleesiek, Multi-modal brain tumor segmentation using deep convolutional neural networks, MICCAI BraTS (Brain Tumor Segmentation) Challenge. Proceedings, winning contribution (2014) 31–35.
  • (125)

    K. Vaidhya, S. Thirunavukkarasu, V. Alex, G. Krishnamurthi, Multi-modal Brain Tumor Segmentation Using Stacked Denoising Autoencoders, in: International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Springer, 2015, pp. 181–194.

  • (126) S. Valverde, M. Cabezas, E. Roura, S. González-Villà, D. Pareto, J. C. Vilanova, L. Ramió-Torrentà, À. Rovira, A. Oliver, X. Lladó, Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach, NeuroImage 155 (2017) 159–168.
  • (127) L. G. Nyúl, J. K. Udupa, X. Zhang, New variants of a method of MRI scale standardization, IEEE transactions on medical imaging 19 (2) (2000) 143–150.
  • (128) S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz, T. Greer, B. ter Haar Romeny, J. B. Zimmerman, K. Zuiderveld, Adaptive histogram equalization and its variations, Computer vision, graphics, and image processing 39 (3) (1987) 355–368.
  • (129) M. F. Stollenga, W. Byeon, M. Liwicki, J. Schmidhuber, Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation, in: Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS’15, MIT Press, Cambridge, MA, USA, 2015, pp. 2998–3006.
  • (130) P. Dvorak, B. Menze, Structured prediction with convolutional neural networks for multimodal brain tumor segmentation, Proceeding of the Multimodal Brain Tumor Image Segmentation Challenge (2015) 13–24.
  • (131) Y. Bengio, Practical recommendations for gradient-based training of deep architectures, in: Neural Networks: Tricks of the Trade, Springer, 2012, pp. 437–478.
  • (132) M. Lai, Deep learning for medical image segmentation, CoRR abs/1505.02000,
  • (133) F. Milletari, S.-A. Ahmadi, C. Kroll, A. Plate, V. Rozanski, J. Maiostre, J. Levin, O. Dietrich, B. Ertl-Wagner, K. Bötzel, N. Navab, Hough-CNN: Deep learning for segmentation of deep brain regions in {MRI} and ultrasound, Computer Vision and Image Understanding (2017) –doi:
  • (134) W. Li, G. Wang, L. Fidon, S. Ourselin, M. J. Cardoso, T. Vercauteren, On the compactness, efficiency, and representation of 3d convolutional networks: brain parcellation as a pretext task, in: International Conference on Information Processing in Medical Imaging, Springer, 2017, pp. 348–360.
  • (135) A. de Brebisson, G. Montana, Deep neural networks for anatomical brain segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015, pp. 20–28.
  • (136) L. Zhao, K. Jia, Multiscale CNNs for Brain Tumor Segmentation and Diagnosis, Computational and mathematical methods in medicine 2016.
  • (137) B. Likar, M. A. Viergever, F. Pernus, Retrospective correction of MR intensity inhomogeneity by information minimization, IEEE transactions on medical imaging 20 (12) (2001) 1398–1410.
  • (138) M. Shakeri, S. Tsogkas, E. Ferrante, S. Lippe, S. Kadoury, N. Paragios, I. Kokkinos, Sub-cortical brain structure segmentation using F-CNN’s, in: Biomedical Imaging (ISBI), 2016 IEEE 13th International Symposium on, IEEE, 2016, pp. 269–272.
  • (139) R. Mehta, A. Majumdar, J. Sivaswamy, BrainSegNet: a convolutional neural network architecture for automated segmentation of human brain structures, Journal of Medical Imaging 4 (2) (2017) 024003–024003.
  • (140) P. Krähenbühl, V. Koltun, Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials, in: Advances in Neural Information Processing Systems, 2011, pp. 109–117.
  • (141) K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, CoRR abs/1409.1556,
  • (142) B. Hariharan, P. Arbeláez, R. Girshick, J. Malik, Hypercolumns for object segmentation and fine-grained localization, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 447–456.
  • (143) O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 234–241.
  • (144) I. Arganda-Carreras, S. C. Turaga, D. R. Berger, D. Cireşan, A. Giusti, L. M. Gambardella, J. Schmidhuber, D. Laptev, S. Dwivedi, J. M. Buhmann, T. Liu, M. Seyedhosseini, T. Tasdizen, L. Kamentsky, R. Burget, V. Uher, X. Tan, C. Sun, T. D. Pham, E. Bas, M. G. Uzunbas, A. Cardona, J. Schindelin, H. S. Seung, Crowdsourcing the creation of image segmentation algorithms for connectomics, Frontiers in Neuroanatomy 9 (2015) 142. doi:10.3389/fnana.2015.00142.
  • (145) L. Yu, X. Yang, H. Chen, J. Qin, P. Heng, Volumetric convnets with mixed residual connections for automated prostate segmentation from 3d MR images, in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA., 2017, pp. 66–72.
  • (146) Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, O. Ronneberger, 3D u-net: learning dense volumetric segmentation from sparse annotation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2016, pp. 424–432.
  • (147) P. F. Christ, M. E. A. Elshaer, F. Ettlinger, S. Tatavarty, M. Bickel, P. Bilic, M. Rempfler, M. Armbruster, F. Hofmann, M. D’Anastasi, et al., Automatic liver and lesion segmentation in ct using cascaded fully convolutiochen2016voxresnetnal neural networks and 3d conditional random fields, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2016, pp. 415–423.
  • (148) H. Lombaert, J. Sporring, K. Siddiqi, Diffeomorphic spectral matching of cortical surfaces, in: IPMI, Vol. 7917, 2013, pp. 376–389.
  • (149) C. Wachinger, M. Brennan, G. Sharp, P. Golland, On the importance of location and features for the patch-based segmentation of parotid glands, in: MICCAI Workshop on Image-Guided Adaptive Radiation Therapy, 2014.
  • (150) C. Wachinger, M. Brennan, G. Sharp, P. Golland, Efficient Descriptor-Based Segmentation of Parotid Glands with Non-Local Means, IEEE Transactions on Biomedical Engineering PP (99) (2016) 1–1.
  • (151) V. Vezhnevets, V. Konouchine, GrowCut: Interactive multi-label ND image segmentation by cellular automata, in: proc. of Graphicon, Vol. 1, Citeseer, 2005, pp. 150–156.
  • (152) M. Styner, J. Lee, B. Chin, M. Chin, O. Commowick, H. Tran, S. Markovic-Plese, V. Jewells, S. Warfield, 3D segmentation in the clinic: A grand challenge II: MS lesion segmentation, Midas Journal 2008 (2008) 1–6.
  • (153) I. Išgum, M. J. Benders, B. Avants, M. J. Cardoso, S. J. Counsell, E. F. Gomez, L. Gui, P. S. Hűppi, K. J. Kersbergen, A. Makropoulos, et al., Evaluation of automatic neonatal brain segmentation algorithms: the NeoBrainS12 challenge, Medical image analysis 20 (1) (2015) 135–151.
  • (154) A. M. Mendrik, K. L. Vincken, H. J. Kuijf, M. Breeuwer, W. H. Bouvy, J. De Bresser, A. Alansary, M. De Bruijne, A. Carass, A. El-Baz, et al., MRBrainS challenge: online evaluation framework for brain image segmentation in 3T MRI scans, Computational intelligence and neuroscience 2015 (2015) 1.
  • (155) B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al., The multimodal brain tumor image segmentation benchmark (BRATS), IEEE Transactions on Medical Imaging 34 (10) (2015) 1993–2024.
  • (156) A. Carass, S. Roy, A. Jog, J. L. Cuzzocreo, E. Magrath, A. Gherman, J. Button, J. Nguyen, F. Prados, C. H. Sudre, et al., Longitudinal multiple sclerosis lesion segmentation: Resource and challenge, NeuroImage 148 (2017) 77–102.
  • (157) O. Maier, B. H. Menze, J. von der Gablentz, L. Häni, M. P. Heinrich, M. Liebrand, S. Winzeck, A. Basit, P. Bentley, L. Chen, et al., ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI, Medical image analysis 35 (2017) 250–269.
  • (158) O. Commowick, F. Cervenansky, R. Ameli, MSSEG Challenge Proceedings: Multiple Sclerosis Lesions Segmentation Challenge Using a Data Management and Processing Infrastructure (2016).
  • (159) C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, M. J. Cardoso, Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations, in: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, 2017, pp. 240–248.
  • (160) L. Fidon, W. Li, L. C. Garcia-Peraza-Herrera, J. Ekanayake, N. Kitchen, S. Ourselin, T. Vercauteren, Generalised wasserstein dice score for imbalanced multi-class segmentation usingholistic convolutional networks, in: A. Crimi, S. Bakas, H. Kuijf, B. Menze, M. Reyes (Eds.), Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Springer International Publishing, Cham, 2018, pp. 64–76.
  • (161)

    Y. Zhang, M. Brady, S. Smith, Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm, IEEE Transactions on Medical Imaging 20 (1) (2001) 45–57.

  • (162) J. Ashburner, K. J. Friston, Unified segmentation, NeuroImage 26 (3) (2005) 839 – 851.
  • (163) J. Ashburner, G. Barnes, C. Chen, SPM8 Manual,, [Online; accessed 18 May 2017] (2012).
  • (164) S. Valverde, A. Oliver, E. Roura, S. González-Villà, D. Pareto, J. C. Vilanova, L. Ramió-Torrentà, À. Rovira, X. Lladó, Automated tissue segmentation of MR brain images in the presence of white matter lesions, Medical Image Analysis 35 (2017) 446–457.
  • (165) A. Lavin, S. Gray, Fast algorithms for convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4013–4021.
  • (166) T. Paine, H. Jin, J. Yang, Z. Lin, T. S. Huang, GPU asynchronous stochastic gradient descent to speed up neural network training, CoRR abs/1312.6186,