Medical image segmentation is often part of medical image analysis pipelines, being crucial for diagnostic, treatment planning, and follow-up [18, 22, 21]. However, when manually done, segmentation is time demanding and prone to inter- and intra-rater variability [22, 21]. Automatic segmentation can mitigate those issues, thus improving the efficiency and quality of medical care, allowing the experts to focus on other tasks 
. In recent years, several approaches have been based on machine learning methods. Among them, many rely on Deep Learning using Convolutional Neural Networks (CNNs).
Convolutional Neural Networks are composed by convolutional layers that learn features directly from the data. Each of these layers consists of a set of learnable kernels that are convolved over the input data to generate a stack of feature maps. These models gathered a lot of interest after the results achieved by Krizhevsky et al.  in object recognition. Since then, CNN-based approaches have reached state-of-the-art results across many applications in medical image analysis [28, 32, 14]. In recent years, active research is being conducted regarding architectural design in deep neural networks. Several studies focused on enabling training of deeper networks, such as VGGNet  that uses small kernels to reduce the number of parameters, or residual learning, elegantly implemented through short skip connections . However, other dimensions of CNNs, besides depth, are being studied, such as the cardinality , or the cross-channels relationships in a stack of feature maps .
The relationships among the channels may be used to enhance the representational power of the features. For instance, convolutional layers with kernels were used as parametric pooling , or as bottlenecks in ResNets . Interestingly, in the latter, these layers were also used to increase the number of feature maps after the previous bottleneck. Since layers with kernels do not take a neighborhood into account, they work solely by recombining the channels of the feature maps. Differently, Hu et al.  explored the relationships among channels and proposed recalibration of feature maps with Squeeze-and-Excitation (SE) blocks. In this case, the end goal is to adaptively suppress the less relevant feature maps.
Many advances in CNN‘s design for image applications are often proposed in the context of object recognition . In this problem, a single class label is inferred for the whole image. Similar approaches can be used for segmentation. For instance, Pereira et al. 
used a conventional CNN-based method that classifies the central voxel of a patch. So, in this case, it is straightforward to introduce general CNN blocks. However, recently, the more efficient Fully Convolutional Networks (FCNs)[32, 19] are being preferred for image segmentation over conventional CNNs. In these architectures, the fully-connected layers of CNNs are replaced by convolutional layers. In this way, it is possible to segment a full patch of voxels in just one forward pass. While in conventional CNNs the whole feature maps characterize the class of just one voxel, in FCNs there is a direct correspondence between the features in a given location of the feature maps and the voxel being classified in the same spatial location. Therefore, some architectural designs may not be well suited for FCNs, such as the SE block. The reason is that this block weights whole feature maps, while we may be interested in suppressing regions of the feature maps that are irrelevant when predicting voxels in the same spatial location.
Besides learning channel-wise relationships to enhance the discriminative power of the feature maps, the SE block  may be interpreted as a channel-wise attention mechanism . Different kinds of attention have been recently proposed in the context of medical image segmentation using FCNs. Qin et al.  proposed a scale attention scheme that adaptively chooses the receptive field. This is achieved by processing several parallel branches with convolutional layers with different dilation rates. However, this increases the memory and computational requirements. Oktay et al.  utilized attention as a soft mask to enhance the region of interest. However, this approach scales the same regions across all feature maps. Roy et al.  learn attention at the channel and spatial levels, but a single spatial attention map is inferred for all feature maps.
I-a Motivation and Contributions
The cross-channel relationships among feature maps were shown to encode relevant information . Therefore, in this work, we study the recombination and recalibration of feature maps. Regarding recombination, convolutional layers with kernels were previously used to decrease the number of feature maps [17, 7], but also to increase their number afterwards  to boost their representational power. We propose recombination, where we linearly expand the number of feature maps before compressing again, allowing the network to learn how to mix the information to generate more discriminative features. In the case of recalibration, the SE block has the desirable property of learning cross-channel relationships and suppressing the less relevant feature maps. However, it recalibrates whole feature maps, which makes it not well adapted for semantic segmentation with FCNs where it may be better to have spatially adaptive recalibration. So, in this work we also propose a recalibration block (the SegSE block) that is able to spatially recalibrate feature maps, while still considering the cross-channel relationships. This can be alternatively interpreted as a spatially adapted attention mechanism for each feature map; unlike , where a single attention map is generated. Related to our work is the block proposed by Roy et al. . But, channel and spatial recalibration are considered separately, while we learn them jointly. Also, a single spatial recalibration map is estimated for all channels without taking context into account, while we infer channel-specific maps.
An early version of this work was presented at a conference . This paper extends the previous work with further validation across several datasets and applications. Additionally, we provide more detailed descriptions on the method and discussion, and insights on the working process of the proposed approach. We observe that it indeed learns how to suppress or enhance features accordingly to the structures being segmented. The contributions in this paper can be summarized as follows. 1) We propose feature recombination by means of linear expansion and compression of the number of feature maps. 2) We propose a novel feature recalibration approach for FCNs – the SegSE block. This block may be interpreted as an attention mechanism for each channel of the stack of feature maps. 3) We validate our methods in several publicly available datasets for brain tumor segmentation, stroke penumbra estimation, and ischemic stroke lesion outcome prediction. Finally, 4) we inspect the attention maps to verify that they enhance specific features, while the original SE block suppresses whole feature maps, including potentially important local features.
In this work, we approach semantic segmentation using 2D FCNs operating over patches. The architecture of our network is inspired in U-Net , as depicted in Fig. 1 (note that rectangles represent a layer or set of layers). This type of FCN is well-established in medical image segmentation tasks . U-Net belongs to a broader class of architectures called encoder-decoder. As input we have an image patch, which may have several stacked channels. The encoder path encompasses different levels of abstraction, which are responsible for learning higher order features. Features computed by higher (deeper) convolutional layers are more abstract. However, these features may lack the fine details that are important for segmentation, which are better captured by the lower layers. Since the feature maps are down-sampled, we need to map the lower resolution feature maps back to the input patch resolution. This is gradually done by up-sampling in the decoder path. As we up-sample feature maps, we sum them with the feature maps of equivalent size of lower layers of the encoder path, through long skip connections. Further convolutional layers fuse the lower and higher level features. Finally, the last layer is a convolutional layer with32].
represent Block 1 and Block 2, respectively. Input sizes correspond to the RR block with SegSE. Down-sampling is obtained by max-pooling. We use nearest neighbor up-sampling to increase the feature maps size, and
convolutional layers to adjust the number of feature maps, before addition. b) Recombination block. c) RR block, the SE block, the SegSE blocks, and the other variants. BN stands for batch normalization, and Sp. Drop. for spatial dropout.
We may think of a layer in a FCN as implementing a transformation function () that maps the input feature maps into some output feature maps . Therefore, it can be defined as , with and , where and represent the height and width of the transformed feature maps, respectively. and represent the same measures but in relation to the input feature maps . and are the number of feature maps, such that , and . So, a convolutional layer defines a function , such that , where , , and represent the kernel size, the dilation rate, and the number of kernels, respectively. In this case, each output channel is computed as , where is the set of learnable kernels (the bias term is not represented for the sake of simplicity), and is the convolution operation symbol.
In recombination we are interested in increasing the representational power of the features by mixing them linearly. To this end, we employ convolutional layers with kernels. First, we expand the number of features maps (), before compressing again to the original number (). So, recombination is defined as , with . This translates into the following operation:
where is the expansion factor. These operations are depicted in Fig. 1.
Recalibration consists in learning the relationship among the feature maps of a layer, and suppressing the less relevant features. Originally, Hu et al.  proposed to recalibrate whole feature maps, which we refer as channel SE blocks (c.f. Fig. 2). However, in this work, we propose spatially adaptive recalibration (c.f. Fig. 2).
Ii-B1 Channels Squeeze-and-Excitation
Let us consider a function , with , that transforms the input feature maps into their recalibrated form. This is achieved by two sequential operations: squeeze and excitation 
. The spatial squeeze operation consists in summarizing each channel into a scalar descriptor, such that we obtain a descriptor vector, with . This operation can be performed by global average pooling, where a given channel is squeezed as,
In the excitation operation, the network learns the dependencies among the channels in order to adaptively estimate the excitation or scaling factors . This can be understood as a gating mechanism that provides channel-wise attention. To this end, two fully-connected layers111Equivalently implemented as convolutional layers with kernels. are employed, where the first one acts as bottleneck and the second one restores the dimension of the vector. These operations are described as,
with , and .
denotes the ReLU activation function,
denotes the sigmoid function, anddenotes the reduction factor. Therefore, we can define the function . These operations are visually described in the SE block of Fig. 1.
Finally, the recalibration, or scaling, of the feature maps is done by simple multiplication of the original channels by the corresponding recalibration factor. So, it may be defined as . In this way, the recalibration of a feature map is defined as,
A visual depiction of channel SE can be found in Fig. 2.
Ii-B2 Spatially adaptive Squeeze-and-Excitation
In semantic segmentation with FCNs there is a spatial correspondence between the units in the features maps and the pixels/voxels being segmented in the same locations. Hence, a given feature map may have relevant features for some voxels, whereas in other locations it may not be so important. Therefore, due to its global squeeze operation, channel SE blocks may end up suppressing whole feature maps that may contain important regions. For this reason, we argue that spatially adaptive SE blocks, as depicted in Fig. 2, are more appropriate for semantic segmentation, at least with FCNs.
In this subsection, we start by presenting our simultaneous spatial and channel Squeeze-and-Excitation proposal. Next, we describe two variants that were studied in this work.
Spatially adaptive Squeeze-and-Excitation for segmentation — SegSE
The proposed block can be observed in Fig 1 — SegSE block. In order to preserve the spatial structure and correspondence of the feature maps, we first replace the squeeze operation through global average pooling in the SE block by a convolutional layer with kernels. The motivation for the squeeze operation is to aggregate contextual information, which in this case is captured by the kernel operating over neighboring voxels. Hence, we employ convolutional layers with dilated kernels  to capture a larger context than simple kernels, but without increasing the number of parameters over those kernels. In this way, instead of having a descriptor vector , as in the case of the channels SE block, we obtain feature maps as,
where , , and represents batch normalization followed by the ReLU activation function. The dilation factor may be chosen according to the scale of the layer it is operating. Layers that already take a large field of view and context into account may require less dilation. In general, the field of view increases, for instance, with the number of convolutional, or pooling layers. After these layers, each unit of the feature maps represents a larger region of the input space.
Having , the feature maps with recalibration factors are obtained by a convolutional layer with kernels, followed by the sigmoid activation function, as
with , , and . Therefore, we combine the squeeze and excitation procedures, since the convolutional layer with dilation also includes the bottleneck by decreasing the number of feature maps. Finally, the recalibrated feature maps are obtained by element-wise multiplication () of the input with . So, having a feature map , it is recalibrated as
In this way, our SegSE block may be described by the function .
Variant 1 - No context
A simpler approach consists in removing the global average pooling in the SE block, and replacing the fully-connected layers by convolutional layers with kernels (Fig. 1 — variant 1). Taking the previous description into account, this would translate into setting and . The issue with this approach is that contextual information is not considered, which we previously obtained through convolutional kernels with dilation.
Variant 2 - Pooling-based context
Contextual information may be obtained by average pooling. Contrasting with global pooling, this acts over a kernel instead of the complete channel of the feature maps. This can be implemented through a pooling layer defined as , such that , where
is the size of the pooling kernel and stride, which we assume to be an even number. Therefore, we obtain. Then, a convolutional layer with kernels combines the feature maps and learns their relationship.
Average pooling results in a down-sampled feature map. Hence, it is necessary to restore it to the original shape. We accomplish this through transposed convolutional layers . We employ these layers to double the size of the feature maps each time. So, if , it is necessary to employ more than one layer of transposed convolution. Therefore, the result of this block is , where is the kernel size, is the stride, is the number of convolutional kernels, and is the number of transposed convolution layers. Each operation consists of , with being batch normalization.
Having described these operations, this variant 2 is defined by the following procedures (and in Fig. 1 — variant 2),
where, , , , and .
Finally, recalibration is obtained in a similar way as in equation 7.
Ii-C Recombination and Recalibration
We combine both recombination and recalibration (RR) into the same block, as illustrated in Fig. 1. As in recombination, we first expand the number of feature maps, then we recalibrate them, and, finally, we compress the number of feature maps into the original number. This can be defined as,
Iii Experimental Setup
We evaluate the proposed blocks in three medical applications: brain tumor segmentation, penumbra estimation in acute ischemic stroke, and ischemic stroke lesion outcome prediction. Although all the applications use Magnetic Resonance Imaging (MRI), they are quite diverse in terms of acquisitions. Brain tumor segmentation only takes structural MRI sequences into account. Penumbra estimation has a mixture of low resolution structural, diffusion, and perfusion MRI acquisitions. Finally, in ischemic stroke lesion outcome prediction we deal with perfusion and diffusion acquisitions only. Besides the differences in imaging data, these applications are distinct, too. Brain tumor represents a multi-class classification problem, while the others are binary problems. Additionally, in ischemic stroke lesion outcome prediction we are interested in predicting the status of a stroke lesion three months after the image acquisition and intervention. This differs from a pure segmentation problem, where we segment the objects that are visible at the moment.
The proposed blocks were studied in the brain tumor segmentation task. Then, the best hyperparameters and network architecture were evaluated in the other applications for further validation.
Some hyperparameters were kept constant across experiments. The expansion factor for recombination was set to 4. In the case of recalibration, the dilation factors of our SegSE blocks were defined according to the scale of the feature maps (c.f. Fig. 1) as . Similarly, the kernel sizes and strides of the average pooling of Variant 2 were defined as . Finally, we set the reduction factor to
. These hyperparameters were tuned using the validation set. During training, the cross entropy loss function was minimized using the Adam optimizer with learning rate of . Furthermore, we employed spatial dropout with probability of 0.05, and weight decay of . Artificial data augmentation consisted of random sagittal flipping and random rotations of 222https://github.com/sergiormpereira/rr_segse.
Iii-a Brain tumor segmentation
. BRATS 2017 has two publicly available sets: Training (285 subjects) and Leaderboard (46 subjects). BRATS 2013 encompasses three sets: Training (30 subjects), Leaderboard (25 subjects), and Challenge (10 subjects). For each subject, there are four MRI sequences available: T1-weighted (T1), post-contrast T1-weighted (T1c), T2-weighted (T2), and Fluid-Attenuated Inversion Recovery (FLAIR). All images are already interpolated toisotropic resolution, skull stripped, and aligned. We further pre-processed them by correcting the bias field , and standardizing the intensity histogram of each MRI sequence . Only the Training sets contain manual segmentations publicly available. In BRATS 2017 it distinguishes three tumor regions: edema, necrotic/non-enhancing tumor core, and enhancing tumor. In BRATS 2013 the manual segmentations have necrosis and non-enhancing tumor separately, although we fuse these labels to be similar to BRATS 2017. Hence, the last layer of the FCN has 4 feature maps, three for tumor classes and one for normal tissue. Evaluation is performed for the whole tumor (all regions combined), tumor core (all, excluding edema), and enhancing tumor. Since annotations are not publicly available for 2017 Leaderboard, 2013 Leaderboard, and 2013 Challenge, metrics are computed by the CBICA IPP333https://ipp.cbica.upenn.edu/ and SMIR444https://www.smir.ch/BRATS/Start2013 online platforms. The development of the RR block was conducted in the larger BRATS 2017 Training set, which was randomly divided into training (60%), validation (20%), and test (20%); the identification of the subjects in each set is also available in the online repository. All the hyperparameters were found using the validation set, before evaluation in the test set. Afterwards, the test set was added to the training and the FCN was fine-tuned before evaluation in the Leaderboard set. However, networks tested in BRATS 2013 were trained in the 2013 Training set.
Given the problem of data imbalance between the tumor tissues and normal tissue, and following recent developments in brain tumor segmentation [27, 30, 38], we employ a hierarchical FCN-based brain tumor segmentation approach, as described in . First, we detect the whole tumor as a binary segmentation problem. Then, we use this information to identify the multi-class segmentation object representing the tumor tissues in the region of interest. The binary brain tumor segmentation FCN is 3D and possesses a large field of view. These two properties are important to reduce the number of false positive detections. The architecture is depicted in Fig. 3, and the pipeline for brain tumor segmentation can be found in the Supplementary Material. Note that the 3D binary whole tumor detection network does not contain the RR block and is kept constant across all experiments. Therefore, the variants of the RR block in the more challenging multi-class FCN are the only source of variation in results, allowing us to better compare them and evaluate the benefits of our proposal.
Iii-B Stroke penumbra estimation
In the case of penumbra estimation in acute ischemic stroke, we used the Stroke Perfusion Estimation (SPES) dataset of the MICCAI Ischemic Stroke Lesion Segmentation (ISLES) Challenge . This is a binary classification problem, so, the last layer of the FCN outputs two feature maps. The dataset contains two sets: Training and Challenge. Training has 30 subjects with publicly available annotations. The Challenge set does not have publicly available annotations of its 20 patients, so, evaluation is performed by an online platform555https://www.smir.ch/ISLES/Start2015. Each patient contains 7 sequences: T1c, T2, Diffusion Weighted Imaging (DWI), Cerebral Blood Flow (CBF), Cerebral Blood Volume (CBV), Time-to-Peak (TTP), and Time-to-Max (Tmax). The images are already registered to the T1c, interpolated to a resolution of mm resolution, and skull stripped. Further pre-processing included the bias field correction  and histogram standardization  of the structural sequences, the clipping of the Tmax values over 60 ( s), and linear scaling of all sequences to the intensity range.
Iii-C Ischemic stroke lesion outcome prediction
In this experiment, we used the dataset from ISLES 2017 Challenge . Similarly to penumbra estimation, this is a binary problem. There are two sets available: Training (43 patients), and Challenge (32 patients). While the former has publicly available annotations, in the latter the evaluation is conducted by an online platform666https://www.smir.ch/ISLES/Start2017. For each patient, there are available 5 perfusion maps (CBV, CBF, Mean Transit Time (MTT), TTP, and Tmax), and 1 diffusion map (ADC). The images were already aligned and skull stripped. Further pre-processing was similar to , and consisted in resizing the images to , clipping of the Tmax values to and the ADC values to , and linear scaling to . Finally, and following , we employed the Dice loss function during training.
Iii-D Evaluation metrics
For quantitative evaluation we follow the metrics used in each challenge associated with each dataset. Therefore, for BRATS 2017 Leaderboard we use the Dice Coefficient (DC) and the 95th percentile of the Hausdorff Distance (HD). However, BRATS 2013 employs the DC together with the Sensitivity and Positive Predictive Value (PPV) metrics. In the case of SPES, we report DC and Average Symmetric Surface Distance (ASSD). Regarding ISLES 2017, results are reported in terms of DC, PPV, and Sensitivity. The DC is a measure of overlap, but it is sensitive to the size of the lesions, and does not provide information regarding over- and under-segmentation. Nonetheless, such behavior can be inferred from Sensitivity and PPV. The distance metrics provide insights regarding the correctness of the contour of the segmentations [22, 21, 39].
Iv Results and Discussion
In this section we evaluate the proposed recombination and recalibration blocks, and we observe experimentally the improvements obtained by the SegSE block. Then, we evaluate the baseline model and the best model in the independent sets of BRATS, SPES, and ISLES 2017. Finally, we compare with the state of the art.
Iv-a Evaluation of Recombination and Recalibration
We evaluate the effect of the variants of recombination and recalibration of feature maps in the brain tumor segmentation task. To that end, we use the controlled test set that consists of 20% of the BRATS 2017 Training set. Table I shows the quantitative results, while qualitative results can be found in Fig. 4 as segmentation examples. So, we start from a competitive baseline (as will be shown in Sections IV-B, IV-C, and IV-D) consisting of a FCN similar to the one depicted in Fig. 1, but where the RR block is absent. Then, we incrementally evaluate the proposed blocks. These results can be found in Table I. It is possible to observe that the addition of recombination (Subsection II-A) through linear expansion followed by compression, leads to better DC and Sensitivity in all the tumor regions. This is especially expressive in the core region, where it achieves the highest DC. However, the HD increased, which may be due to over-segmentation, since the sensitivity scores are higher than the ones obtained by the baseline.
|Baseline + Recomb.||0.865||0.769||0.687||0.885||0.832||0.706||0.859||0.768||0.705||9.720||11.453||7.790|
|Baseline + RR SE||0.859||0.756||0.672||0.898||0.793||0.717||0.836||0.785||0.675||8.939||13.306||7.319|
|Baseline + RR Var. 1||0.855||0.753||0.686||0.902||0.816||0.705||0.825||0.766||0.707||8.698||12.177||7.294|
|Baseline + RR Var. 2||0.855||0.757||0.685||0.910||0.855||0.702||0.818||0.742||0.714||8.569||9.819||6.326|
|Baseline + RR SegSE||0.866||0.766||0.698||0.898||0.820||0.718||0.846||0.786||0.718||8.475||10.513||6.131|
In Table I we can also observe the results obtained by recalibration of the feature maps with recombination. Considering DC, the original SE block  yields worse scores in all tumor classes when compared with the recombination alone. Indeed, it manages to improve over the Baseline only in the tumor core region. The enhancing region is the region that suffers the most, since its DC deteriorates to values even lower than the baseline, which is due to a large decrease in sensitivity. Hence, it may be in accordance with the intuition that the SE block may end up suppressing features that are important for thinner and smaller structures, as observed in Fig. 5. This block behaves in this way because it squeezes whole feature maps through their average and recalibrates them as a whole. Since the sensitivity of the enhancing region decreased, we may conclude that it is under-segmenting this region. Moreover, when we observe the results obtained by the different proposed attention mechanisms, DC and Sensitivity of the enhancing region are always better than with the SE block. Note that all the other settings recalibrate the feature maps in a spatially adaptive way. So, we conclude that the SE block, acting as whole feature map recalibration is not adapted for segmentation with FCNs.
Considering the RR block with spatially adaptive recalibration, we may conclude that variants 1 and 2 also achieve worse DC when comparing with the baseline with recombination (Baseline + Recomb.), although in general they perform better in terms of HD. Still, they recover some enhancing tumor that the original SE (Baseline + RR SE) is unable to detect. However, the proposed RR block with SegSE achieves the best results. The DC of the tumor core and the whole tumor are similar to the ones obtained by recombination alone. Nevertheless, we observe an improvement in the enhancing region that achieved the highest score, being the tumor region with the finest details. This was a result of balanced PPV and sensitivity scores that denotes a good quality segmentation, since the HD was simultaneously the lowest among all the evaluated settings. Additionally, it achieved the best HD for the whole tumor, and the second for the core region. Qualitatively examining Fig. 4, the proposed RR block with SegSE appears to result in better segmentations. The reasons why RR with SegSE performs better than RR with variants 1 and 2 may be because variant 1 does not consider any context, while variant 2 suffers from the checkerboard effect introduced by pooling and up-sampling.
In Supplementary Materials it can be found results obtained with 3D versions of the baseline networks and the proposed RR with SegSE. We observe that the proposed blocks also improve the performance in a 3D setting.
A drawback of the SegSE block is that it increases the number of parameters. Therefore, to evaluate if its performance is due to the extra capacity, we proportionally increased the width of the baseline, such that the number of parameters becomes similar to the baseline + RR SegSE network. The results obtained with this larger network can be found in Table I as Wide Baseline. It is possible to observe from DC and HD that it generally performed worse than the proposed RR SegSE block. Indeed, the Wide Baseline was able to achieve better scores only in the whole tumor and tumor core classes in the PPV metric. So, the improvements of the proposed block are due to learning better features, and not directly to the higher number of parameters.
To further inspect the effect of the spatially adaptive recalibration procedure with the SegSE block, we can observe a feature map in Fig. 6. In the left side it is represented a feature map before recalibration, where we can observe a high response for the tumor core, but also substantial response in the surrounding structures. After recalibration, however, we note how the feature map enhances the tumor core, while suppressing most of the surroundings. Hence, we may conclude that the SegSE block is able to selectively and adaptively recalibrate regions of interest in the feature maps, thus acting as an intra-channel attention mechanism.
Iv-B Brain tumor segmentation
Table II presents the results obtained in BRATS 2017 Leaderboard with the baseline FCN, the RR with SE block, and the RR with SegSE block777In the earlier version of this work  the fine-tuning stage described in Subsection III-A was not performed. Hence, the results in the Leaderboard set reported in  were obtained after training the models with 60% of the subjects in BRATS 2017 Training set. This explains the improved results in Table II.. It is possible to observe that the FCN with the proposed RR SegSE block outperforms the baseline in both DC and HD in every tumor regions. This is especially noticeable in the tumor core where the DC improves from 0.758 to 0.798, and the HD decreases from 11.1 mm to 8.947 mm. Additionally, with the RR SegSE block the FCN was able to segment the enhancing tumor better, which is a detailed and difficult region . This can also be observed when comparing with the RR SE block, indicating that the proposed SegSE block is more suited for segmenting detailed regions. Moreover, the proposed SegSE block outperforms the SE block in terms of HD, which suggests that the contours of the obtained segmentations are more detailed and closer to the annotated ones.
It is also possible to compare with the state of the art in BRATS 2017 Leaderboard in Table II, where all the methods are CNN-based. Additionally, we evaluate the block proposed by Roy et al.  in our multi-class FCN. The authors implement an alternative channel and spatial attention mechanism, where a single spatial attention map is inferred for all channels. We evaluate this alternative using the parameters proposed by Roy, but in the same conditions as our RR SegSE block, i.e., after fine-tuning and in a hierarchical approach using the same binary segmentation FCN. We separate single prediction approaches from ensembles888In single prediction approaches the final segmentation results from the predictions of a model, instead of the combination of several predictions, as in ensembles.
. The reason is that ensembles have a competitive advantage, since it is known that it is a way of significantly improving the performance if the models make different mistakes, as it alleviates the effect of the possible high variance of each of its models[5, 13]. In the considered methods, ensembles resulted from training a variety of FCN architectures with different settings , from training a FCN in each of the MRI planes (axial, coronal, and sagittal) [38, 42, 12], or from using several models trained previously for k-fold cross-validation in the Training set .
|Ensemble||Kamnitsas et al. ||0.901||0.797||0.738||4.230||6.560||4.500|
|Wang et al. ||0.905||0.838||0.786||3.890||6.479||3.282|
|Zhao et al. ||0.887||0.794||0.754||—||—||—|
|Isensee et al. ||0.896||0.797||0.732||6.970||9.480||4.550|
|Jungo et al. ||0.901||0.790||0.749||5.409||7.487||5.379|
|Single prediction||Islam et al. ||0.876||0.761||0.689||9.820||12.361||12.938|
|Jesson et al. ||0.899||0.751||0.713||4.160||8.650||6.980|
|Roy et al. ||0.892||0.793||0.716||6.735||9.806||6.612|
|Baseline + RR SE||0.891||0.799||0.704||7.270||11.180||5.840|
|Baseline + RR SegSE||0.895||0.798||0.733||5.920||8.947||5.074|
Comparing the single prediction approaches, it is possible to observe that our Baseline is competitive. This is relevant in the sense that it is hard to improve over a competitive approach, which sustains the added value of the RR SegSE block for semantic segmentation. In fact, our FCN with this block outperforms the other single prediction approaches in DC of the enhancing and the tumor core, as well as in HD of the enhancing tumor, being very close to Jesson et al.  in DC of the whole tumor. However, Jesson used a FCN with multiple prediction layers and loss functions in different scales. Additionally, the authors employed a learning curriculum to deal with class imbalance. Comparing with Roy et al. , we observe that their block obtains lower, but competitive, DC of complete and core, but the proposed RR SegSE block enables us to achieve better scores in DC of enhancing tumor and HD of all tumor regions. The block proposed by Roy includes a branch where the original SE block is used to learn to recalibrate at the channels level. However, according to our observations with the Baseline + RR SE variant in Table I and Fig. 5, it may not be adequate for segmenting fine structures such as enhancing tumor, which may be a reason why the proposed RR SegSE block performs better. When we compare with ensemble methods, we observe that incorporating the RR SegSE block allowed our approach to achieve competitive results, especially in DC. The HD metric suffers more from the presence of false positive detections, especially if they are far away from the object, and ensembles may effectively tackle this problem since different models do different mistakes. However, our results in DC are similar to the ones obtained by Kamnitsas et al. , who won BRATS 2017 Challenge edition by generalizing well to the Challenge set, despite the fact that Wang et al.  held the best Leaderboard set results.
In Table III we present the results on BRATS 2013 Leaderboard and Challenge sets. Even though these datasets are older and smaller, they allow us to compare with a larger variety of methods, such as CNN-based [28, 34, 43, 6]
, Random Forest and handcrafted features-based methods[37, 30], and multi-atlas patch-based methods . Comparing our Baseline with the FCN with RR SegSE block, we can observe that the latter achieves an overall better performance in terms of DC in both the Leaderboard and Challenge set. Still, it is worth noting the competitiveness of the Baseline FCN. We also observe that the proposed RR SegSE block obtains better Dice and Sensitivity than the RR SE block, but lower PPV, except in the core region in the Challenge set. This may be a hint that the SE block may fall into undersegmentation by suppressing the tumor regions. Comparing with the state of the art, we verify that in the Challenge set our proposal obtains higher DC and Sensitivity than the other approaches. In the case of Leaderboard, the results are competitive with Zhao et al. 
. However, their approach is based on an ensemble of three CNNs (one for each MRI plane), followed by a Conditional Random Field formulated as a Recurrent Neural Network and extensive post-processing. To our knowledge, these are new state-of-the-art results in BRATS 2013 dataset.
|Tustison et al. ||2015||0.87||0.78||0.74||0.85||0.74||0.69||0.89||0.88||0.83|
|Pereira et al. ||2016||0.88||0.83||0.77||0.88||0.87||0.74||0.89||0.83||0.81|
|Shen et al. ||2017||0.88||0.83||0.76||0.87||0.87||0.73||0.90||0.81||0.81|
|Zhao et al. ||2018||0.88||0.84||0.77||0.90||0.87||0.76||0.86||0.82||0.80|
|Havaei et al. ||2017||0.88||0.79||0.73||0.89||0.79||0.68||0.87||0.79||0.80|
|Pinto et al. ||2018||0.85||0.78||0.75||0.88||0.86||0.71||0.84||0.73||0.80|
|Cordier et al. ||2016||0.87||0.77||0.72||0.85||0.80||0.71||0.89||0.76||0.77|
|Baseline + RR SE||—||0.88||0.81||0.76||0.88||0.83||0.83||0.89||0.81||0.71|
|Baseline + RR SegSE||—||0.89||0.84||0.78||0.86||0.83||0.71||0.93||0.89||0.88|
|Tustison et al. ||2015||0.79||0.65||0.53||0.83||0.70||0.51||0.81||0.73||0.66|
|Pereira et al. ||2016||0.84||0.72||0.62||0.85||0.82||0.60||0.86||0.76||0.68|
|Zhao et al. ||2018||0.86||0.73||0.62||0.89||0.77||0.60||0.85||0.77||0.69|
|Havaei et al. ||2017||0.84||0.71||0.57||0.88||0.79||0.54||0.84||0.72||0.68|
|Pinto et al. ||2018||0.84||0.68||0.60||0.86||0.86||0.61||0.84||0.65||0.68|
|Cordier et al. ||2016||0.72||0.55||0.47||0.71||0.56||0.41||0.84||0.69||0.59|
|Baseline + RR SE||—||0.84||0.71||0.60||0.90||0.80||0.71||0.83||0.72||0.60|
|Baseline + RR SegSE||—||0.86||0.72||0.63||0.84||0.78||0.61||0.92||0.76||0.73|
As previously mentioned, the hierarchical segmentation approach for brain tumor segmentation is helpful when dealing with class imbalance. A possible limitation of this strategy is that if the first stage is poor in a given challenging subject, it will affect the quality of the second, multi-class, stage. Nevertheless, the results in Table II and Table III are competitive with the state of the art, which suggests that the proposed method does not fail to segment more tumors than the other methods, at least in these datasets.
In summary, in both BRATS 2017 Leaderboard and BRATS 2013 Leaderboard and Challenge sets we can draw two observations. 1) The variant with the RR SegSE block improves over our competitive Baseline FCN. 2) Although our baseline FCN is simple, our results with the proposed block are competitive, or superior, when comparing with the state of the art.
Iv-C Stroke penumbra estimation
Table IV depicts the results obtained in stroke penumbra estimation in the SPES dataset with the proposed approach, as well as other state-of-the-art methods. We can observe that the Baseline FCN is competitive when comparing with the other methods. Nevertheless, the Baseline + RR SegSE block was able to improve results further in terms of both DC and ASSD. Therefore, the benefits of the proposed RR SegSE block appear to generalize in this problem as well. Moreover, the proposed SegSE block outperforms the SE block, both in DC and ASSD. In fact, the SE has a negative effect over the Baseline FCN in SPES. Although in Table IV we report the metrics used in , the Baseline + RR SE obtained PPV and Sensitivity of and , respectively, while Baseline + RR SegSE scored and in terms of PPV and Sensitivity, respectively. Hence, we conclude that in SPES the SE block may suppress large portions of the lesions, since it suppresses complete feature maps. In contrast, the SegSE block is spatially adaptive, thus adaptively suppressing regions of the feature maps. In this way, the proposed blocks appear to be able to balance PPV and Sensitivity.
|CH-Insel ||0.82 0.08||1.65 1.40|
|Clèrigues et al. ||0.82 0.09||—|
|DE-Uzl ||0.81 0.09||1.36 0.74|
|BE-Kul2 ||0.78 0.09||2.77 3.27|
|CN-Neu ||0.76 0.09||2.29 1.76|
|Pereira et al. ||0.75 0.14||2.43 1.93|
|DE-UKF ||0.73 0.13||2.44 1.93|
|BE-Kul1 ||0.67 0.24||4.00 3.39|
|CA-Usher ||0.54 0.26||5.53 7.59|
|Baseline||0.81 0.10||1.30 0.74|
|Baseline + RR SE||0.80 0.12||1.54 1.40|
|Baseline + RR SegSE||0.82 0.09||1.27 0.72|
Results obtained in SPES Challenge set. Bold results show the best score considering mean and standard deviation in each metric.
are based on Representation Learning approaches. While CA-Usher and Clèrigues employ CNNs, Pereira learned features from data using Restricted Boltzmann Machines. Observing TableIV, we verify that our approach achieves better scores than CA-Usher and Pereira et al. , which may be due to those models being shallow. When we compare with the more recent FCN-based approach by Clèrigues et al. , we verify that both approaches obtained similar DC. The results of the participants in the SPES challenge are officially reported in  in terms of DC and ASSD. However, Clèrigues et al.  opted to use HD instead of ASSD. In this way, the authors report a HD score of , while the proposed Baseline + RR SegSE obtained . Therefore, although our approach performs similar to Clèrigues et al.  in DC, it is slightly better in HD, which may indicate a better delineation of the lesions. Two other top performing methods among the compared ones are CH-Insel and DE-Uzl, being both based on machine learning approaches with handcrafted features. The method proposed by the CH-Insel team employed the Random Forest classifier with texture features, and a bootstrapping scheme during training at the subject and voxel level. DE-Uzl is also based on the Random Forest classifier, but using intensity, hemispheric difference, local histograms, and center distances as features. Note that while CH-Insel obtained better DC than DE-Uzl, in terms of ASSD it was the other way around. When we compare the proposed Baseline + RR SegSE with those methods, it is possible to observe that it achieves the same DC as CH-Insel and better ASSD than both of those teams. Hence, the proposed approach appears to be competitive or outperform the state of the art in the SPES dataset.
Iv-D Ischemic stroke lesion outcome prediction
Table V presents the results obtained in ISLES 2017 Challenge set regarding ischemic stroke lesion outcome prediction. We observe that most of the top performing methods obtain higher sensitivity than PPV scores. This may be due to the existence of very small lesions with just a few voxels. Therefore, the models may tend towards over-estimation to avoid missing any lesion. Taking the Baseline FCN into consideration, we note that it is competitive with the state of the art. Indeed, it achieves a DC score of 0.31, placing it only below HKU-1. However, the proposed SegSE block is able to improve the DC score by 0.03, achieving a score of 0.34. This results from an increase of the Sensitivity metric, while PPV decreases only 0.01. Hence, we conclude that the proposed blocks contribute for predicting the lesions better. This is maybe due to their ability to suppress irrelevant regions, thus allowing the model to focus on the important parts of the image. The SegSE block also outperforms the SE block in this application, although the latter is also able to improve the DC of the Baseline FCN to 0.32. The RR SE block results in a large gap between Sensitivity and PPV, which may result in a poorer prediction than the one obtained by the RR SegSE block. Indeed, this is confirmed by the HD and ASSD of and , respectively, obtained by our RR SegSE block, against the HD and ASSD of and , respectively, of the RR SE block.
|SNU-2 ||0.31 0.23||0.36 0.27||0.45 0.31|
|UL ||0.29 0.21||0.34 0.26||0.51 0.33|
|HKU-1 ||0.32 0.23||0.34 0.27||0.39 0.28|
|INESC ||0.30 0.22||0.34 0.27||0.51 0.31|
|KUL ||0.27 0.22||0.44 0.33||0.39 0.31|
|SNU-1 ||0.28 0.23||0.36 0.31||0.41 0.31|
|UM ||0.29 0.22||0.26 0.24||0.61 0.28|
|MIPT ||0.27 0.20||0.31 0.28||0.39 0.29|
|SU ||0.26 0.21||0.28 0.25||0.56 0.26|
|KUL ||0.17 0.16||0.23 0.28||0.36 0.33|
|Pinto et al. ||0.29 0.21||0.23 0.21||0.66 0.29|
|Baseline||0.31 0.22||0.37 0.29||0.49 0.33|
|Baseline + RR SE||0.32 0.22||0.30 0.25||0.60 0.29|
|Baseline + RR SegSE||0.34 0.20||0.36 0.25||0.55 0.30|
The methods in Table V represent the top-10 methods that participated in ISLES 2017 challenge, together with Pinto et al. , which, to our knowledge, was the first approach to incorporate information from the perfusion Dynamic Susceptibility Contrast-enhanced MRI sequence. All of these methods are based on CNNs, mostly FCN-based. Moreover, SNU-1, SNU-2, HKU-1, and MIPT are ensembles of different FCN architectures and training settings. As expected, these ensemble-based approaches are among the highest ranked ones. SNU-2 and HKU-1 obtained DC of 0.31 and 0.32, respectively. However, the proposed Baseline + RR SegSE was able to outperform both methods by achieving a DC of 0.34. Moreover, the standard deviation was also smaller. Regarding PPV, both our method and SNU-2 performed similarly, with a 0.36 score, while the proposed method obtained sensitivity of 0.55, which is higher than both SNU-2 and HKU-1.
Ischemic stroke lesion outcome prediction differs from segmentation in the sense that we are predicting the final infarct core at a three month follow-up acquisition. In this scenario, the proposed blocks were able to improve an already competitive baseline FCN. This may be due to the capabilities of the proposed blocks to suppress irrelevant regions of the features maps, by taking context into account. To the best of our knowledge, the results obtained in ISLES 2017 are state-of-the-art.
In summary, channel recalibration of feature maps consists in learning the dependencies among channels, and use it to suppress the less relevant ones. Although desirable, this approach is not well suited for semantic segmentation with FCNs, where several voxels in a patch are segmented simultaneously. In this case, a feature map may contain regions that are relevant for certain voxels, but are less important for others. Also, convolutional layers with kernels were used before as bottlenecks in a way to decrease the computational complexity. In this work, we propose to use layers with kernels to recombine features, by an expansion followed by compression of the feature maps. Additionally, we propose a spatially adaptive recalibration block. With this block, we are able to suppress only the less relevant regions of the feature maps, while maintaining the important parts, behaving as an intra-channel attention mechanism. The proposed recalibration block (SegSE) employs dilated convolution for aggregating context. Experimentally, we show that the proposed RR with SegSE block leads to improvements over a competitive baseline. This behavior was observed in all three applications: brain tumor segmentation, stroke penumbra estimation, and ischemic stroke lesion outcome prediction. Our baseline FCN is a simple encoder-decoder FCN, and in this work we aimed at studying spatially adaptive recalibration. However, when we added the RR SegSE block we were able to achieve competitive or state-of-the-art results. Finally, the proposed block is general and it should be possible to add it to other FCN architectures for semantic segmentation.
Sérgio Pereira was supported by a scholarship from the Fundação para a Ciência e Tecnologia (FCT), Portugal (scholarship number PD/BD/105803/2014). This work is supported by FCT with the reference project UID/EEA/04436/2013, COMPETE 2020 with the code POCI-01-0145-FEDER-006941 and by COMPETE: POCI-01-0145-FEDER-007043 and FCT within the Project Scope:UID/CEC/00319/2013.
-  (2017) Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data 4. Cited by: §III-A.
-  (2018) SUNet: a deep learning architecture for acute stroke lesion segmentation and outcome prediction in multimodal mri. arXiv preprint arXiv:1810.13304. Cited by: §IV-C, TABLE IV.
-  (2016) A patch-based approach for the segmentation of pathologies: application to glioma labelling. IEEE T Med Imaging 35 (4), pp. 1066–1076. Cited by: §IV-B, TABLE III.
-  (2016) A guide to convolution arithmetic for deep learning. arXiv preprint arXiv:1603.07285. Cited by: §II-B2.
-  (2016) Deep learning. MIT press Cambridge. Cited by: §IV-B.
-  (2017) Brain tumor segmentation with deep neural networks. Med Image Anal 35, pp. 18–31. Cited by: §IV-B, TABLE III.
-  (2016) Deep residual learning for image recognition. In CVPR, pp. 770–778. Cited by: §I-A, §I, §I.
-  (2018) Squeeze-and-excitation networks. CVPR. Cited by: §I-A, §I, §I, §I, §II-B1, §II-B, Fig. 5, §IV-A.
-  (2017) Brain tumor segmentation and radiomics survival prediction: contribution to the brats 2017 challenge. In International MICCAI Brainlesion Workshop, pp. 287–297. Cited by: §IV-B, TABLE II.
-  (2017) Multi-modal pixelnet for brain tumor segmentation. In International MICCAI Brainlesion Workshop, Cited by: TABLE II.
-  (2017) Brain tumor segmentation using a 3d fcn with multi-scale loss. In International MICCAI Brainlesion Workshop, pp. 392–402. Cited by: §IV-B, TABLE II.
-  (2017) Towards uncertainty-assisted brain tumor segmentation and survival prediction. In International MICCAI Brainlesion Workshop, Cited by: §IV-B, TABLE II.
-  (2018) Ensembles of multiple models and architectures for robust brain tumour segmentation. In International MICCAI Brainlesion Workshop, Cited by: §IV-B, §IV-B, TABLE II.
-  (2017) Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med. Image Anal. 36, pp. 61–78. Cited by: §I.
-  (2015) Adam: a method for stochastic optimization. In ICLR, Cited by: §III.
-  (2012) Imagenet classification with deep convolutional neural networks. In NIPS, Cited by: §I.
-  (2013) Network in network. arXiv preprint arXiv:1312.4400. Cited by: §I-A, §I.
-  (2017) A survey on deep learning in medical image analysis. Med Image Anal 42, pp. 60–88. Cited by: §I, §I, §II.
-  (2015) Fully convolutional networks for semantic segmentation. In CVPR, pp. 3431–3440. Cited by: §I.
Machine learning in medical applications.
Advanced Course on Artificial Intelligence, Cited by: §I.
-  (2017) ISLES 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. Med Image Anal 35. Cited by: §I, §III-B, §III-D, §IV-C, §IV-C, TABLE IV.
-  (2015) The multimodal brain tumor image segmentation benchmark (brats). IEEE T Med Imaging 34 (10). Cited by: §I, §III-A, §III-D, §IV-B.
-  (2000) New variants of a method of mri scale standardization. IEEE T Med Imaging 19 (2), pp. 143–150. Cited by: §III-A, §III-B.
-  (2018) Attention u-net: learning where to look for the pancreas. International Conference on Medical Imaging with Deep Learning (MIDL). Cited by: §I-A, §I.
-  (2018) Adaptive feature recombination and recalibration for semantic segmentation: application to brain tumor segmentation in mri. In MICCAI, Cited by: §I-A, §III-A, footnote 7.
-  (2018) Enhancing interpretability of automatically extracted machine learning features: application to a rbm-random forest system on brain lesion segmentation. Med Image Anal 44. Cited by: §IV-C, TABLE IV.
-  (2017) On hierarchical brain tumor segmentation in mri using fully convolutional neural networks: a preliminary study. In Portuguese Meeting on Bioengineering, Cited by: §III-A.
-  (2016) Brain tumor segmentation using convolutional neural networks in mri images. IEEE T. Med. Imaging 35. Cited by: §I, §I, §IV-B, TABLE III.
-  (2018) Enhancing clinical mri perfusion maps with data-driven maps of complementary nature for lesion outcome prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 107–115. Cited by: §III-C, §IV-D, TABLE V.
-  (2018) Hierarchical brain tumour segmentation using extremely randomized trees. Pattern Recogn. Cited by: §III-A, §IV-B, TABLE III.
-  (2018) Autofocus layer for semantic segmentation. MICCAI. Cited by: §I.
-  (2015) U-net: convolutional networks for biomedical image segmentation. In MICCAI, pp. 234–241. Cited by: §I, §I, §II.
-  (2018) Recalibrating fully convolutional networks with spatial and channel’squeeze & excitation’blocks. arXiv preprint arXiv:1808.08127. Cited by: §I-A, §I, §IV-B, §IV-B, TABLE II.
-  (2017) Boundary-aware fully convolutional network for brain tumor segmentation. In MICCAI, Cited by: §IV-B, TABLE III.
-  (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Cited by: §I.
-  (2010) N4ITK: improved n3 bias correction. IEEE T. Med. Imaging 29 (6). Cited by: §III-A, §III-B.
-  (2015) Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (simplified) with antsr. Neuroinformatics 13 (2). Cited by: §IV-B, TABLE III.
-  (2017) Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. In International MICCAI Brainlesion Workshop, pp. 178–190. Cited by: §III-A, §IV-B, §IV-B, TABLE II.
-  (2018) ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral mri. Frontiers in neurology 9. Cited by: §III-C, §III-D, TABLE V.
-  (2017) Aggregated residual transformations for deep neural networks. In CVPR, Cited by: §I.
-  (2016) Multi-scale context aggregation by dilated convolutions. In ICLR, Cited by: §II-B2.
-  (2017) 3D brain tumor segmentation through integrating multiple 2d fcnns. In International MICCAI Brainlesion Workshop, pp. 191–203. Cited by: §IV-B, TABLE II.
-  (2018) A deep learning model integrating fcnns and crfs for brain tumor segmentation. Med. Image Anal. 43, pp. 98–111. Cited by: §IV-B, TABLE III.