Stroke is the second leading cause of death worldwide (World Health Organization and others, 2014)
, being classified in two types: ischaemic and haemorrhagic(Grysiewicz et al., 2008). Ischaemic stroke is the most common type, resulting from an occlusion of a vessel, which can be caused by thrombolysis, haemodynamic factors, or embolic causes (Grysiewicz et al., 2008). Due to vessel occlusion, the insufficient supply of oxygenated blood to brain cells leads to hypoperfused brain tissue, triggering cellular mechanisms to preserve the integrity of the cell. The hypoperfused area consists of tissue at risk that can be salvaged, being designated penumbra. As time passes, in the absence of flow restoration or sufficient collateral blood flow supply, the hypoperfused tissue eventually reaches a non-salvageable state designated core or infarct tissue (Memezawa et al., 1992).
Diagnosis and treatment of ischaemic stroke relies on neuroimaging acquisitions, where Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are the preferred imaging modalities (Gonzalez et al., 2007). CT imaging remains the most used acquisition due to its rapidity and availability (Gonzalez et al., 2007). However, multi-parametric MRI provides a higher sensitivity in detecting early ischaemic stroke and assessing the penumbra region (Gonzalez et al., 2007)
. Treatment consists in restoring tissue perfusion levels, also known as reperfusion, by performing mechanical thrombectomy or thrombolysis. Since ischaemic stroke is a dynamic process that evolves over time, the treatment is only possible up to 24 hours, where viable neurones still persist(El Tawil and Muir, 2017; Zivelonghi and Tamburin, 2018). So, expert physicians must evaluate the benefits and risks of mechanical thrombectomy before an intervention, since it may cause haemorrhage, vascular injury, and other complications (Powers et al., 2018). If performed, the success of the intervention is assessed radiologically via angiography imaging and scored by a qualitative expert-generated scale designated the standardized Thrombolysis in Cerebral Infarction (TICI) scale (Higashida et al., 2003). During the decision-making process, the physician needs to assess the nature and location of the lesion alongside pathophysiological factors such as age, presence of comorbidities, and collateral circulation (Liebeskind, 2003). The latter is of utmost importance in ischaemic stroke. The presence of collateral circulation, where a secondary network of vessels is responsible for granting cerebral blood flow to the lesioned tissue, increases the chances of a successful reperfusion (Liebeskind, 2003). Asserting the potential efficacy of treatment can be time-consuming and prone to inter- and intra-variability among physicians, which is further potentiated when performed in a clinical emergency environment (Coutts et al., 2003). Moreover, since time is critical, MRI acquisitions are optimized for speed, which is accomplished by reducing the resolution (González et al., 2011), making the prediction of the final stroke lesion an intricate task. Thus, automatic prediction of a stroke lesion at a given time since stroke has a great potential to guide physicians in this time-critical decision-making process.
We propose a novel automatic method based on unsupervised and supervised deep learning. We utilize Restricted Boltzmann Machines (RBMs) to jointly characterise the lesion and blood flow information through a two-pathway architecture, trained with two subsets of standard parametric MRI maps. One subset encompasses the Time-To-Peak (TTP), Mean Transit Time (MTT), Time-to-Maximum (Tmax), and Apparent Diffusion Coefficient (ADC). The second set contains the ADC, the relative Cerebral Blood Volume (rCBV), and the relative Cerebral Blood Flow (rCBF). In a second stage, the feature maps computed by the RBMs are combined with the standard parametric MRI maps to form the input of a supervised deep learning architecture composed by Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The proposed architecture was evaluated using the publicly-available ISLES 2017 dataset.
1.1 On the complexity of predicting the final infarct stroke lesion
In acute ischaemic stroke, the clinical evaluation of the standard parametric maps (e.g. ADC and Tmax) can identify infarct tissue and tissue that will infarct in the absence of therapeutic intervention. In this analysis, the infarct tissue, is identified by the hypointense regions of the ADC map, which characterize tissue with limited diffusion (Butcher and Emery, 2010a). Hypoperfused tissue, i.e. tissue that will infarct, is identified by hyperintense regions of the Tmax map, indicating an increased arrival time of contrast agent (Butcher and Emery, 2010b). However, to correctly predict the final ischaemic stroke lesion, besides considering the complex time-evolving transformation of hypoperfused tissue to infarcted tissue, it is also necessary to appraise the impact of the clinical intervention, thrombectomy, on the underlying brain perfusion and diffusion.
A successful thrombectomy should restore the perfusion levels, recovering the hypoperfused tissue. However, several factors may affect the reperfusion, limiting the degree of success of the intervention. To better understand the nuances of the clinical intervention, consider the cases presented in Figure 1. In the first case, Figure 0(a), the ADC does not present any hypointense region, so no infarct tissue may be identified, and we should expect a complete recovery of the hypoperfused tissue indicated by Tmax; however, the follow-up delineation obtained after thrombectomy presents a large final lesion, which is explained by an unsuccessful intervention. In the second case, Figure 0(b), we observe a final infarct lesion that is smaller than the hypointense region present in the ADC (Figure 0(b) arrow). This indicates reversible diffusion restriction, which is a rare case (Labeyrie et al., 2012) and was only possible to identify by a follow-up T2-weighted acquisition. So, an automatic method for predicting the final stroke lesion has not only to capture the time-evolving process of diffusion and perfusion, but also to consider directly or indirectly the degree of success of the thrombectomy, which may condition the final lesion either to be confined to the hypointense region of the ADC map, or to grow to brain tissue areas that are hyperintense in the Tmax. Due to the time-evolving process of diffusion and perfusion, the complexity of predicting the lesion will aggravate as we move from a target window of some days to several months.
The complexity of the evaluation process may be also observed in the inter-rater agreement of expert radiologists in ISLES 2017 dataset, which obtained a Dice score of on delineating the lesion using a 90-day follow-up T2-weighted acquisition (Winzeck et al., 2018).
1.2 Previous Work
, the complexity of predicting the final stroke lesion has only recently attracted attention in the medical imaging community. For predicting the final stroke lesion several methods have been already proposed based on multivariate linear regression models(Scalzo et al., 2012; Rose et al., 2001; Kemmling et al., 2015) et al., 2017; Bauer et al., 2014), and CNNs (Choi et al., 2016). Furthermore, with the release of Ischaemic Stroke LEsion Segmentation (ISLES) Challenge in 2016 and 2017, new methods have been proposed. These aim to predict at a 90-day time-window.
Rose et al. (2001)
proposed a two-stage method based on parametric perfusion and diffusion MRI maps. On the first stage, the method defines a region of interest (ROI) based on the intensity signal of the standard parametric maps, the MTT, Cerebral Blood Flow (CBF), Cerebral Blood Volume (CBV), and Diffusion-Weighted Imaging (DWI). The second stage performs stroke tissue prediction using Gaussian mixture models trained in different sets of parametric maps.Bauer et al. (2014)
used Random Forests to segment or predict the final stroke lesion depending on whether acute stroke imaging or three-month follow-up imaging was available, respectively.McKinley et al. (2017) also used a two-stage classification approach as in Rose et al. (2001)
for lesion characterisation and lesion prediction, where each stage consists of two sets of Random Forests (RFs) classifiers. The first stage aims to define a ROI that encompasses the hypoperfused region. In the first set, each classifier is trained with features extracted from different sets of MRI parametric maps. Having defined the location and extension of the lesion, a second set of two RFs performs stroke tissue prediction. Such classifiers were trained on different sets of patients, stratified by the TICI score. One classifier is trained in patients with unsuccessful reperfusion interventions, whereas a second classifier is trained in patients with successful reperfusion. The final prediction is obtained by combining the results of both classifiers, using a logistic regression model.Scalzo et al. (2012) proposed a framework for stroke tissue prediction, which characterises the state of the lesion four days after clinical intervention (thrombectomy). From the Fluid Attenuation Inversion Recovery (FLAIR) MRI sequence, ADC and Tmax MRI maps, the method applies a regression model that learns the behaviour of neighbouring voxels within a cuboid. Kemmling et al. (2015) proposed a multi-modality approach based on CT and MRI maps with non-imaging clinical meta-data, namely the TICI score and the time to treatment of each patient, to perform stroke tissue prediction.
In another line of research, authors have investigated the use of deep learning (Choi et al., 2016; Nielsen et al., 2018; Robben et al., 2020) for stroke tissue prediction. Choi et al. (2016), the winner approach at ISLES 2016 Challenge, proposed an ensemble of twelve CNN architectures, grouped into two sets of networks. The first group comprehends four 3D U-Nets (Ronneberger et al., 2015) performing voxel-wise tissue prediction. The second group of networks uses two-pathway Fully Connected Networks (FCNs) performing two types of patch-wise classification. One set of FCNs classifies a patch as lesion if it includes any lesion voxel. The other set of FCNs classifies a patch as lesion if the central voxel is a lesion. After merging the two pathway FCN, the method incorporates meta-data by adding a dense layer of clinical predictors merged with the imaging output of each network. The final stroke lesion prediction results from a weighted merging of all models. Mok and Chung (2017) applied deep adversarial training for stroke tissue prediction in an ensemble of U-Nets. Monteiro and Oliveira (2017) proposed a method based on the V-Net architecture (Milletari et al., 2016)
. The training was conducted with a custom loss function that applies a weighted sum between Dice score and cross entropy.Lucas and Heinrich (2017)
proposed the use of a U-Net architecture, which combines patches from the MRI maps in the same slice, with patches from 3 neighbouring slices and 2 hemispheric flips. In the expanding path of the U-Net, each level computes a Dice loss for the healthy tissue and for the final lesion, after the softmax activation. Afterwards, all losses are summed up, having the loss of the lesion and healthy tissue weighted according to a prior probability(Winzeck et al., 2018). Robben and Suetens (2017) employed a CNN-based architecture inspired by Kamnitsas et al. (2017). The authors proposed to combine the MRI inputs with clinical meta-data, before feeding them to each branch of a two-pathway 3D network. In the first branch the input is kept with the original resolution, while in the second branch the input resolution was lowered by a factor of 3. The output of each branch is transformed to the same scale and merged by two fully connected layers. The network is trained with four different sets of hyper-parameters. These four networks are used as an ensemble, whose prediction is obtained by averaging the output of each one. Similarly, Niu et al. (2018) used multiple scales of overlapping 3D patches to capture local and global spatial information. In the review paper of Winzeck et al. (2018), Rivera et al.also built on the work of Kamnitsas et al. (2017) and Milletari et al. (2016), by proposing a scheme to extract different patch resolutions, independent of each other, that are fed into four different paths. Afterwards, a fully connected layer combines all the outputs to perform stroke tissue prediction. Pisov et al. (2017) employed an ensemble strategy by combining different CNN-based architectures to overcome the strong anisotropy of the data. As summarized by Winzeck et al. (2018)
, the work of Yoon et al.proposed a two-stage gated CNN. In a first stage, the authors perform lesion detection and delineation. Afterwards, based on the probability maps of the first stage, a second CNN architecture processes the regions where the probability maps of healthy tissue and lesion are close to each other.Pinto et al. (2018b)
made use of temporal perfusion imaging, the Dynamic Susceptibility Contrast-MRI, in a U-Net architecture. This architecture aims to temporally process and extract deep features, which are then combined with a second feature step of another U-Net network, which was trained on the standard parametric maps. Using a large CT dataset,Robben et al. (2020) predicted the final infarct stroke lesion with a temporal window ranging from hours to 5 days. The authors considered spatio-temporal CT perfusion as input to a deep neural network inspired in the architecture proposed by Kamnitsas et al. (2017). Additionally, the model combines CT neuroimaging with clinical meta-data. Nielsen et al. (2018) proposed a method based on the SegNet architecture (Badrinarayanan et al., 2015), predicting on a 30-day follow-up acquisition based on a private dataset.
Principal and collateral blood flow has been considered either directly by modelling the temporal perfusion imaging (Pinto et al., 2018b), or indirectly by perfusion and diffusion parametric maps (Choi et al., 2016; Maier et al., 2017; Scalzo et al., 2012), or through clinical information that characterises the success of the revascularization (McKinley et al., 2017)
. We hypothesize that modelling the haemodynamics of the brain when artery occlusion occurs can be beneficial for predicting the final stroke lesion. So, in this work, we investigate the representation of the haemodynamics through an unsupervised learning model. Contrary to previous approaches, we propose grouping the input maps according to their subjacent physical meaning and encoding each group separately with an RBM. As groups, we investigated the time-resolved perfusion maps (Tmax, TTP, MTT), and the blood-flow-dynamic related maps (rCBF, rCBV)(Butcher and Emery, 2010a, b). Our proposal of combining features obtained unsupervisedly and supervisedly was motivated by the knowledge that unsupervised models learn structural features of the original image, while the supervised models learn features conditioned on the label, so there is potential for obtaining richer and more discriminative features by joining both types of models.
This work presents an automatic approach for predicting the final stroke lesion, using onset neuroimaging data. The main contributions are:
The use of unsupervised models for extracting structural features of time-resolved perfusion and blood-flow-dynamic related MRI maps for predicting stroke lesion.
The use of local and long spatial context provided by gated recurrent neural networks for relating structural features and image information when learning features conditioned on the label in a supervised model.
The proposal of a competitive system which outperforms state-of-the-art methods to predict the final infarct stroke lesion, in ISLES 2017 Challenge dataset.
The remainder of the paper is organized as follows. Section 2 describes the fundamental components of the proposed method. Section 3 describes the dataset, the evaluation procedure and the setup. The results and the discussion are addressed in Section 4. Finally, in Section 5 we present the main conclusions.
In this work, predicting the final infarct stroke lesion consists of delineating the lesion’s spatial extent at a 90-day follow-up time-point, using multi-parametric MRI imaging, namely the ADC, MTT, TTP, Tmax, rCBF, and rCBV, which are acquired at the onset time-point. The architecture of the proposed system and its main components are described in the following subsections.
The overall architecture of the proposed method can be divided into two functional blocks, as shown in Figure 2.
The first functional block performs unsupervised representation learning using two unsupervised models, namely RBMs. This unsupervised block provides new features that represent structural information that complements the standard parametric MRI maps, enhancing the capacity of our model to predict the final infarct lesion. In our approach, we aim to model the clinical procedure, which first locates and delineates the lesion at current time, and then considers the blood flow haemodynamic that might influence the final stroke lesion prediction. This procedure is encoded in our two-path RBM. The first RBM is responsible for capturing information on lesion location and extension, referred to as the RBMLesion. The second RBM, RBMHaemo, aims to capture blood flow haemodynamics information (e.g. collateral circulation), which has been identified as a key factor by physicians when assessing stroke final infarct lesion in clinical reports (Berkhemer et al., 2016; Menon et al., 2015). On one hand, to locate the onset ischaemic stroke lesion, the RBMLesion considers standard parametric maps that characterise the arrival times and mean transit times of the contrast agent. In the presence of an ischaemic lesion, the occluded vessel can decrease or interrupt the normal brain perfusion, translating into hyperintense regions on time-related parametric maps (Butcher and Emery, 2010b). On the other hand, the RBMHaemo considers standard parametric maps that characterise the amount of blood being delivered in unit of time, which correlates to the cerebral blood flow haemodynamics (Butcher and Emery, 2010b). Thus, the RBMLesion considers the MTT, TTP and Tmax perfusion maps, while the RBMHaemo the rCBV and rCBF perfusion maps. Regarding the ADC standard diffusion map, it is present in both RBMLesion and RBMHaemo, since it provides higher brain structural information and allows the identification of tissue that is already infarcted. This separation of the input imaging allows the RBM to learn specific feature sets, which may enable the method to analyse difficult cases where information concerning the blood flow can have a favourable impact on the lesion prediction.
The second functional block consists of a deep learning architecture that comprehends 2D convolutional blocks in a U-Net-based structure, alongside recurrent blocks. As imaging input data, we combine the standard parametric maps with feature maps from each RBM, totalling input feature maps.
2.2 Restricted Boltzmann Machines
The RBM is an undirected graphical model constituted by two layers of nodes: a visible layer and a hidden layer (Rumelhart and McClelland, 1986). Each node has a weighted connection to all nodes in the other layer (Rumelhart and McClelland, 1986). However, there are no connections among nodes of the same layer. Originally, Rumelhart and McClelland (1986)
proposed the RBM to learn from binary data on both layers. However, this does not represent well continuous real-valued input data, which is the case of MRI data. Therefore, we model the visible nodes as linear units with independent Gaussian noise. The hidden nodes are modelled as Noisy Rectifier Linear Units (NReLU), since they have been reported to be suitable for feature extraction(Hinton, 2012). This kind of RBM was previously used in segmentation tasks, such as in Pereira et al. (2019)
. Mapping the input data into a feature vector is performed through the interaction of states between the visible and hidden units, which is learned by minimizing an energy function.
The complete pipeline of the unsupervised block is shown in Figure 3 and detailed in Section 3.4. The RBMLesion and RBMHaemo function as feature generators that output two complementary sets of feature maps and
. These features characterise the structure of the images; however, we are interested only on the most distinctive details. So, after training the RBMs, we perform feature selection to reduce the generated feature space, obtaining smaller but representative feature setsand , such that , for , where the operator denotes the cardinality of a set. In the feature selection, we would like to select the features from the RBM that encodes the MRI maps, but also that correlates with the stroke prediction. Since the RBM is an unsupervised method, we compute the Normalized Mutual Information to quantify the statistical dependence between each generated feature and the respective input MRI map, as defined by Equation 1 (Vinh et al., 2010):
where is the mutual information between an MRI parametric map, , and an output feature, ; defines the entropy of a map, namely, and . To relate the features of the RBM with the class label, we could use a classifier supervisedly trained. Since the neural network is trained iteratively, we use a RF classifer trained with the Mean Decrease Impurity (MDI) as a surrogate to make the feature selection tractable. After, we compute the MIRBM and and MDIRF, normalize the MIRBM by the maximum value, add both ranks and sort decreasingly. The best set will be the first features. Our selection method was inspired on the work of Pereira et al. (2018); however, their method cannot be directly applied, since it would generate too many features for our problem.
2.3 Convolutional and Recurrent Neural Networks
Our supervised functional block is based on the U-Net architecture as proposed by Ronneberger et al. (2015). The input of the U-Net considers the concatenation of standard parametric maps with the sets of feature maps extracted from the unsupervised block. In the first level of our encoder architecture we use four 2D convolutional blocks with kernel size of and channels. Afterwards, the output of the final convolutional block is down-sampled by a factor of 2, starting the second encoding level formed by two convolutional blocks with equal kernel size but doubling the number of feature maps. The third level of encoding follows the same pattern. The decoder level mimics the encoder counterpart. As in Ronneberger et al. (2015) we only used long skip connections among encoder and decoder levels. These encoder-decoder deep CNNs provide high levels of abstraction from the input data, increasing the global notion of context as the network grows deeper. However, it comes at a cost of a high receptive field (Zeiler and Fergus, 2014)
. Thus, we used a 2D architecture in the plane with the highest resolution, since the acquisition resolution is anisotropic in the dataset. Also, in the end of the decoding path we expanded our learning block with Gated RNNs. Due to their nature, Gated RNNs can capture short- and long-term spatial relations, by retaining information from previous nodes encoded in the time-steps. Hence, Gated-RNNs consider information from all previous nodes when analysing the current one. This property, when applied to imaging data, allows considering intra-slice contextual dependencies. In our work, we used a particular Gated-RNN, namely the Long-Short Term Memory (LSTM)(Hochreiter and Schmidhuber, 1997). However, the LSTM was intrinsically developed to process 1D data (Hochreiter and Schmidhuber, 1997) (e.g. time-series). To be applicable to 2D data, we developed an online 2D Partition layer that transforms a grid-structure input (e.g. an image) into a one-dimensional sequence. Inspired by Visin et al. (2016), the 2D Partition layer was predefined with a neighbourhood of , where each time-step is characterised by a feature space of four voxels. After, two Bidirectional LSTM layers are employed along the left-right and frontal-dorsal directions followed by an up-sampling layer. These four layers, referred as the Gated Recurrent block, are shown in Figure 2. In our supervised functional block, two Gated Recurrent blocks were used, where the Bidirectional LSTMs have and hidden layers, respectively. The impact of the main components is evaluated in an ablation study in the experiments.
3 Experimental Setup
We evaluated the proposed approach on the publicly available ISLES 2017 dataset and on a private dataset. ISLES has an online benchmark platform (Kistler et al., 2013) that performs automatic evaluation (SMIR Online Platform, 2017). In this section we describe the dataset, the training and evaluation, and the main hyper-parameters of our method.
ISLES 2017 dataset encompasses 75 ischaemic stroke patients, which are separated into two sets: training () and testing (). Both sets have patients that underwent mechanical thrombectomy. Each patient is characterised by six 3D parametric MRI maps: diffusion ADC map, perfusion rCBF, rCBV, TTP, MTT and Tmax maps. In addition to the standard parametric maps, each case is also characterised by a manual delineation of the lesion. This refers to the 90-day stroke lesion delineated with access to the follow-up T2-weighted acquisition. However, the manual delineation is only available for the training set, while the follow-up T2-weighted imaging is not disclosed for any set. All parametric MRI maps are already co-registered and skull-stripped (Winzeck et al., 2018). Figure 4 (top row) shows an example of MRI maps, alongside the manual lesion delineation, the Ground Truth (GT), of a patient.
The private dataset considers 23 acute ischaemic stroke patients that underwent clinical therapy, acquired at Bern University Hospital in Switzerland. As in ISLES 2017, each patient is characterized by the same six parametric maps, being the final lesion manually delineated at 90-day follow-up T2. The parametric maps were co-registered followed by skull-stripping with FSL BET2 on the co-registered follow-up T2 image (Jenkinson et al., 2012).
3.2 Evaluation Metrics
We evaluated our proposal with five metrics, which are the same ones computed by the online ISLES 2017 benchmark platform: Dice Similarity Score, Hausdorff Distance (HD), Average Symmetric Surface Distance (ASSD), Precision, and Recall(Kistler et al., 2013).
Dice score measures the spatial overlap between two volumes. HD corresponds to the highest distance between surface points of different volumes, which characterise spatial outliers in the prediction. ASSD quantifies the average distances between the volumes’ surface. Precision quantifies the proportions of correctly classified cases within a class, while Recall corresponds to the proportion of positive cases correctly identified as such.
3.3 Image pre- and post-processing
Since MRI acquisitions were acquired from different centers and configurations (Winzeck et al., 2018), for each patient we resized all maps to a common volume of dimension of . Afterwards, the ADC maps were clipped between and the Tmax maps were clipped to , since values beyond these ranges are known to be biologically meaningless (McKinley et al., 2017). Finally, a linear scaling was applied across all maps, to the range . The images are resized to its original size, after we perform the prediction.
We applied a morphological filtering as post-processing, but since the final stroke lesion presents a wide range of lesion volumes (Winzeck et al., 2018), we removed only small connected components with less than 250 voxels. This step was kept fixed for all the evaluated models.
3.3.1 Data Augmentation
Data augmentation can be used to increase the number of training samples and reduce over-fitting (Krizhevsky et al., 2012). Due to the relatively small size of the training dataset, we employed artificial data augmentation in the supervised portion of our proposal. For each sample, we applied rotations of , , .
3.4 Settings and model training
Unsupervised functional block
The unsupervised functional block was trained by optimizing the negative log-likelihood of the data. However, since computing the gradient is generally intractable, we performed the training by approximating the gradient with Contrastive Divergence with one step of alternating Gibbs sampling(Hinton, 2012). The training process of an RBM can be difficult if one tries to learn the parameter
of the energy function, which corresponds to the standard deviation of the Gaussian noise of a visible node(Hinton, 2012). According to Hinton (2012)
, we normalize each component of the data with zero mean and unit variance, and define. In Table LABEL:tab:params, we present the settings used for training the unsupervised model.
For training each RBM, we randomly extract 3D patches of shape from the respective input set of MRI maps, . Then, the 3D patches are reshaped into a 1D vector and fed into the visible layer of the RBM, having an input of size , as shown in Figure 3. After training, we extract features from the NReLU units noise-free activations. These units exhibit intensity equivariance when the bias has zero value, and they are noise free units (Nair and Hinton, 2010). Due to the large number of extracted feature maps (), we perform a feature selection step, as described in Section 2.2, where . The most appropriate cardinality of is discussed in Section 4.1.1.
Supervised functional block
As for the supervised functional block, the complete settings of the training are given in Table LABEL:tab:params. For each subject, patches were randomly sampled. The training comprehended subjects, while the remaining subjects were used for validation. The settings were optimized through cross-validation in a previous work (Pinto et al., 2018a). For training, we used soft Dice loss function (Milletari et al., 2016). It is defined as:
In the soft dice loss, the sum occurs over the set of voxels belonging to the predicted output patch, where denotes the probability of a voxel in the output patch and corresponds to the respective ground-truth label voxel.
4 Results and Discussion
In this section, we discuss the impact of the main contributions, namely, the incorporation of unsupervised learning with supervised learning and the Gated Recurrent blocks. Then, we compare our method with the state of the art in ISLES 2017 Challenge. Finally, we delve on the difficulty of predicting the final infarct stroke lesion.
4.1 Ablation Study
The ablation study aims to gradually measure the importance of the main components and consequently assert on the contribution of each component to the overall performance. Thus, we start by evaluating the importance of the unsupervised feature generator and the proposed input grouping. After, we focus on the use of the Gated Recurrent block and the choice of the dimensionality of the spatial context.
|Unsupervised Block||Supervised Block||Dice||HD||ASSD||Precision||Recall|
|height10pt width0pt–||U-Net||LSTM||0.30 0.21||36.58 16.62||6.96 5.08||0.30 0.26||0.55 0.31|
|height10pt width0ptRBMSingle (3 Feat.)||U-Net||LSTM||0.30 0.21||38.93 18.80||6.55 4.22||0.29 0.24||0.61 0.31|
|RBMSingle (6 Feat.)||U-Net||LSTM||0.30 0.21||36.94 19.19||6.72 4.43||0.29 0.24||0.59 0.31|
|RBMSingle (12 Feat.)||U-Net||LSTM||0.28 0.20||41.07 18.67||6.81 3.88||0.24 0.21||0.65 0.30|
|height10pt width0ptRBMHaemo||U-Net||LSTM||0.28 0.24||38.50 22.78||11.09 14.79||0.35 0.30||0.44 0.34|
|RBMLesion||U-Net||LSTM||0.31 0.21||35.38 15.75||6.44 4.43||0.30 0.24||0.59 0.30|
|height10pt width0ptRBMLesion RBMHaemo||U-Net||LSTM||0.38 0.22||29.21 15.04||5.52 5.06||0.41 0.26||0.53 0.29|
|Two-RBMsMixed||U-Net||LSTM||0.27 0.21||40.89 14.63||6.92 3.64||0.25 0.23||0.68 0.28|
|Three-RBMs1||U-Net||LSTM||0.35 0.23||29.32 14.33||5.27 3.54||0.34 0.27||0.59 0.30|
RBMHaemo/Less + RBMLesion/Less + RBMADC
4.1.1 Unsupervised feature generation
We hypothesize that grouping the parametric MRI maps according to their physical meaning and encoding each group with an RBM has potential to extract better features to characterise the stroke lesion and the blood haemodynamics. We perform several experiments to corroborate this working hypothesis. In all experiments, the parametric MRI maps are also used as input to the supervised block. The results are presented in Table 2. Figure 4 presents feature maps encoded by the RBMs and the respective MRI maps.
Grouping all parametric MRI maps in a single group
We considered, first, the effect of encoding all parametric maps using a single RBM. We varied the number of selected features from the RBM, observing that in all cases, the average Dice score is equal or lower than using only the parametric maps as input to the supervised block. Also, using features presented the lowest average Dice score. The use of or obtained the same average Dice score, having the second a lower average Hausdorff distance. So, based on the metrics, we may conclude that there is no clear gain in using the features generated by the RBM, at least, when we encode all the parametric maps with a single RBM.
Since, the selection of features also includes the previous top features, we compared the normalized mutual information between them. As shown in Figure 4, the top features have low values of normalized mutual information in relation to the additional features, which indicates that there is additional information. For this reason, we chose as the number of features in the subsequent experiments.
Grouping parametric MRI maps according to the subjacent physical meaning
In this experiment, we grouped the parametric maps according to their underlining physical meaning together with ADC map in each group. Each group was encoded with an RBM. Comparing isolatedly the use of each group of features, we verify that RBMLesion had a higher average Dice score compared to using only the parametric maps as input to the supervised block. The increase in the average Dice score was obtained by a higher average Recall. Also, we observe an improvement in all distance metrics. The experiment of using RBMHaemo presented the lowest average Dice and Recall, as well as higher average distance metrics. However, RBMHaemo presented higher average Precision, contrary to RBMLesion, which motivated the study on the combination of features from RBMHaemo with RBMHaemo besides the parametric maps. The results of this experiment are presented in Table 2. We may observe that this combination obtained the highest average Dice and Precision, as well as the lowest average distance metrics. However, this improvement could have been originated from the combination of maps according to a specific common property, subjacent physical meaning of the parametric maps, in each group, or because we reduced the number of maps from to in each group. And this reduction could have allowed a better training of the RBM. So, we performed two complementary experiments. In the first experiment, we formed two groups with similar size, but we randomly chose the parametric maps to include in each group. In the second experiment (Three-RBMs), we changed the groups of MRI maps encoded in RBMLesion and RBMHaemo by removing the ADC map from each one. These two new groups were encoded in RBMLesion/Less and RBMHaemo/Less, respectively. The ADC was separately encoded in RBMADC. As presented in Table 2, the first experiment presented the lowest average Dice score and higher average distance metrics, while the second experiment attained the second highest average Dice score, thus showing the importance of splitting the parametric perfusion maps and including the ADC map in both the RBMHaemo and RBMLesion.
|Unsupervised Block||Supervised Block||Dice||HD||ASSD||Precision||Recall|
|height10pt width0pt–||U-Net||–||0.30 0.21||38.83 21.10||7.08 5.15||0.26 0.23||0.64 0.30|
|U-Net||LSTM||0.30 0.21||36.58 16.62||6.96 5.08||0.30 0.26||0.55 0.31|
|height10pt width0ptRBMLesion RBMHaemo [3D]||U-Net||–||0.32 0.23||34.09 16.51||7.60 7.14||0.35 0.27||0.48 0.32|
|U-Net||LSTM||0.38 0.22||29.21 15.04||5.52 5.06||0.41 0.26||0.53 0.29|
Considering these experiments together, we may draw some conclusions. First, although CNNs are very effective in generating features from raw data, they can generate even better features if rich and complementary information is provided. A similar conclusion was inferred by Oliveira et al. (2018) that observed improvement when the coefficients of the Wavelet were added as input in the problem of retinal vessel segmentation. Here, we observe a similar effect, but using the encoding provided by an RBM trained unsupervisedly for the problem of stroke lesion prediction. Second, at least to the problem of stroke lesion prediction, when we have data with different latent factors and we are able to group it, according to those factors, then there is potential to extract complementary information from each group, but to mix them all together can be detrimental.
4.1.2 Context aggregation based on gated recurrent blocks
In medical imaging segmentation, which is similar to our problem of inferring the extension of the lesion 90 days ahead, the use of a cascade of convolutional layers to elaborate the features is the prevalent practise. However, as discussed previously, Gated-RNN layers are able to capture long distance spatial relations among input voxels, so we performed some experiments to evaluate its contribution. The results are presented in Table 3.
|Unsupervised Block||Supervised Block||Dice||HD||ASSD||Precision||Recall|
|height10pt width0ptRBMLesion RBMHaemo [2D]||U-Net [2D]||–||0.27 0.23||36.35 14.89||9.14 12.35||0.31 0.28||0.53 0.34|
|height10pt width0ptRBMLesion RBMHaemo [3D]||U-Net [2D]||–||0.32 0.23||34.09 16.51||7.60 7.14||0.35 0.27||0.48 0.32|
|U-Net [3D]||–||0.30 0.21||34.17 14.86||6.16 3.82||0.32 0.27||0.54 0.30|
|Unsupervised Block||Supervised Block||Dice||HD||ASSD||Precision||Recall|
|height10pt width0pt–||U-Net [2D]||–||0.32 0.18||32.58 20.09||5.17 3.34||0.31 0.26||0.68 0.28|
|height10pt width0ptRBMLesion RBMHaemo [3D]||U-Net [2D]||LSTM||0.36 0.18||26.68 15.60||3.88 2.17||0.38 0.30||0.68 0.27|
Analysing Table 3, we verify that when we just had parametric maps as input to the supervised block, adding a LSTM layer increased the average Precision, but the average Recall decreased, resulting in the same average Dice score. But, a different behaviour is observed when we added the features computed from the RBMs. In this scenario, we verify that using only CNN layers improved over having just parametric maps, which came by a higher average Precision. However, when we add the LSTM, we have an even higher improvement, which is observed in a larger increase in the average Precision, and a decrease in the average distance metrics.
Based on these experiments, we may conclude that the CNN layers were able to extract additional information from the RBM features; however, at least to the problem of inferring the extension of the lesion months ahead, long and local distance spatial relations among input voxels introduced by Gated RNN was critical to reduce the detection of false positives, increasing substantially the average Dice score by .
|Ensemble||Mok et al.*||0.32 0.23||40.74 27.23||8.97 9.52||0.34 0.27||0.39 0.27|
|Kwon et al.*||0.31 0.23||45.26 21.04||7.91 7.31||0.36 0.27||0.45 0.30|
|Robben et al.*||0.27 0.22||37.84 17.75||6.72 4.10||0.44 0.32||0.39 0.31|
|Pisov et al.*||0.27 0.20||49.24 32.15||9.49 10.56||0.31 0.27||0.39 029|
|Single Model||Monteiro et al.*||0.30 0.22||46.60 17.50||6.31 4.05||0.34 0.27||0.51 0.30|
|Pinto et al. (2018a)||0.29 0.21||41.58 22.04||7.69 5.71||0.21 0.21||0.66 0.29|
|Lucas et al.*||0.29 0.21||33.85 16.82||6.81 7.18||0.34 0.26||0.51 0.32|
|Choi et al.*||0.28 0.22||43.89 20.70||8.88 8.19||0.36 0.31||0.41 0.31|
|Niu et al.*||0.26 0.20||48.88 11.20||6.26 3.02||0.28 0.25||0.56 0.26|
|Sedlar et al.*||0.20 0.19||58.30 20.02||11.19 9.10||0.23 0.24||0.40 0.29|
|Rivera et al.*||0.19 0.16||63.58 18.58||11.13 7.89||0.27 0.25||0.21 0.17|
|Islam et al.*||0.19 0.18||64.15 28.51||14.17 15.80||0.29 0.28||0.25 0.25|
|Chengwei et al.*||0.18 0.17||65.95 25.94||9.22 6.99||0.37 0.30||0.21 0.23|
|Yoon et al.*||0.17 0.16||45.23 19.14||12.43 11.01||0.23 0.27||0.36 0.32|
|Baseline||0.30 0.21||36.58 16.62||6.96 5.08||0.30 0.26||0.55 0.31|
|Proposal||0.38 0.22||29.21 15.04||5.52 5.06||0.41 0.26||0.53 0.29|
|Methods presented in Winzeck et al. (2018), whose results were retrieved from SMIR Online Platform (2017).|
4.1.3 Spatial context: 2D or 3D?
MRI images are 3D by nature, so the use of 3D filters would allow capturing more context, which has the potential to provide better prediction. Since 2D filters are confined to a plane, unnatural discontinuous contour may occur in the perpendicular axis. However, as presented previously, the resolution of MRI images in ISLES dataset is not equal in all axis, being coarser along the axial axis. So, we studied the effect of the spatial context in our architecture. As we have two blocks, unsupervised and supervised blocks, the effect on each one was evaluated separately. The results are presented in Table 4. Considering the results, we observe that using 2D patches in both blocks has lower average Dice score, than using only the parametric maps as input (baseline), because the increase in the average Precision was not enough to compensate the drop in the average Recall. Using 3D patches for both blocks had the same performance as our baseline. However, when we used 3D patches for the RBM but 2D blocks for the U-Net block, we improved over our baseline. This is the model with the highest average Dice score without LSTM. So, we may conclude that for our architecture, larger context using 3D patches was more effective for encoding features in the unsupervised block, while 2D patches were better suited for encoding features in the supervised U-Net-based block.
4.2 Private dataset
To further evaluate the generalization capacity of our proposal, we tested it on a private dataset and compare it with the baseline method. Table 5 presents the results obtained by the two methods.
On the overall, our proposal was capable of surpassing the baseline model, attaining an higher average Dice, Precision and distance metrics, which were statistically significant (Wilcoxon Signed Ranked test with ). Comparing to ISLES 2017 testing set, there was a slight decrease in performance. This could be explained by the shift on the intensity distribution of the MRI maps, due to different acquisition protocols or the differences in the preprocessing step.
4.3 State-of-the-art: ISLES 2017 Challenge
The results of published methods for final infarct stroke lesion prediction using ISLES 2017 testing set (Winzeck et al., 2018), together with our baseline and proposal methods are presented in Table 6. The metrics were computed by the online platform, so the ground-truth data, which was manually delineated based on a follow-up T2 MRI acquisitions, are not disclosed for public access.
Considering the results, we observe that our baseline is competitive with an average Dice, being among the top 3 methods, and surpassing the ensemble methods of Pisov et al. (2017) and Robben and Suetens (2017). Our proposed method presented the lowest distance metrics among all methods, especially for the Hausdorff distance. It obtained the second-best average Precision score, being surpassed by Robben and Suetens (2017) The authors proposed the integration of meta-data information, using a two-pathway 3D network in an ensemble; however, our experiments did not indicate any improvement using 3D patches for the U-Net, at least for our architecture. So, this improvement could have come from a combination of the effect of the ensemble and the meta-data. But we note that their method presented a lower average Recall, which explains their lower average Dice score. Regarding the average Recall score, our method was fourth, but when we consider the top 3 methods, specially Pinto et al. (2018a), we conclude that it was obtained with a much lower average Precision, which means that to increase the true positive detections, they had to increase substantially the false positives. So, comparing with the state of the art, our method presented a better balance between Precision and Recall, which reflected into a higher average Dice score.
Based on the results, we may conclude that the use of complementary features provided by the RBMs and the use of LSTM for a larger context allowed our baseline to surpass current state-of-the-art methods.
Results from ChallengeR Benchmark
The SMIR platform of ISLES 2017 provides a weekly benchmark report of the current top-10 methods in the testing set, according to the average Dice score. So, some of the methods may not be published, lacking a description on their implementation, and, for this reason, were not included in the previous discussion.
Figure 5 presents the boxplots of each method considered in the report.
We observe that the top-10 methods failed to predict the lesion of one or more cases (lowest outliers), which may indicate the degree of complexity of predicting infarct stroke lesion 90 days ahead in ISLES 2017 Challenge dataset. But we verify that our method is the only one to have the first quartile above 0.20 in the Dice score. In Figure6 we have the significance maps of the pairwise significant test with one-side Wilcoxon signed rank test (p-value ), showing that our method reached higher Dice score statistically significant when compared with other five ranked methods of the top-10.
Figure 7 shows the podium plot of each method for each case in the testing set, and its ranking. We observe that our proposal is the method, which ranked first most of the times, as well as second and third. Also, when we consider the methods ranked bellow fourth, our method is in general among those with the lowest counts. Analysing the cases individually, we note two trends, for some cases all methods presented similar performance, while for others, we find a large variation from the first to the other methods. The first trend may be found in the most difficult case, where all methods had zero or a close value for the Dice score. In the second trend, we observe that our method is ranked as first most of the cases.
Based on the results of the benchmark, we may infer that our method is competitive among current state of the art, presenting the highest average Dice score and lowest average distance score. Considering the ablation study, this performance was attained due to the combination of adding extra features obtained by encoding the parametric maps with RBMs, according to the underlining physical meaning, and the elaboration provided by the long context of the LSTM layers.
In this work, we present a deep learning approach for predicting the final stroke lesion, based on unsupervised and supervised learning. We proposed to group the input maps according to the underlying physical principle behind their creation, namely, the time-resolved perfusion maps (Tmax, TTP, MTT), and the blood-flow-dynamic related maps (rCBF, rCBV). Each group was encoded using an unsupervised model to obtain structural features specific to its underlying physical principle. These structural features together with the standard parametric maps were fed to a supervised model to learn features conditioned on the label, which in our problem, means to condition on the results of the medical intervention — lesion at 90-days follow-up. We also investigated the use of Gated Recurrent Neural Networks to provide long spatial context, which were critical in relating the structural features to the information on input parametric maps. Our results showed that either the encoding or the long spatial context improved over our baseline. Also, these two together interacted positively increasing the performance when considering separately each one.
When evaluating our proposal on ISLES 2017 testing dataset, we observe a prediction improvement over current state-of-the-art methods. The proposed method obtained the first place in Dice and also in HD and ASSD.
Recent works (Pinto et al., 2018a; Robben et al., 2020) have shown the importance of clinical meta-data to predict the final stroke lesion in different revascularization scenarios. So as future work, we aim to study how such meta-data (i.e. TICI score) could be incorporated in our architecture, to consolidate the impact of the clinical intervention and to further improve the 90-day lesion prediction.
Adriano Pinto was supported by a scholarship from the Fundação para a Ciência e Tecnologia (FCT), Portugal (scholarship number PD/BD/113968/2015). This work was supported by FCT national funds, under the national support to R&D units grant, through the reference project UIDB/04436/2020 and UIDP/04436/2020.
- Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561. Cited by: §1.2.
- Towards automatic MRI volumetry for treatment selection in acute ischemic stroke patients. In Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE, pp. 1521–1524. Cited by: §1.2, §1.2.
- Collateral status on baseline computed tomographic angiography and intra-arterial treatment effect in patients with proximal anterior circulation stroke. Stroke 47 (3), pp. 768–776. Cited by: §2.1.
- Acute stroke imaging part i: fundamentals. Canadian Journal of Neurological Sciences 37 (1), pp. 4–16. Cited by: §1.1, §1.2.
- Acute stroke imaging part ii: the ischemic penumbra. Canadian Journal of Neurological Sciences 37 (1), pp. 17–27. Cited by: §1.1, §1.2, §2.1.
- Ensemble of deep convolutional neural networks for prognosis of Ischemic Stroke. In International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 231–243. Cited by: §1.2, §1.2, §1.2.
- Reliability of assessing percentage of diffusion-perfusion mismatch. Stroke 34 (7), pp. 1681–1683. Cited by: §1.
- Thrombolysis and thrombectomy for acute ischaemic stroke. Clinical Medicine 17 (2), pp. 161–165. Cited by: §1.
- Acute ischemic stroke. Springer. Cited by: §1.
- Acute ischemic stroke: imaging and intervention. American Journal of Neuroradiology 28 (8), pp. 1622. Cited by: §1.
- Epidemiology of ischemic and hemorrhagic stroke: incidence, prevalence, mortality, and risk factors. Neurologic clinics 26 (4), pp. 871–895. Cited by: §1.
- Trial design and reporting standards for intraarterial cerebral thrombolysis for acute ischemic stroke. Journal of Vascular and Interventional Radiology 14 (8), pp. E1–E31. Cited by: §1.
- A practical guide to training restricted Boltzmann machines. In Neural networks: Tricks of the trade, pp. 599–619. Cited by: §2.2, §3.4.
- Long short-term memory. Neural computation 9 (8), pp. 1735–1780. Cited by: §2.3.
- Fsl. Neuroimage 62 (2), pp. 782–790. Cited by: §3.1.
- Efficient multi-scale 3d CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis 36, pp. 61–78. Cited by: §1.2.
- Multivariate dynamic prediction of ischemic infarction and tissue salvage as a function of time and degree of recanalization. Journal of Cerebral Blood Flow & Metabolism 35 (9), pp. 1397–1405. Cited by: §1.2, §1.2.
- The virtual skeleton database: an open access repository for biomedical research and collaboration. Journal of medical Internet research 15 (11). Cited by: §3.2, §3.
- Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105. Cited by: §3.3.1.
- Diffusion lesion reversal after thrombolysis: a mr correlate of early neurological improvement. Stroke 43 (11), pp. 2986–2991. Cited by: §1.1.
- Collateral circulation. Stroke 34 (9), pp. 2279–2284. Cited by: §1.
- 2d multi-scale res-net for stroke segmentation. Cited by: §1.2.
- ISLES 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Medical image analysis 35, pp. 250–269. Cited by: §1.2, §1.2.
Fully automated stroke tissue estimation using random forest classifiers (faster). Journal of Cerebral Blood Flow & Metabolism 37 (8), pp. 2728–2741. Cited by: §1.2, §1.2, §1.2, §3.3.
- Penumbral tissues salvaged by reperfusion following middle cerebral artery occlusion in rats.. Stroke 23 (4), pp. 552–559. Cited by: §1.
- Differential effect of baseline computed tomographic angiography collaterals on clinical outcome in patients enrolled in the interventional management of stroke iii trial. Stroke 46 (5), pp. 1239–1244. Cited by: §2.1.
- V-net: fully convolutional neural networks for volumetric medical image segmentation. In 3D Vision (3DV), 2016 Fourth International Conference on, pp. 565–571. Cited by: §1.2, §3.4.
- Deep adversarial networks for stroke lesion segmentation. Cited by: §1.2.
- Fully convolutional neural network for 3d stroke lesion segmentation. Cited by: §1.2.
Rectified linear units improve restricted Boltzmann machines.
Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814. Cited by: §3.4.
- Prediction of tissue outcome and assessment of treatment effect in acute ischemic stroke using deep learning. Stroke 49 (6), pp. 1394–1401. Cited by: §1.2.
- Improved prediction of the final infarct from acute stroke neuroimaging using deep learning. In STROKE, Vol. 49. Cited by: §1.2.
- Retinal vessel segmentation based on fully convolutional neural networks. Expert Systems with Applications 112, pp. 229–242. Cited by: §4.1.1.
- Adaptive feature recombination and recalibration for semantic segmentation with fully convolutional networks. IEEE Transactions on Medical Imaging 38 (12), pp. 2914–2925. External Links: Cited by: §2.2.
- Enhancing interpretability of automatically extracted machine learning features: application to a RBM-Random Forest system on brain lesion segmentation. Medical image analysis 44, pp. 228–244. Cited by: §2.2.
- Stroke lesion outcome prediction based on MRI imaging combined with clinical information. Frontiers in Neurology 9, pp. 1060. Cited by: §3.4, §4.3, Table 6, §5.
- Enhancing clinical MRI perfusion maps with data-driven maps of complementary nature for lesion outcome prediction. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, pp. 107–115. Cited by: §1.2, §1.2.
- Neural networks ensembles for ischemic stroke lesion segmentation. Cited by: §1.2, §4.3.
- 2018 guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 49 (3), pp. e46–e99. Cited by: §1.
- Medical image analysis methods in MR/CT-imaged acute-subacute ischemic stroke lesion: segmentation, prediction and insights into dynamic evolution simulation models. a critical appraisal. NeuroImage: Clinical 1 (1), pp. 164–178. Cited by: §1.2.
- Prediction of final infarct volume from native ct perfusion and treatment parameters using deep learning. Medical image analysis 59, pp. 101589. Cited by: §1.2, §5.
- Dual-scale fully convolutional neural network for final infarct prediction. In Ischemic stroke lesion segmentation-ISLES challenge 2017, held in conjunction with MICCAI 2017, Date: 2017/09/10-2017/09/10, Location: Quebec City, Quebec, Canada, Cited by: §1.2, §4.3.
- U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Cited by: §1.2, §2.3.
- MRI based diffusion and perfusion predictive model to estimate stroke evolution. Magnetic resonance imaging 19 (8), pp. 1043–1053. Cited by: §1.2, §1.2.
- Parallel distributed processing: explorations in the microstructure of cognition. volume 1. foundations. Cited by: §2.2.
- Regional prediction of tissue fate in acute ischemic stroke. Annals of Biomedical Engineering 40 (10), pp. 2177–2187. External Links: Cited by: §1.2, §1.2, §1.2.
- Ischemic stroke lesion segmentation 2017. Note: https://www.smir.ch/ISLES/Start2017[Acessed: 2019-11-18] Cited by: §3, Table 6.
- Information theoretic measures for clusterings comparison: variants, properties, normalization and correction for chance. Journal of Machine Learning Research 11 (Oct), pp. 2837–2854. Cited by: §2.2.
- Reseg: a recurrent neural network-based model for semantic segmentation. In , pp. 41–48. Cited by: §2.3.
- ISLES 2016 & 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. Frontiers in Neurology 9, pp. 679. Cited by: §1.1, §1.2, §3.1, §3.3, §3.3, §4.3, Table 6.
- Global status report on noncommunicable diseases 2014.. World Health Organization. Cited by: §1.
- Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818–833. Cited by: §2.3.
- Mechanical thrombectomy for acute ischemic stroke: the therapeutic window is larger but still “time is brain”. Functional neurology 33 (1), pp. 5. Cited by: §1.