Stroke ranks second as leading cause of deaths worldwide, with ischemic stroke being the most common type. Ischemic stroke arises from a sudden occlusion of a cerebral artery. Diagnosis and treatment begins with the acquisition of multi-modal MRI or CT images, followed by appropriate medical intervention. While several clinical trials have proven the efficacy of mechanical thrombectomy, the treating physician must carefully evaluate the associated risks and benefits: the volume of ill-perfused tissue potentially salvageable, versus the risk of causing haemorrhage or other complications [9, 14]. Hence, predicting the outcome of a stroke lesion (i.e. lesion status at three-month follow-up), and thereby evaluating the effect of a successful or unsuccessful mechanical thrombectomy, has a great potential to guide the decision of the physician.
, or through more advanced models such as decision trees and CNN-based deep learning architectures . However, up to our knowledge none of the approaches takes into consideration the temporal Perfusion Weighted Imaging data (4D PWI) for stroke lesion prediction. We hypothesize that a data-driven approach to model the raw perfusion imaging data can unveil information complementing the standard clinical perfusion maps derived using kinetic analysis.
In this paper, we propose a novel end-to-end deep learning multi-data branched network that incorporates information from the 4D PWI alongside the standard clinical perfusion and diffusion maps. From the time-stamp acquisitions of the 4D PWI, that characterize the bolus passage, we aim to learn brain blood flow hemodynamics (principal and collateral) to characterize tissue at risk of infarction (penumbra), and the unsalvageable tissue (ischemic core). Since standard perfusion and diffusion maps are generated from kinetic models followed by thresholding (based on clinical knowledge and experience), we hypothesize that there might be loss of relevant information. Hence, we aim to enhance the clinical MRI diffusion and perfusion maps with data-driven maps for stroke lesion outcome prediction. The proposed architecture was evaluated using the publicly-available ISLES 2017 dataset.
2.0.1 4d Pwi
At the arrival of the contrast agent to the brain, during the acquisition of the 4D PWI, the healthy tissue will present a drop in the signal intensity value, which then increases as contrast agent starts diluting throughout the system. However, in the presence of ill-perfused tissue the signal intensity values barely changes, since there is no propagation of the contrast agent to the damaged tissue . Figure 1 depicts such signal intensity behaviour in a patient with ischemic stroke. The perfusion blood flow dynamic, captured by the temporal slices of the 4D PWI, is responsible for the generation of the 3D MRI perfusion maps, through the application of kinetic models, deconvolutions in the time space, and clinical thresholding. Therefore, rCBF, rCBV, MTT, TTP, and Tmax perfusion maps can be viewed as surrogate parametric summaries of the raw 4D PWI, encompassing specific blood flow dynamics. From this knowledge emerged the intention to evaluate the encoding of the blood flow hemodynamics directly from 4D PWI, considering altogether complementary information over the diffusion and perfusion maps. Our approach was based on the peak concentration of contrasting agent, which is of extreme importance, since it characterizes the point where the differences of perfusion between healthy tissue and the ill-perfused tissue are higher, allowing a better detection of the penumbra 
. We developed an automatic approach to detect the time slice where the concentration of contrast agent is higher, which corresponds to a lower signal intensity. We detect automatically the peak of concentration using k-means on the mean signal intensity and standard deviation. The peak is used to define a temporal window to retrieve specific temporal acquisitions regarding the blood dynamics needed to estimate the tissue at risk of infarction. Besides reducing the total number of temporal slices, we also enforce the same spatial-temporal space across patients. Aligning patient data across the peak concentration of contrasting agent yields a common time interval for the retrieval of information. A fixed temporal window size of 26 slices was used, based on the sampling rate of the MRI acquisition.
2.0.2 Baseline Architecture.
For stroke outcome lesion prediction, we based our network on the U-Net , which has proved to be competitive in many biomedical image segmentation applications. The output of the U-Net is fed to a bi-dimensional GRU layer  that processes the information in four directions (superior-inferior, inferior-superior, anterior-posterior and posterior-anterior), to enforce a greater spatial context in stroke lesion outcome prediction. The baseline architecture only considers standard diffusion and perfusion MRI maps (as employed by the state of the art approaches).
2.0.3 Multi-Data Branched Network.
To merge information from the standard perfusion and diffusion maps and the data-driven 4D PWI data, the proposed architecture fuses two U-Nets as shown in Figure 2
. The top branch U-Net models the raw 4D PWI information, where the temporal information is coded as input channels. The first two layers consist of a feature expansion and feature reduction by 4. Recombining the feature maps allows complex interactions between temporal slices within the temporal window. Since each branch is able to learn different specific features, we then merge the output of each branch, which is fed into a smaller architecture in order to take advantage from complementary information for stroke lesion outcome prediction. The final portion of the network encompasses also a bi-dimensional GRU layer present in both branches of the network.
Our proposal was validated on the publicly available ISLES 2017 database, with a total of 75 cases divided into two datasets: Training dataset (n=43) and Challenge dataset (n=32). Each case contains a raw 4D PWI, five 3D MRI perfusion maps (rCBF, rCBV, MTT, TTP, Tmax), one 3D MRI diffusion map (ADC), and the final lesion outcome, which was manually segmented by a clinician on a 90-day follow-up (only available for the training dataset). All MRI maps are already co-registered and skull-stripped .
Since MRI acquisitions are from different centers, all maps were resized to the same volume space: . was clipped to , and the ADC was clipped to be within the range , as values beyond these ranges are known to be biologically meaningless . Afterwards, a linear scaling was performed between . Bias field correction was performed to the 4D PWI using the N4ITK method  before the resizing and scaling steps.
The training dataset was divided into cases for training and cases for validation. For each case 550 patches of size were randomly extracted. The network was trained with ADAM optimizer (learning rate), with a mini-batch of size
, using as loss function the soft-dice loss. The sum is performed for the voxels of the patch both in the binary prediction and the ground truth . The gradient of the Dice score for the voxel of prediction, was calculated as in Equation 1.
We compare the performance of our proposal with three different studies: Standard Branch, Data-Driven Branch, and Multi-Data Single Branch. The Standard Branch architecture considers diffusion and perfusion maps. The Data-Driven Branch studies the 4D PWI. In the Multi-Data Single Branch we combined the inputs from both branches into a single network. In Table 1 we report results on ISLES 2017 test dataset, which enables us to compare with state of the art methods.
|Challenge||Mok et al.*||0.32 0.23||40.74 27.23||8.97 9.52||0.34 0.27||0.39 0.27|
|Kwon et al.*||0.31 0.23||45.26 21.04||7.91 7.31||0.36 0.27||0.45 0.30|
|Bertels et al.*||0.30 0.21||33.85 16.82||6.81 7.18||0.34 0.26||0.51 0.32|
|Monteiro et al.*||0.30 0.22||46.60 17.50||6.31 4.05||0.34 0.27||0.51 0.30|
|Lucas et al.*||0.29 0.21||33.85 16.82||6.81 7.18||0.34 0.26||0.51 0.32|
|Choi et al.*||0.28 0.22||43.89 20.70||8.88 8.19||0.36 0.31||0.41 0.31|
|Robben et al.*||0.27 0.22||37.84 17.75||6.72 4.10||0.44 0.32||0.39 0.31|
|Pisov et al.*||0.27 0.20||49.24 32.15||9.49 10.56||0.31 0.27||0.39 029|
|Niu et al.*||0.26 0.20||48.88 11.20||6.26 3.02||0.28 0.25||0.56 0.26|
|Sedlar et al.*||0.20 0.19||58.30 20.02||11.19 9.10||0.23 0.24||0.40 0.29|
|Rivera et al.*||0.19 0.16||63.58 18.58||11.13 7.89||0.27 0.25||0.21 0.17|
|Islam et al.*||0.19 0.18||64.15 28.51||14.17 15.80||0.29 0.28||0.25 0.25|
|Chengwei et al.*||0.18 0.17||65.95 25.94||9.22 6.99||0.37 0.30||0.21 0.23|
|Yoon et al.*||0.17 0.16||45.23 19.14||12.43 11.01||0.23 0.27||0.36 0.32|
|Standard Branch||0.20 0.19||70.04 20.37||11.66 7.40||0.16 0.20||0.61 0.28|
|Data-Driven Branch||0.20 0.18||54.59 18.69||9.95 5.79||0.18 0.21||0.61 0.27|
|Multi-Data Single Branch||0.26 0.21||46.35 17.59||8.37 6.43||0.21 0.20||0.61 0.28|
|Multi-Data Branched||0.29 0.21||41.58 22.04||7.69 5.71||0.23 0.21||0.66 0.29|
|Static results from .|
4 Results and Discussion
Table 1 contains the results obtained on the ISLES 2017 testing dataset. The Standard Branch and the Data-Driven Branch achieved the same Dice score, but with differences in the distance metrics. The Data-Driven Branch was capable of predicting the lesion outcome with higher robustness, since the Hausdorff and ASSD are lower when compared to the Standard Branch. Nevertheless, both approaches are not capable of reaching state of the art performance. However, when we fuse the information of both models, as proposed in this paper, we observe an improvement on the average Dice score, but also on Hausdorff and ASSD. Being so, both models provide distinct information of value for stroke lesion outcome prediction. Additionally, we also study the performance of a single model with all the inputs combined, referred to as the Multi-Data Single Branch architecture. Such approach reached higher Dice score than the two branches separately, and with lower distance metrics. However, it was not capable of reaching the same performance of our proposal. Therefore, having 2 U-Nets for different input data has benefits on modularity and specificity on how the information is modelled. Using separate deep learning models to directly learn intrinsic biological phenomena allowed a higher robustness and accuracy in lesion outcome location and delineation, which is sustained by the lower distance metric values and higher Dice score.
In addition, we compare our proposal with other approaches. However, such comparison needs to consider the fact that top rank approaches use multiple models (i.e. ensembling). Our proposal was able to achieve a Dice score among the top 5 ranking methods, being within the same Hausdorff range of those methods, with just a single network. From this comparison we highlight the low distance metric values obtained, and also a Dice score in the same level of the 4th ranked method. Figure 3 shows the average Dice score and respective Hausdorff for each method.
From this analysis we emphasize the benefits of the proposed approach to extract and model information that might not be fully characterized by the standard perfusion and diffusion maps. To assess the complementarity of the data-driven perfusion maps, we computed the normalized mutual information between all the standard perfusion maps and each of the learned feature maps from the data-driven raw 4D PWI branch. As shown in Figure 4, low association values (less than 0.2) were obtained for all the extracted feature maps, meaning that both branches introduce new and complementary features.
Figure 5 shows the extracted features from the data-driven raw 4D PWI branch. Visually, feature 10 can reflect some descriptions of collateral blood flow, where features 16 and 18 focus on the surrounding lesion area itself in a complementary way. This analysis is particularly important since it can provide complementary information in the decision making process performed by the clinician, providing a better prediction of potentially salvageable tissue.
5 Conclusions and Future Work
Parametric perfusion maps can be affected by intrinsic patient physiology . To cope with this effect mathematical models are applied to standardize the behaviour of the contrast bolus passing. Nonetheless, it cannot be independent from patient specific blood flow hemodynamics, which can highly affect the perfusion parametric maps by adding a wide variability in the penumbra delineation . Therefore, in this work, we propose a deep learning architecture, that can process the information from raw 4D PWI data and generate complementary information to the perfusion parametric sequences, which as shown here can increase stroke lesion outcome prediction.
In the future, we intend to perform an interpretability analysis of the data-driven perfusion maps as well as an analysis of the learning patterns of the architecture to ensure the correctness of the predictions with respect to the data being used to drive the lesion outcome predictions.
Adriano Pinto was supported by a scholarship from the Fundação para a Ciência e Tecnologia (FCT), Portugal (scholarship number PD/BD/113968/2015). This work has been supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/2013.
-  ISLES 2017 Challenge. https://www.smir.ch/ISLES/Start2017, accessed: 2018-02-28
-  Barber, P., et al.: Identification of major ischemic change: diffusion-weighted imaging versus computed tomography. Stroke 30(10), 2059–2065 (1999)
-  Cho, K., et al.: On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 (2014)
-  Hosseini, M.B., Liebeskind, D.S.: The role of neuroimaging in elucidating the pathophysiology of cerebral ischemia. Neuropharmacology (2017)
-  Kemmling, A., et al.: Multivariate dynamic prediction of ischemic infarction and tissue salvage as a function of time and degree of recanalization. J CEREBR BLOOD F MET 35(9), 1397–1405 (2015)
-  Maier, O., et al.: Isles 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. MED IMAGE ANAL 35, 250–269 (2017)
Milletari, F., et al.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3DV, 2016 Fourth International Conference on. pp. 565–571. IEEE (2016)
-  Organization, W.H., et al.: Global status report on noncommunicable diseases 2014. World Health Organization (2014)
-  Ronneberger, O., et al.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on MICCAI. pp. 234–241. Springer (2015)
-  Scalzo, F., et al.: Regional prediction of tissue fate in acute ischemic stroke. ANN BIOMED ENG 40(10), 2177–2187 (2012)
-  Song, S., et al.: Temporal similarity perfusion mapping: A standardized and model-free method for detecting perfusion deficits in stroke. PloS one 12(10) (2017)
-  Tustison, N.J., et al.: N4itk: improved n3 bias correction. IEEE T MED IMAGING 29(6), 1310–1320 (2010)
-  Wardlaw, J.: Neuroimaging in acute ischaemic stroke: insights into unanswered questions of pathophysiology. J INTERN MED 267(2), 172–190 (2010)
-  Xie, S., et al.: Aggregated residual transformations for deep neural networks. In: CVPR, 2017 IEEE Conference on. pp. 5987–5995. IEEE (2017)