Fast Dynamic Perfusion and Angiography Reconstruction using an end-to-end 3D Convolutional Neural Network

08/24/2019 ∙ by Sahar Yousefi, et al. ∙ Leiden University Medical Center 0

Hadamard time-encoded pseudo-continuous arterial spin labeling (te-pCASL) is a signal-to-noise ratio (SNR)-efficient MRI technique for acquiring dynamic pCASL signals that encodes the temporal information into the labeling according to a Hadamard matrix. In the decoding step, the contribution of each sub-bolus can be isolated resulting in dynamic perfusion scans. When acquiring te-ASL both with and without flow-crushing, the ASL-signal in the arteries can be isolated resulting in 4D-angiographic information. However, obtaining multi-timepoint perfusion and angiographic data requires two acquisitions. In this study, we propose a 3D Dense-Unet convolutional neural network with a multi-level loss function for reconstructing multi-timepoint perfusion and angiographic information from an interleaved 50 50 present a framework to generate dynamic pCASL training and validation data, based on models of the intravascular and extravascular te-pCASL signals. The proposed network achieved SSIM values of 92.89 ± 1.18 and 96.50 ± 0.70 respectively for 4D perfusion and angiographic data reconstruction for 313 test data-sets.



There are no comments yet.


page 9

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Arterial spin labeling (ASL) is a non-invasive MRI technique which uses magnetically labeled blood water as an endogenous tracer for assessing cerebral blood flow (CBF) [5]. Hadamard time-encoded(te)-ASL is a time-efficient approach which provides the possibility to combine the superior SNR of ASL to acquire data at different inflow times to obtain dynamic ASL-data [15]. When Hadamard te-ASL is done with and without flow-crushing, 4D magnetic resonance angiography (MRA) and arterial input function (AIF) measurements can be obtained next to the perfusion scans. While this approach improves quantification and enhances information content, it is a factor two slower, since both crushed and non-crushed data need to be acquired. Accelerating te-ASL quantification can be done either by acquiring sub-sampled data in k-space or by reducing the rank of the Hadamard matrix. However, these methods can end up reducing image quality and/or signal-to-noise (SNR) ratio.

In this work, we propose an end-to-end 3D convolutional neural network (CNN) for the reconstruction of multi-timepoint 4D MRA and perfusion scans by using half-sampled crushed as well as half-sampled non-crushed Hadamard te-ASL scans, to maintain image quality and provide accurate CBF quantification. Recently, CNNs have shown outstanding performance in medical imaging [16, 4, 7]. However, very few CNN reconstruction techniques have been proposed in the context of MRA and perfusion reconstruction. In [6] a U-net shape CNN for boosting SNR and resolution of ASL scans has been proposed. Guo et al. proposed a CNN based method for improving 3D perfusion image quality by the combined use of single- and multi-delay pseudo-continuous arterial spin labeling (PCASL) and an anatomical scan [8]. In Guo’s study, ground truth perfusion maps were obtained by positron emission tomography scans. In [10]

a temporal CNN approach was proposed for perfusion parameter estimation in stroke. The proposed CNN takes in the signals of interest (i.e., concentration-time curves and the AIF) to produce estimated perfusion parameter maps including cerebral blood volume (CBV), CBF, time-to-maximum, and mean transit time.

In this work, different from the previous works, we employ CNNs in order to accelerate the simultaneous acquisition of 4D MRA and perfusion measurements. One of the challenging issues is the different properties of the outputs of the proposed CNN since MRA is intrinsically sparse and has much more elongated structures than the smooth perfusion map. We tackle this issue by employing different weighting of the loss functions of these two output-types and by balancing extracted samples during training. The proposed CNN leverages the idea of dense blocks [11], arranging them in a typical U-shape [12]. Loop connectivity patterns in dense blocks improve the flow of gradients throughout the network and strengthen feature propagation and feature re-usability [16]. In this investigation, we compare the performance of several loss functions: mean square error (MSE), VGG-16 perceptual loss, structural similarity index (SSIM) and multi-level SSIM (ML-SSIM). The main contributions of our work are:

  • To the best of our knowledge, we are the first to propose acceleration of the reconstruction of 4D MRA and perfusion images using interleaved sub-sampled crushed and non-crushed Hadamard te-ASL scans. To allow sub-sampling, we employed an end-to-end 3D CNN for decoding.

  • We employed a framework for generating training and validation 4D MRA and perfusion scans by generalizing the Buxton kinetic model for a Hadamard te-ASL signal. Different from [17], we consider the kinetic arterial model to take into account the arterial compartment.

  • We propose a CNN with a multi-level loss function and compare the proposed method with several loss functions, i.e. MSE, VGG-16 perceptual loss, and SSIM.

2 Proposed approach

2.1 Proposed Network

Fig. 1 illustrates the proposed network, which takes sub-sampled crushed and sub-sampled non-crushed interleaved ASL data as input and outputs dynamic MRA and perfusion scans. For managing GPU memory, the network was implemented patch-based. The input patches (of size ) are extracted from sub-sampled Hadamard te-crushed and te-non-crushed scans. The outputs of the network are 14 patches of size containing perfusion and angiography patches, each at seven different time-points. In this study, we considered a Hadamard matrix of rank 8, so the inputs are 8 patches in total (4 crushed, 4 non-crushed). In each dense block two (

)conv-BN-leaky ReLu and one (

)conv-BN-leaky ReLu, as a bottleneck layer, are stacked. Loop connectivity patterns in dense blocks are employed to improve the flow of gradients [11]. The bottleneck layers are used to increase the number of feature maps in a tractable fashion, which make the training process easier while leading to a more compact model. A down-sampling unit is followed by one max-pooling layer with a stride of . In order to solve the well-known checkerboard issue of the conv-transpose layer, for the up-sampling layer the feature maps are re-sized by a constant trilinear resize convolution kernel, similar to [3]. In this work we investigate the impact of several loss functions for the defined problem: MSE, which is the -norm, VGG-16 perceptual loss [13], SSIM which is composed of luminance, contrast and structural error, and ML-SSIM, which is calculated based on weighting the SSIM loss function for different levels of the network, see Fig. 1.

Figure 1: Proposed network with single- and multi-level loss functions. The training data contains sub-sampled and interleaved Hadamard-crushed and non-crushed scans. For the single- and multi-level loss functions, and are considered respectively. GT stands for ground truth, which is the set of angiographic and perfusion data reconstructed by the standard full-sampled decoding approach [15].

2.2 Dataset generation

In pCASL the arterial spins are magnetically labeled with a radiofrequency inversion pulse applied below the imaging slices in the neck vessels. The labeled blood then travels via the arteries towards the brain tissue, where they pass from the capillary compartment into the extravascular compartment. After a certain delay time after labeling which is known as the post-labeling delay (PLD) a so-called labeled image is acquired. A control image is acquired without prior labeling and by subtraction of these two images, the perfusion image can be generated. For the Hadamard te-pCASL technique, the labeling module (the typical duration of 3-4 seconds) is divided into several blocks (sub-boli) and a Hadamard matrix is used to determine whether a block will be played-out in label or control condition. For each voxel the Hadamard te-pCASL signal can contain both perfusion signal as well as label still residing in the arteries, i.e. angiography signals.

Since it is difficult to acquire substantial amounts of real data, we propose to model the input data, allowing to generate a sufficient amount of training data. The ground truth output data is created by decoding fully sampled Hadamard te-ASL crushed and non-crushed data [15]. For this purpose, we create datasets based upon a tracer kinetic model for the Hadamard time-encoded pCASL signal that describes the signal a function of arterial arrival time (AAT), bolus arrival time (BAT) and CBF. In this study, for calculating the signal, the AAT and BAT information are obtained from in vivo data. The CBF maps are taken from the BrainWeb dataset by assigning CBF-values to white matter (WM), gray matter (GM) and cerebrospinal fluid (CSF).

Fig. 2 shows the proposed framework for synthetically generating training and validation datasets. For this goal, we leverage the well-known Buxton kinetic model [1], which has been defined for normal ASL, and defined a tracer delivery function (for tissue voxels and arteries) and a tracer accumulation (perfusion) function for each bolus of Hadamard encoded labeling scheme. The final kinetic model is then generated by performing the convolution of the AIF and the residue function. Equations (1) and (2) define the obtained model for large arteries and tissue signals for a Hadamard scheme of 8 encoding steps respectively. These equations can be generalized for Hadamard matrices of higher rank.


in which is label duration for the sub-bolus, is the number of sub-boluses, is the magnetization of arterial blood, is AAT which represents the arrival time of the labeled blood in the artery, is BAT which represents the arrival time of labeled blood in the tissue, is the arterial blood relaxation time, is CBF (millimeter per gram per second), is static tissue signal, aCBV is arterial cerebral blood volume,




is if the bolus in the row is control, and it is if the bolus in the row is label. For voxels containing large arteries, the pCASL signal can be computed by . The calculated tracer kinetic model is a function of AAT, BAT and CBF. In this study, the anatomic structures are obtained from the BrainWeb database [2]. In order to obtain the tracer signal, the AAT and BAT and blood maps are extracted from in vivo data, then registered with a subject from the BrainWeb dataset by Elastix [14]. The ground truth, i.e. 4D MRA and perfusion scans, are obtained by normal Hadamard decoding of the pCASL images [9]. To evaluate the generated data, the signal evolution pattern was validated by the Buxton curve model [1].

Figure 2: Data generator framework for one subject, the inputs of the framework (shown in bold) are: in vivo information (include BAT, AAT, and CBF) and anatomical data from BrainWeb [2], the outputs are pCASL scans which by decoding the perfusion and angiographic maps are obtained (see Section 2.2), Left: a map of the arrival time of the label in arteries (’arterial arrival time’ or AAT) is created from high resolution 4D MRA ASL scan (Cinema, 8 time-points with 200ms temporal resolution, and a spatial resolution of mm) and the data are registered to a subject from BrainWeb dataset, and leads to an AAT map for the subject, then the AAT map is fed into Eq (1) to calculate the kinetic arterial model. Middle pipeline: using the GM and WM segmentation of the subject and assigning literature values to the flow map (or CBF) of GM and WM the flow map is calculated, which serves as one of the inputs to the kinetic model of the tissue (Eq 2). Right: an in vivo te-crushed pCASL data is registered to the same subject of BrainWeb and by Hadamard decoding the registered data, the arrival of the label at tissue level (’bolus arrival time’, or BAT-map) is calculated. This serves as the other input to the kinetic model of the tissue (Eq 2). The tissue signal and arterial signal are summed together to form the te-pCASL.

In this study, the dataset was generated for a Hadamard-8 matrix with seven blocks of respectively 1300, 600, 400, 400, 400, 300 and 300 ms with an additional 265 ms delay before the start of readout. Using the permuted in vivo information (include 6 BAT, 4 AAT) and registering those with the anatomical information from the BrainWeb dataset (consisting of 20 normal subjects and CBF) and calculating the Hadamard te-pCASL (equations (1) and (2)), this study contains 1564 distinct simulated data-sets each including crushed and non-crushed input data for 8 Hadamard-encodings. By decoding each of the generated crushed and non-crushed te-pCASL data, the corresponding angiographic and perfusion output data at 7-time points, as the ground truth, are obtained. The scans were divided into 1096 subjects for training, 155 for validation and 313 for testing.

3 Experimental Results

We implemented the proposed networks in Google’s Tensorflow. The patch extraction was done parallel and randomly using a multi-threaded daemon process on the CPU and then patches were fed to the network on the GPU during the training process. To tackle the sparsity of MRA with respect to the perfusion scans, 75 percent of the patches were extracted from the region containing arteries. The input patches were augmented by white noise extracted from a Gaussian distribution with zero mean and random standard deviation between 0 and 5, left-to-right flipping, and random rotation (up to


Evaluation of the proposed networks has been performed by calculating SSIM, MSE, signal to noise ratio, and peak signal to noise ratio (pSNR), comparing the ground truth reconstruction using full sampling with that of the neural network using subsampling. Table 1 tabulates a quantitative comparison between the mentioned loss functions. Fig. 3 depicts the boxplots of the metrics on the test set. The network with SSIM loss function with a value of has the highest value for SSIM metric, and the network with ML-SSIM loss function with has the second rank for perfusion reconstruction. While the network with ML-SSIM loss function with the values of , and for MSE, SNR and pSNR respectively, has a better performance than its competitors. Also, this network with the values of , and for respectively SSIM, MSE and SNR, exceeded the other mentioned networks for angiography reconstruction. Fig. 4 exemplifies the qualification compassion for 4D MRA and perfusion reconstruction between the different CNNs. It takes an average of 205232 ms from the ML-SSIM network to reconstruct all perfusion and angiography scans from the interleaved sparsely-sampled crushed and non-crushed data of size .

Loss function SSIM MSE SNR pSNR of param
91.60 0.17 1.15 32.24 169,152
69.34 2.28 0.09 5.95
91.59 0.17 1.30 32.19 169,152
96.34 2.60 5.38 0.09
91.66 0.92 1.29 33.15 169,152
96.15 2.64 0.06 5.33
91.96 0.18 1.33 21.07 181,692
96.38 2.40 0.14 5.74
Table 1: Comparison of the different networks for perfusion and angiographic images (in gray the perfusion and in white the angiographic results), the best results for perfusion and angiography are shown in blue and green respectively, PL stands for perceptual loss
Figure 3: Boxplots for different metrics and CNNs for perfusion and angiographic results, PL stands for VGG-16 perceptual loss.


(a) 4D MRA

(b) 4D perfusion
Figure 4: Qualification results, 4D a) MRA and b) perfusion and error at multiple time points after labeling of arterial spins in a single-slice for the different networks, GT stands for ground truth.

4 Conclusion

We proposed a 3D end-to-end fully convolutional CNN for accelerating 4D MRA and perfusion reconstruction from half-sampled crushed and non-crushed pCASL data. We leveraged loop connectivity patterns in the network architecture to improve the flow of information during the gradient updates. For training and validation purposes we developed a data generator framework based on the generalized kinetic model for the pCASL signal. The generated dataset included 1096 scans for training, 155 scans for validation and 313 for testing. The proposed network with ML-SSIM loss function achieved a SSIM of , MSE of , SNR of , and pSNR of for perfusion/angiography reconstruction.

In conclusion, the proposed network obtained promising results for the challenging problem of 4D MRA and perfusion reconstruction. The method, therefore, may assist an accelerated MRI scanning workflow. A further step of this study is enriching the training and validation datasets with in vivo data.

Acknowledgements. This work is financed by the Netherlands Organization for Scientific Research (NWO), VICI project 016.160.351.


  • [1] Buxton, R.B., Frank, L.R., Wong, E.C., Siewert, B., Warach, S., Edelman, R.R.: A general kinetic model for quantitative perfusion imaging with arterial spin labeling. MRM 40(3), 383–396 (1998)
  • [2] Cocosco, C.A., Kollokian, V., Kwan, R.K.S., Pike, G.B., Evans, A.C.: Brainweb: Online interface to a 3d mri simulated brain database. In: NeuroImage. Citeseer (1997)
  • [3]

    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. TPAMI

    38(2), 295–307 (2015)
  • [4]

    Elmahdy, M.S., Jagt, T., Zinkstok, R.T., Qiao, Y., Shahzad, R., Sokooti, H., Yousefi, S., Incrocci, L., Marijnen, C., Hoogeman, M., et al.: Robust contour propagation using deep learning and image registration for online adaptive proton therapy of prostate cancer. Medical physics (2019)

  • [5] Ferlay, J., Soerjomataram, I., Dikshit, R., Eser, S., Mathers, C., Rebelo, M., Parkin, D.M., Forman, D., Bray, F.: Cancer incidence and mortality worldwide: sources, methods and major patterns in globocan 2012. International journal of cancer 136(5), E359–E386 (2015)
  • [6] Gong, E., Pauly, J., Zaharchuk, G.: Boosting snr and/or resolution of arterial spin label (asl) imaging using multi-contrast approaches with multi-lateral guided filter and deep networks. In: Proceedings of the Annual Meeting of the International Society for Magnetic Resonance in Medicine, Honolulu, Hawaii (2017)
  • [7] Gong, K., Guan, J., Kim, K., Zhang, X., Yang, J., Seo, Y., El Fakhri, G., Qi, J., Li, Q.: Iterative pet image reconstruction using convolutional neural network representation. TMI 38(3), 675–685 (2018)
  • [8] Guo, J., Gong, E., Goubran, M., Fan, A.P., Khalighi, M.M., Zaharchuk, G.: Improving perfusion image quality and quantification accuracy using multi-contrast mri and deep convolutional neural networks. In: ISMRM, Paris, France (2018)
  • [9] Hirschler, L., Munting, L.P., Khmelinskii, A., Teeuwisse, W.M., Suidgeest, E., Warnking, J.M., van der Weerd, L., Barbier, E.L., van Osch, M.J.: Transit time mapping in the mouse brain using time-encoded pcasl. NMR in Biomedicine 31(2), e3855 (2018)
  • [10] Ho, K.C., Scalzo, F., Sarma, K.V., El-Saden, S., Arnold, C.W.: A temporal deep learning approach for mr perfusion parameter estimation in stroke. In: 23rd ICPR. pp. 1315–1320. IEEE (2016)
  • [11] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR. pp. 4700–4708 (2017)
  • [12] Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: CVPR. pp. 11–19 (2017)
  • [13] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV. pp. 694–711. Springer (2016)
  • [14] Klein, S., Staring, M., Murphy, K., Viergever, M.A., Pluim, J.P.: Elastix: a toolbox for intensity-based medical image registration. TMI 29(1), 196–205 (2010)
  • [15] van Osch, M.J., Teeuwisse, W.M., Chen, Z., Suzuki, Y., Helle, M., Schmid, S.: Advances in arterial spin labelling mri methods for measuring perfusion and collateral flow. Journal of Cerebral Blood Flow & Metabolism 38(9), 1461–1480 (2018)
  • [16] Yousefi, S., Sokooti, H., Elmahdy, M.S., Peters, F.P., Shalmani, M.T.M., Zinkstok, R.T., Staring, M.: Esophageal gross tumor volume segmentation using a 3d convolutional neural network. In: MICCAI. pp. 343–351. Springer (2018)
  • [17] Zhao, L., Fielden, S.W., Feng, X., Wintermark, M., Mugler III, J.P., Meyer, C.H.: Rapid 3d dynamic arterial spin labeling with a sparse model-based image reconstruction. Neuroimage 121, 205–216 (2015)