An Unsupervised Deep Learning Method for Parallel Cardiac MRI via Time-Interleaved Sampling

12/20/2019 ∙ by Yanjie Zhu, et al. ∙ 0

Deep learning has achieved good success in cardiac magnetic resonance imaging (MRI) reconstruction, in which convolutional neural networks (CNNs) learn the mapping from undersampled k-space to fully sampled images. Although these deep learning methods can improve reconstruction quality without complex parameter selection or a lengthy reconstruction time compared with iterative methods, the following issues still need to be addressed: 1) all of these methods are based on big data and require a large amount of fully sampled MRI data, which is always difficult for cardiac MRI; 2) All of these methods are only applicable for single-channel images without exploring coil correlation. In this paper, we propose an unsupervised deep learning method for parallel cardiac MRI via a time-interleaved sampling strategy. Specifically, a time-interleaved acquisition scheme is developed to build a set of fully encoded reference data by directly merging the k-space data of adjacent time frames. Then these fully encoded data can be used to train a parallel network for reconstructing images of each coil separately. Finally, the images from each coil are combined together via a CNN to implicitly explore the correlations between coils. The comparisons with classic k-t FOCUSS, k-t SLR and L+S methods on in vivo datasets show that our method can achieve improved reconstruction results in an extremely short amount of time.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 4

page 5

page 7

page 8

page 9

page 10

page 11

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Cardiac magnetic resonance (CMR) imaging is a noninvasive imaging technique that can be used to evaluate cardiac function and ventricular wall motion abnormalities, providing rich information for the clinical diagnosis of heart conditions [9]. However, cardiac motion adversely affects the quality of MR images and therefore limits the temporal and spatial resolution of cardiac MR imaging [22], especially in some cardiac diseases, such as tachycardia. Therefore, it is important to accelerate cardiac MR imaging without sacrificing image quality.

Usually, fast CMR approach requires priori information to remove the aliasing artifacts caused by the violation of the Nyquist sampling theorem [14]. The advanced approach is compressed sensing (CS)/Low-rank(LR) [3, 24]. Amounts of CS/LR-based methods has been proposed to accelerate cardiac MR imaging [15, 33, 21, 37, 26, 23, 25, 30]. For example, k-t FOCUSS [15] took advantage of the sparsity of x-f support to reconstruct x-f images from the undersampled k-t space. And k-t ISD [21] incorporated additional information on the support of the dynamic image in x-f space based on the theory of CS with partially known support. Wang et al [37] employed a 3D patch-based spatiotemporal dictionary for sparse representations of dynamic image sequences. A typical example of low-rank is L+S [26], in which the nuclear norm was used to enforce low rank in L and the L1 norm was used to enforce sparsity in S. And k-t SLR [23] exploited the correlations in a dynamic imaging dataset by modeling the data to have a compact representation in the Karhunen-Louve transform (KLT) domain. These methods have greatly improved the spatiotemporal resolution of dynamic MR imaging. However, these methods take a relatively long time during the iterative solution procedure to achieve high-quality reconstruction, and the selection of the regularization parameter is empirical. Additionally, most of these approaches exploit a priori information only from the to-be-reconstructed images or from very few reference images [20].

Recently, deep learning based methods have been proposed and successfully applied to MR imaging [36, 18, 13, 41, 7, 32, 28, 29, 27, 38, 31, 12, 5, 2, 6]. There are mainly two categories of deep learning-based fast MRI: (1) end-to-end learning methods [36, 18, 13, 41, 7, 32, 28, 29, 27, 38] and (2) model-based unrolling methods [31, 12, 5, 2]. The first category utilizes the information from big data to train a universal network for learning the mapping between the undersampled and fully sampled data pairs in an end-to-end manner. For example, in [36], a plain convolutional neural network was trained to learn the mapping relationship between undersampled brain MR images and fully sampled brain MR images. AUTOMAP [41]

used a combination of fully connected networks and CNNs to learn the mapping from undersampled k-space to reconstructed image. The model-based methods unroll the iterations of the optimization algorithm to the neural network so that the network can automatically learn the hyperparameters or transformations in the optimization algorithm. Typical model-based networks includes ADMM-Net

[31], VN-Net [12], Learned PD [5], etc. Model-based unrolling methods often achieve better reconstruction quality with less data than end-to-end learning methods [6]. There are mainly three works for dynamic MR imaging, namely DC-CNN [29], CRNN [27], DIMENSION [38]. DC-CNN proposed a deep cascade of convolutional neural networks to accelerate the data acquisition process by combining data consistency and data sharing approaches. CRNN simultaneously learned the spatio-temporal dependencies of cardiac image series by exploiting bidirectional recurrent hidden connections across time sequences. DIMENSION developed a multi-supervised network training technique to simultaneously constrain both the frequency and the spatial domain information to improve the reconstruction accuracy. However, all three of these methods require a large amount of fully sampled cardiac MR images as the ground truth. Collections of these fully sampled images are always difficult to obtain, especially breath-holding and regular heart rhythms are required in the acquisition.

In addition to priori information, the spatial variance coil sensitivity provided by the phase array coil also plays an important role in fast MRI. Parallel MRI utilizes the coil sensitivity to accelerate MR image acquisition and has been widely used in clinical scans. However, most CMR deep learning reconstruction studies utilize simulated single-channel k-space data to train the network. These approaches may lose exploration of coil correlation and more importantly, cannot be applied to real MR scans.

To solve the above issues, we propose an unsupervised deep learning framework for parallel cardiac MRI in this paper. A time-interleaved acquisition scheme is designed that can build a set of fully encoded reference data by merging adjacent time frames. These fully encoded data can be used to train a parallel network for reconstructing the image of each coil separately. Our developed model-based reconstruction network (ADMM-Net-III) was employed in our study. Finally, the coil correlations are explored, and the coil images are combined together via another CNN. Our contributions could be summarized as follows:

  1. We propose an unsupervised framework for dynamic MRI. In our framework, the acquisition of fully sampled data for network training is no longer needed, which is one of the greatest difficulties in deep learning-based cardiac imaging, especially breath-holding conditions and regular heart rhythms are required in the acquisition scheme. This is the first time that an unsupervised approach has been applied to dynamic MR imaging.

  2. Dynamic MR images have a lot of redundancy in the temporal direction. In the proposed unsupervised framework, a time-interleaved acquisition scheme is used, and the signals from directly adjacent time frames can be merged to build a set of fully encoded reference data for network training. In this way, temporal redundancy can be effectively utilized.

  3. Different from other deep learning methods for dynamic MRI, which focus on single-channel MRI, we propose a parallel imaging strategy. The proposed parallel imaging technique has three advantages. First, a model-based unrolling method has been employed in our study, which can achieve better reconstruction quality with less data. To the best of our knowledge, this is the first time to apply model-based unrolling method to dynamic MR imaging. Second, unlike other deep learning-based methods using single-channel signals as input and output, our parallel network focuses on multichannel scenario and could explore coil correlations. Third, multichannel data have a less complex distribution than single-channel data, which undoubtedly decreases the difficulty of network learning. To the best of our knowledge, this is the first time that a parallel imaging network has been applied to dynamic MRI.

  4. Although our proposed framework is based on the time-interleaved sampling scheme, network training and testing can be performed to any sampling patterns. The time-interleaved sampling scheme is used only during the data preparation phase. Once the fully encoded training data are constructed, retrospective undersampling is no longer dependent on the time-interleaved sampling pattern. Moreover, the model trained on one sampling pattern can be well generalized to other sampling patterns.

  5. The experimental results show that the proposed method is superior to conventional CS-based methods such as k-t FOCUSS, k-t SLR and L+S with a much shorter runtime. These findings demonstrate the effectiveness of the unsupervised learning and the parallel network in cardiac MRI.

The rest of this paper is organized as follows. Section II states the problem and the proposed methods. Section III summarizes experimental details and the results to demonstrate the effectiveness of the proposed method, while the discussion and conclusions are presented in Section IV and Section V, respectively.

Ii Methodology

Ii-a Problem Formulation

The goal of our work is to estimate an unknown image from undersampled k-space data. Specifically for 2D cardiac imaging, reconstruction was performed by solving the following optimization problem:

(1)

Here is the encoding operator. is the 2D dynamic image series and is the corresponding multichannel k-space data. is a prior regularization of . is a regularization parameter. The first term is the data fidelity, which ensures that the k-space of reconstructed data is consistent with the actual measurements. The second term is often referred to as prior regularization. In CS-based methods, is usually a sparse prior of in the temporal dimension.

In CNN-based methods, is a CNN prior, which forces to match the output of the networks:

(2)

where is the output of the networks under the parameters . The purpose of the network training process is to find the optimal parameters . Once the network is trained, the networks’ output is the reconstruction we want. The data fidelity term is important to achieve high quality reconstruction. Therefore, data consistency (DC) layers are often introduced in the CNN-based methods.

Ii-B The Proposed Unsupervised Framework

The proposed unsupervised learning framework for cardiac MRI is shown in Fig.1. To simplify the symbols, we omit the channel dimension in this section, and it should be noted that Fig.1 is specific to each coil. The whole unsupervised framework can be divided into three components:

Fig. 1: The proposed unsupervised learning framework for dynamic MRI via time-interleaved sampling. It should be noted that except for the output of the network in a coil-combined signal, all the other signals are multichannel signals. In the data preparation stage, fully encoded data are built by directly merging adjacent time frames in a time-interleaved acquisition scheme. Then the fully encoded data produced data pairs of the network’s input and output for network training. In the test stage, in vivo undersampled k-space data are fed into the trained neural network to reconstruct the 2D dynamic MR images.
  1. Data preparation: Undersampled k-space data are acquired according to a time-interleaved acquisition scheme. The time-interleaved acquisition scheme is shown in Fig.2. Adjacent time frames can be merged to build a complete set of k-space data, which called fully encoded k-space. Once the fully encoded dataset is built, the data pairs of the network input and output can be obtained by retrospectively undersampling the fully encoded data with a designed sampling mask. Although the time-interleaved sampling scheme is not limited to uniform or random sampling, for the convenience of the experiments, we focus on uniform time interleaved sampling. Also, more neighboring frames can be averaged to increase the SNR of the full-encoded data.

  2. Network training: The proposed parallel neural network will be described in the next section. The training datasets obtained above can be fed into the networks for network training. The input of the network is multichannel underencoded k-space data and the output is the coil-combined fully encoded image. Although the training datasets are synthetic in this stage, they can effectively represent true fully sampled data. More importantly, the temporal redundancies have been utilized through the construction of this dataset.

  3. Online test: In the test stage, the true in vivo undersampled k-space data are fed into the trained network to reconstruct the 2D dynamic MR images.

Fig. 2: The time-interleaved acquisition scheme. Two different undersampled patterns (uniform+random) at 5-fold acceleration are given via a time-interleaved sampling scheme. In these two examples, at least five adjacent time frames need to be merged to build a complete set of k-space data. In general, more neighboring frames can be averaged to increase the SNR of the fully encoded data.

In summary, we use time-interleaved sampling data to synthesize fully encoded data as references to realize unsupervised learning. This framework has many advantages: (a) there is no need for a fully sampled dynamic MR dataset; (b) coil correlations can be explored; (c) there is no need for coil sensitivity maps; (d) the time redundancies are utilized. In Section III and Section IV, we will demonstrate the effectiveness of this framework with abundant experiments.

Ii-C The Proposed Parallel Network

For the parallel reconstruction of underencoded multichannel k-space data, we propose a novel parallel neural network, as shown in Fig.3. The underencoded multichannel k-space data are fed into the parallel network. For each coil, there is a separate network to reconstruct the coil data. We applied ADMM-Net-III [6] as the reconstruction network because it is a model-based unrolling method that can obtain high-quality reconstruction results with less data. ADMM-Net-III is the generalized version of ADMM-Net [21]. The solution iterations of ADMM-Net-III can be written as:

(3)

In ADMM-Net-III, the operators , , and the parameter are all learned by the network, while only the priori regularization and the parameter are learned in ADMM-Net [31]. The generalization process and implementation results of ADMM-Net-III could be found in [6]. An excessive explanation of ADMM-Net-III is beyond the scope of this article. Different from the original ADMM-Net-III model, each ADMM-Net-III model is embedded with a data consistency (DC) layer in this paper because our previous exploration [35] showed that the data consistency layer could effectively improve the reconstruction quality. Then all of these network reconstructed coil images are concatenated and fed into another CNN to explore the coil correlations and implement coil combination.

Overall, the proposed parallel network has two components: a reconstruction network to reconstruct each coil image, and a coil-combination network to explore coil correlations and combine all the coil images together. Specifically, unlike other methods [29, 27, 38] using single-channel signals as input and output, our parallel network focuses on multichannel scenario and could explore coil correlations. Another advantage to deal with coil images is that single-channel data have a complex distribution [8], which undoubtedly increases the difficulty of network learning. To visually display the statistical distribution of the single-channel and multichannel images, statistical histograms of both are also provided in Fig.4, from which we can see that the coil images have a simpler statistical distribution than the combined single-channel image. In Section III.B, we compare a single-channel model with a multichannel model under the proposed unsupervised framework and find that the multichannel model achieves better reconstruction results.

Fig. 3: The proposed parallel neural networks for MR reconstruction in a coil-by-coil manner.

Fig. 4: The histograms of three coil images and a single-channel image. From (a) to (d): the 11th coil image, the 12th coil image, the 15th coil image and the single-channel image obtained by directly calculating the sum of squares (sos). (e) to (h) are histograms of these images, respectively. The data are normalized to [0, 255] for convenient display.

Iii Experimental Results

Iii-a Setup

Iii-A1 Data acquisition

We collected 386 2D dynamic (2Dt) fully sampled cardiac MR data from 30 healthy volunteers using a 3T scanner (SIEMENS MAGNETOM Trio) with a balanced steady-state free precession (bSSFP) sequence. Written informed consent was obtained from all the subjects. Each scan contains a single-slice bSSFP acquisition with 25 temporal frames. Retrospectively electrocardiogram ECG-gated segmented imaging was conducted, and each slice was acquired in one breath-hold of 15-20 sec. The following parameters were used for the bSSFP scans: FOV mm, acquisition matrix , slice thickness = 6 mm, TR/TE = 3.0 ms/1.5 ms and 20 receiving coils. We randomly selected 25 volunteers for training and the rest for testing. Deep learning typically require a large amount of data for training [19]. Therefore, some data augmentation strategies were applied. We sheared the original images along the x, y and t directions. The sheared size was (

), and the stride along the three directions is 12, 12 and 5 respectively. Finally, we obtained 2149 2Dt multichannel cardiac MR data of size

() for data preparation and 603 data for testing.

In this work, we focus on a time-interleaved acquisition scheme with a uniform sampling pattern, which is shown in Fig.2 (left). For each original k-space data, we retrospectively undersampled it with 16 ACS lines. Specifically, we fully sampled the frequency-encodes (along ) and uniformly undersampled the phase encodes ( along ) according to the time-interleaved scheme. Although fully sampled k-space data are available in our acquisition, they are unseen to the proposed unsupervised learning framework, which are only used to obtain the undersampled data retrospectively. We merged all the frames of the undersampled k-space data rather than the adjacent frames and averaged them to obtain the high SNR fully encoded training data. This also has brought some other benefits, such as the elimination of temporal redundancy, and the requirement of GPU memory is reduced.

Iii-A2 Network training

For network training, we divided each data into two channels for storing real and imaginary parts of the data respectively. Therefore, the inputs of the network are underencoded multichannel k-space data , and the outputs are the coil-combined reconstructed images . The hyperparameters in the network are set as follows: for each ADMM-Net-III, the number of iterations is , the numbers of convolution kernels are set as shown in Fig.3 and the size of each convolution kernel is . Xavier initialization [10]

was used to initialize the network weights. Rectifier linear units (ReLU)

[11]

was selected as the nonlinear activation functions. The minibatch size was 4. The exponential decay learning rate

[40]

was used in all the CNN-based experiments, and the initial learning rate was set to 0.001 with a decay of 0.98. The loss function used in this work was the mean squared error (MSE). All the models were trained by the Adam optimizer

[16] with parameters , and .

The models were implemented on an Ubuntu 16.04 LTS (64-bit) operating system equipped with an Intel Xeon E5-2640 Central Processing Unit (CPU) and Tesla TITAN Xp Graphics Processing Unit (GPU, 12 GB memory) in the open framework TensorFlow

[1]

with CUDA and CUDNN support. The network training took approximately 56 hours and 100 epochs.

Iii-A3 Model configuration

There are many alternative options in our proposed unsupervised framework, for example: 1. In the data preparation stage, the time-interleaved sampling scheme is not limited to uniform or random sampling patterns; 2. In the training stage, the retrospectively undersampling mask is not limited to uniform or random sampling patterns; 3. In the test stage, the trained model can have a good generalization ability to other sampling patterns. Discussing all the cases will make the article very lengthy. Therefore, for the convenience of the experiments, we only experimented on typical cases without loss of generality. The model configurations (sampling patterns and acceleration) are arranged in TABLE I. A 1D Gaussian random undersampling pattern [15], which is one of the most common protocols in the CS/LR based methods, was applied in this paper. Specifically, we fully sampled frequency encodes (along ) and randomly undersampled the phase encodes (along ) according to a zero-mean Gaussian variable density function.

Time-interleaved Training Testing Acceleration
Section III.B Uniform Random Random 4
Section III.C Uniform Random Random/Uniform 4/8
Section IV.A Uniform Random Random 4
Section IV.B Uniform Random Random 4
Section IV.C Uniform Uniform Random/Uniform 4
TABLE I: The model configuration of each section.

Iii-B Does the Proposed Parallel Network Work?

Currently, all three deep learning methods [29, 27, 38] for cardiac MRI use single-channel data for network training and testing. In this section, we refer to them as single-channel methods and correspondingly refer to the proposed parallel imaging method as the multichannel method. We will explore whether the multichannel method exhibits superior reconstruction performance. To ensure a fair comparison, we explored the effectiveness of the single-channel method and multichannel method based on the same network structure (ADMM-Net-III) under the unsupervised framework proposed in this paper. Although we give only the unsupervised scheme in the multichannel case in Fig.1, this unsupervised scheme can be conveniently changed to the single-channel case, because the operations in Fig.1 focus on the temporal dimension and have nothing to do with the coil dimension. The differences between the two models exist only in two respects. First, the raw materials for data preparation are different: one is multichannel fully sampled k-space data, while the other is single-channel fully sampled k-space data by the adaptively combining the above multichannel k-space data [34]. Second, the single-channel model no longer requires the network for combining the coils, so it contains only the reconstruction part.

We trained the model with a 1D random Gauss sampling mask at 4-fold acceleration, and the reconstruction results of the three subjects are shown in Fig.5, which clearly shows that the multichannel model can restore more details (as shown by red arrows) than our single-channel model. The single-channel model not only loses more detail than the multichannel model, but the reconstruction results are also more blurring.

Fig. 5: Three examples of the reconstruction results of the single-channel and multichannel models at 4-fold acceleration. The first row (the results for the first subject) shows, from left to right, the ground truth, the zero-filling image, the reconstruction result of the single-channel method, the reconstruction result of the multichannel method and the enlarged view of the error map (display ranges [0, 0.10]) of their respective heart regions framed by a yellow box . The second and third rows are the experimental results for the other two subjects, respectively. The y-t image (extraction of the 124th slice along the y and temporal dimensions) is also given for each signal to show the reconstruction performance in the temporal dimension.

Iii-C Comparisons to the State-of-the-art Methods

To demonstrate the efficacy of the proposed unsupervised learning method, we compared it with several state-of-the-art CS/LR methods including k-t FOCUSS [15], k-t SLR [23], and L+S [26]. We adjusted the parameters of the competing methods to their best performance. A 1D random Gauss mask was used for training and testing. And to explore the generalization performance of the proposed method with different sampling patterns, we also tested the trained model with a 1D uniform undersampling pattern. The reconstruction results of these methods at 4-fold acceleration are shown in Fig.6. The reconstruction results of three CS-based methods contain fewer structural details and more artifacts than the reconstruction results of the proposed method. We also enlarged the error maps of the cardiac region for demonstration, which show that our method has the best reconstruction performance in the cardiac region, especially the details marked by the red arrow. The y-t images, which were extracted from the 124th slice along the y and temporal dimensions, also clearly illustrate the comparable performance of the proposed method. The red numbers represent the reconstruction time of these methods. Our method has the shortest reconstruction time, which is hundreds of times shorter than that of the other methods. Longitudinal comparisons (comparisons of the reconstruction results of the two masks) show that the reconstruction results under the random mask are better than those under the uniform mask. This conclusion is in line with expectations, because CS-based methods perform better at random mask than uniform masks, and for the deep learning approach, training and testing with the same type of mask could yield better reconstruction results than training and testing with different types of masks. The reconstruction results of the different methods at 8-fold acceleration are shown in Fig.7. At 8- fold acceleration, the same conclusion is reached as with 4-fold acceleration.

Fig. 6: The reconstruction results of the different methods (k-t FOCUSS, k-t SLR, L+S and the proposed method) at 4-fold acceleration under a 1D random mask and 1D uniform mask. The proposed model is trained under a 1D random mask. The first and second rows are the reconstruction results and the corresponding error maps, respectively, with display ranges [0, 0.25] in the case of a 1D random mask. The third and fourth rows are the reconstruction results and the corresponding error maps, respectively, in the case of a 1D uniform mask. To show the reconstruction performance in the temporal dimension, the y-t image (extraction of the 124th slice along the y and temporal dimensions) is given for each signal with display ranges [0, 0.5].

Fig. 7: The reconstruction results of the different methods (k-t FOCUSS, k-t SLR, L+S and the proposed method) at 8-fold acceleration under a 1D random mask and a 1D uniform mask. The proposed model is trained under a 1D random mask. The first and second rows are the reconstruction results and the corresponding error maps, respectively, with display ranges [0, 0.25] in the case of a 1D random mask. The third and fourth rows are the reconstruction results and the corresponding error maps, respectively, in the case of a 1D uniform mask. To show the reconstruction performance in the temporal dimension, y-t image (extraction of the 124th slice along y and temporal dimensions) is given for each signal with display ranges [0, 0.5].

Though we only show two Cartesian sampling patterns (uniform and random Gauss) in this paper, our method can be easily adopted to other sampling patterns. This finding also reflects one of the advantages of deep learning-based approaches: they have no such strict requirements for sampling masks. In another example, Knoll et al [17] also applied a network that was trained from regular undersampling mask to random undersampling with good reconstruction performance. Further exploration will be carried out in future studies.

Iv Discussion

Iv-a Options of the Coil Reconstruction Network

Although we introduced the proposed parallel network in Section II.C, where ADMM-Net-III was selected as the reconstruction network, the whole unsupervised framework is not limited to a specific reconstruction network. The reason to choose ADMM-Net-III is that it is a deep learning model-based unrolling method, which requires less data than vanilla end-to-end methods and usually exhibits superior reconstruction performance to other methods. In this section, we compared the reconstruction results of ADMM-Net-III with those of DC-CNN [29], which is one of the state-of-the-art deep learning methods for dynamic MRI. We focused on a D5C5 model, which works well for the DC-CNN model and consists of five blocks (C5) with each block containing five convolutional layers (D5). We trained two models under the proposed unsupervised framework. The reconstruction results are shown in Fig.8. Both subjects have consistent observations: the coil reconstruction network using the ADMM-Net-III model has smaller artifacts and more details, especially in the heart region (as can be seen from the red and yellow arrows in the error maps, and for obvious comparisons, the display range was narrowed down to [0, 0.07]) than that using the DC-CNN model.

Fig. 8: Cardiac MR reconstruction results under the proposed framework with DC-CNN/ADMM-Net-III at 4-fold acceleration. The test results from two volunteers are shown (from left to right): the label, zero-filling image, reconstruction result of the DC-CNN, reconstruction result with the ADMM-Net-III and the enlarged view of the error maps with display ranges [0, 0.07] of their respective heart regions framed by the yellow box. The y-t image (extraction of the 124th slice along the y and temporal dimensions) is given for each signal.

Iv-B The Importance of the Data Consistency Layer in the Parallel Network

As shown in Fig.3, each ADMM-Net-III model is followed by a data consistency (DC) layer. In [35], we have noted that the DC layer is very important for reconstruction in the single-channel case. Aggarwal et al [2] noted that the system function in the single-channel undersampled MRI acquisition is simple, where

is a sampling matrix obtained by keeping only the relevant rows of an identity matrix and

is the Fourier matrix. In this case, the DC can be analytically computed, as in, for example, [7, 29, 27, 38]. However, is not analytically invertible for complex operators such as multichannel MRI. In this case, an iterative optimization algorithm is needed to approximate DC. For example, [2] proposed to solve DC using a conjugate gradient optimization scheme. In this paper, we developed another method. Although the proposed method is for multichannel MRI, due to the coil-by-coil implementation, DC still can be analytically computed for every channel. So in this method, is DC still effective and important, especially followed by a coil-combination network? To answer this question, we built two models under the proposed framework. The only difference between the two models is whether they contain DC layers. The training configurations of these two models remain the same. The reconstruction results are shown in Fig.9. All three subjects have consistent observations: the model including DC layers has smaller artifacts and more details than the other, especially in the heart region (as can be seen from the red and yellow arrows in the error maps). Therefore, we can conclude that the DC layer still plays a very important role in reconstruction in multichannel MRI with coil-by-coil implementation.

Fig. 9: Cardiac MRI reconstruction results under the proposed framework with/without DC layers at 4-fold acceleration. The test results from three volunteers are shown (from left to right): the label, zero-filling image, reconstruction result without DC layers, reconstruction result with DC layers and the enlarged view of error maps with display ranges [0, 0.10] of their respective heart region framed by the yellow box. The y-t image (extraction of the 124th slice along the y and temporal dimensions) is given for each signal.

Iv-C Training the Model Under Other Sampling Patterns

The models in the above sections are all trained under the time-interleaved acquisition scheme with a 1D random sampling pattern. Although our proposed framework is based on the time-interleaved sampling scheme, the network training and testing can be applied to any sampling patterns. The time-interleaved sampling scheme is used only during the data preparation phase. Once the fully encoded training data are constructed, retrospectively undersampling is no longer dependent on the time-interleaved sampling pattern. Moreover, the model trained on one sampling pattern can be well generalized to other sampling patterns. In Section III.C, the 1D Gaussian random undersampling pattern is adopted. In particular, we trained the model under a 1D Gaussian random undersampling pattern and tested it under 1D random and 1D uniform undersampling patterns. In this section, we trained the model under a 1D uniform undersampling pattern and tested it under 1D random and 1D uniform undersampling patterns. The reconstruction results at 4-fold acceleration are shown in Fig.10. Our method achieves superior reconstruction results with both undersampling patterns, especially in the heart region, which is marked by the red arrow.

Fig. 10: The reconstruction results of the different methods (k-t FOCUSS, k-t SLR, L+S and the proposed method) at 4-fold acceleration under a 1D random mask and 1D uniform mask. The proposed model is trained under a 1D uniform mask. The first and second rows are the reconstruction results and the corresponding error maps, respectively, with display ranges [0, 0.25] in the case of a 1D random mask. The third and fourth rows are the reconstruction results and the corresponding error maps, respectively, in the case of a 1D uniform mask. To show the reconstruction performance in the temporal dimension, the y-t image (extraction of the 124th slice along the y and temporal dimensions) is given for each signal with display ranges [0, 0.5].

Iv-D The Limitations of the Proposed Work

Although our method has many advantages (i.e., superior reconstruction results and the shortest reconstruction time) compared with other state-of-the-art methods, there is still a certain degree of smoothness in the reconstructed images. The reason may be that the loss function we chose is the MSE. The MSE loss has a limited ability to perceive image structure information because it indicates only the mean square information between the reconstructed image and the ground truth. DAGAN [39] couples an adversarial loss with an innovative content loss to reconstruct CS-MRI images, which could preserve perceptual image details. This property motivates us to use more detail-friendly loss functions in future works.

In TGRAPPA [4], more neighboring frames could be averaged to increase the SNR of the fully encoded data. Inspired by this finding, we average all the frames to obtain the highest SNR. This method yielded some benefits, such as the elimination of temporal redundancies and the GPU memory requirement is reduced. However, there are some inconveniences. For example, the temporal correlations are underutilized, and many time-dependent network configurations are not available. In the current GPU condition (12 GB memory), averaging all the frames to obtain only one frame is necessary because the GPU resources cannot meet the requirements of exploring the temporal and coil correlations at the same time. In the future, with the improvement in the hardware conditions, more high-dimensional exploration is expected to further improve the reconstruction of dynamic MR images.

V Conclusion and Outlook

In this paper, we propose an unsupervised deep learning method for parallel MR cardiac imaging via time-interleaved sampling. In our framework, fully sampled reference data are no longer required for network training. The temporal redundancies can be effectively utilized via the proposed data preparation process. We also propose a coil-by-coil parallel imaging technology with many advantages. To the best of our knowledge, this is the first time that a parallel imaging network has been applied to dynamic MR imaging. Although our proposed framework is based on the time-interleaved sampling scheme, the model can be applied to any sampling patterns. The experimental results show that the proposed method is superior to conventional CS-based methods such as k-t FOCUSS, k-t SLR and L+S in an extremely short amount of time. These findings demonstrate the effectiveness of the unsupervised learning and the parallel network in cardiac MR imaging.

Acknowledgment

This research was partly supported by the National Natural Science Foundation of China (61771463, 81830056, U1805261, 81971611, 61871373, 81729003, 81901736); National Key RD Program of China (2017YFC0108802 and 2017YFC0112903); Natural Science Foundation of Guangdong Province (2018A0303130132); Shenzhen Key Laboratory of Ultrasound Imaging and Therapy (ZDSYS20180206180631473); Shenzhen Peacock Plan Team Program (KQTD20180413181834876); Innovation and Technology Commission of the government of Hong Kong SAR (MRP/001/18X); Strategic Priority Research Program of Chinese Academy of Sciences (XDB25000000).

References

  • [1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. (2016)

    Tensorflow: a system for large-scale machine learning

    .
    In 12th Symposium on Operating Systems Design and Implementation), pp. 265–283. Cited by: §III-A2.
  • [2] H. K. Aggarwal, M. P. Mani, and M. Jacob (2018) Modl: model-based deep learning architecture for inverse problems. IEEE transactions on medical imaging 38 (2), pp. 394–405. Cited by: §I, §IV-B.
  • [3] R. G. Baraniuk (2007) Compressive sensing. IEEE signal processing magazine 24 (4). Cited by: §I.
  • [4] F. A. Breuer, P. Kellman, M. A. Griswold, and P. M. Jakob (2005) Dynamic autocalibrated parallel imaging using temporal GRAPPA (TGRAPPA). Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 53 (4), pp. 981–985. Cited by: §IV-D.
  • [5] J. Cheng, H. Wang, L. Ying, and D. Liang (2019) Model learning: primal dual networks for fast MR imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 21–29. Cited by: §I.
  • [6] J. Cheng, H. Wang, Y. Zhu, Q. Liu, L. Ying, and D. Liang (2019) Model-based Deep MR Imaging: the roadmap of generalizing compressed sensing model using deep learning. arXiv preprint arXiv:1906.08143. Cited by: §I, §II-C.
  • [7] T. Eo, Y. Jun, T. Kim, J. Jang, H. Lee, and D. Hwang (2018) KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magnetic resonance in medicine 80 (5), pp. 2188–2201. Cited by: §I, §IV-B.
  • [8] Z. Fan, L. Sun, X. Ding, Y. Huang, C. Cai, and J. Paisley (2018-09) A segmentation-aware deep fusion network for compressed sensing MRI. In

    The European Conference on Computer Vision (ECCV)

    ,
    Cited by: §II-C.
  • [9] J. P. Finn, K. Nael, V. Deshpande, O. Ratib, and G. Laub (2006) Cardiac MR imaging: state of the technology. Radiology 241 (2), pp. 338–354. Cited by: §I.
  • [10] X. Glorot and Y. Bengio (2010) Understanding the difficulty of training deep feedforward neural networks. In

    Proceedings of the thirteenth international conference on artificial intelligence and statistics

    ,
    pp. 249–256. Cited by: §III-A2.
  • [11] X. Glorot, A. Bordes, and Y. Bengio (2011) Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315–323. Cited by: §III-A2.
  • [12] K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and F. Knoll (2018) Learning a variational network for reconstruction of accelerated MRI data. Magnetic resonance in medicine 79 (6), pp. 3055–3071. Cited by: §I.
  • [13] Y. Han, J. Yoo, H. H. Kim, H. J. Shin, K. Sung, and J. C. Ye (2018) Deep learning with domain adaptation for accelerated projection-reconstruction MR. Magnetic resonance in medicine 80 (3), pp. 1189–1205. Cited by: §I.
  • [14] A. J. Jerri (1977) The Shannon sampling theorem—Its various extensions and applications: a tutorial review. Proceedings of the IEEE 65 (11), pp. 1565–1596. Cited by: §I.
  • [15] H. Jung, J. C. Ye, and E. Y. Kim (2007) Improved k–t BLAST and k–t SENSE using FOCUSS. Physics in Medicine & Biology 52 (11), pp. 3201. Cited by: §I, §III-A3, §III-C.
  • [16] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §III-A2.
  • [17] F. Knoll, K. Hammernik, E. Kobler, T. Pock, M. P. Recht, and D. K. Sodickson (2019)

    Assessment of the generalization of learned image reconstruction and the potential for transfer learning

    .
    Magnetic resonance in medicine 81 (1), pp. 116–128. Cited by: §III-C.
  • [18] K. Kwon, D. Kim, and H. Park (2017)

    A parallel MR imaging method using multilayer perceptron

    .
    Medical physics 44 (12), pp. 6209–6224. Cited by: §I.
  • [19] Y. LeCun, Y. Bengio, and G. Hinton (2015) Deep learning. Nature 521 (7553), pp. 436–444. Cited by: §III-A1.
  • [20] D. Liang, J. Cheng, Z. Ke, and L. Ying (2019) Deep magnetic resonance image reconstruction: inverse problems meet neural networks. IEEE Signal Processing Magazine. External Links: Document Cited by: §I.
  • [21] D. Liang, E. V. DiBella, R. Chen, and L. Ying (2012) K-t ISD: dynamic cardiac MR imaging using compressed sensing with iterative support detection. Magnetic resonance in medicine 68 (1), pp. 41–53. Cited by: §I.
  • [22] Z. P. Liang (2007) Spatiotemporal imaging with partially separable functions. In Proceedings of 4th IEEE international symposium on biomedical imaging (ISBI), pp. 988–991. Cited by: §I.
  • [23] S. G. Lingala, Y. Hu, E. DiBella, and M. Jacob (2011) Accelerated dynamic MRI exploiting sparsity and low-rank structure: kt SLR. IEEE transactions on medical imaging 30 (5), pp. 1042–1054. Cited by: §I, §III-C.
  • [24] M. Lustig, D. Donoho, and J. M. Pauly (2007) Sparse MRI: the application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 58 (6), pp. 1182–1195. Cited by: §I.
  • [25] U. Nakarmi, Y. Wang, J. Lyu, D. Liang, and L. Ying (2017) A kernel-based low-rank (KLR) model for low-dimensional manifold recovery in highly accelerated dynamic MRI. IEEE Transactions on Medical Imaging PP (99), pp. 1–1. Cited by: §I.
  • [26] R. Otazo, E. Candes, and D. K. Sodickson (2015) Low-rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components. Magnetic Resonance in Medicine 73 (3), pp. 1125–1136. Cited by: §I, §III-C.
  • [27] C. Qin, J. V. Hajnal, D. Rueckert, J. Schlemper, J. Caballero, and A. N. Price (2019)

    Convolutional recurrent neural networks for dynamic MR image reconstruction

    .
    IEEE Transactions on Medical Imaging 38 (1), pp. 280–290. Cited by: §I, §II-C, §III-B, §IV-B.
  • [28] T. M. Quan, T. Nguyen-Duc, and W. Jeong (2018) Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss. IEEE transactions on medical imaging 37 (6), pp. 1488–1497. Cited by: §I.
  • [29] J. Schlemper, J. Caballero, J. V. Hajnal, A. N. Price, and D. Rueckert (2018) A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging 37 (2), pp. 491–503. Cited by: §I, §II-C, §III-B, §IV-A, §IV-B.
  • [30] G. N. Shetty, K. Slavakis, A. Bose, U. Nakarmi, G. Scutari, and L. Ying (2019) Bi-linear modeling of data manifolds for dynamic-MRI recovery. IEEE Transactions on Medical Imaging. Cited by: §I.
  • [31] J. Sun, H. Li, Z. Xu, et al. (2016) Deep ADMM-Net for compressive sensing MRI. In Advances in neural information processing systems, pp. 10–18. Cited by: §I, §II-C.
  • [32] L. Sun, Z. Fan, Y. Huang, X. Ding, and J. Paisley (2018) Compressed sensing MRI using a recursive dilated network. In Thirty-Second AAAI Conference on Artificial Intelligence, Cited by: §I.
  • [33] J. Tsao, P. Boesiger, and K. P. Pruessmann (2003) K-t BLAST and k-t SENSE: dynamic MRI with high frame rate exploiting spatiotemporal correlations. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 50 (5), pp. 1031–1042. Cited by: §I.
  • [34] D. O. Walsh, A. F. Gmitro, and M. W. Marcellin (2000) Adaptive reconstruction of phased array MR imagery. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 43 (5), pp. 682–690. Cited by: §III-B.
  • [35] S. Wang, Z. Ke, H. Cheng, L. Ying, X. liu, H. Zheng, and D. Liang (2019) Investigation of convolutional neural network based deep learning for cardiac imaging. In Proceedings of the 26th annual meeting of ISMRM, Paris, Cited by: §II-C, §IV-B.
  • [36] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang (2016) Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 514–517. Cited by: §I.
  • [37] Y. Wang and L. Ying (2013) Compressed sensing dynamic cardiac cine MRI using learned spatiotemporal dictionary. IEEE transactions on Biomedical Engineering 61 (4), pp. 1109–1120. Cited by: §I.
  • [38] Wang,Shanshan, Z. Ke, H. Cheng, S. Jia, L. Ying, H. Zheng, and D. Liang (2019) DIMENSION: dynamic MR imaging with both k-space and spatial prior knowledge obtained via multi-supervised network training. NMR in Biomedicine e4131. External Links: Document Cited by: §I, §II-C, §III-B, §IV-B.
  • [39] G. Yang, S. Yu, H. Dong, G. Slabaugh, P. L. Dragotti, X. Ye, F. Liu, S. Arridge, J. Keegan, Y. Guo, et al. (2017) DAGAN: deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE transactions on medical imaging 37 (6), pp. 1310–1321. Cited by: §IV-D.
  • [40] M. D. Zeiler (2012) ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. Cited by: §III-A2.
  • [41] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen (2018) Image reconstruction by domain-transform manifold learning. Nature 555 (7697), pp. 487. Cited by: §I.