Magnetic resonance imaging (MRI) is an important medical imaging technique, but its slow imaging speed poses a limitation on its widespread application. Compressed sensing (CS) theory [2, 4] has been a significant development of the signal acquisition and reconstruction process that has allowed for significant acceleration of MRI. The CS-MRI problem can be formulated as the optimization
where is the complex-valued MRI to be reconstructed, is the under-sampled Fourier matrix and () are the k-space data measured by the MRI machine. The first data fidelity term ensures agreement between the Fourier coefficients of the reconstructed image and the measured data, while the second term regularizes the reconstruction to encourage certain image properties such as sparsity in a transform domain.
Recently, deep learning approaches have been introduced for the CS-MRI problem, achieving state-of-the-art performance compared with conventional methods. For example, an end-to-end mapping from input zero-filled MRI to a fully-sampled MRI was trained using the classic CNN model in, or its residual network variant in . Greater integration of the data fidelity term into the network has resulted in a Deep Cascade CNN (DC-CNN) . Compared with previous models proposed for CS-MRI inversion, deep learning is able to capture more intricate patterns within the data, which leads to their improved performance.
Recently, the compressed sensing MRI is approved by the Food and Drug Administration (FDA) to two main MRI vendors: GE and Siemens . As the growing needs for application of compressed sensing MRI, improving reconstruction accuracy of the CS-MRI is of great significance. In this paper, we propose a deep learning framework in which an arbitrary CS-MRI inversion algorithm is combined with a deep learning error correction network. The network is trained for a specific inversion algorithm to exploit structural consistencies in the errors they produce. The final reconstruction is found by combining the information from the original algorithm with the error correction of the network.
Ii Related Work
Much focus of previous work has been on proposing appropriate regularizations that lead to better MRI reconstructions. In the pioneering work of CS-MRI called SparseMRI , this regularization adds an penalty on the wavelet coefficients and the total variation of the reconstructed image. Based on SparseMRI, more efficient optimization methods have been proposed to optimize this objective, such as TVCMRI , RecPF  and FCSA . Variations on the wavelet penalty exploit geometric information of MRI, such as PBDW/PBDWS [22, 20] and GBRWT , for improved results. Dictionary learning methods [24, 11, 16, 25] have also been applied to CS-MRI reconstruction, as have nonlocal priors such as NLR , PANO  and BM3D-MRI . These previous works can be considered sparsity-promoting regularized CS-MRI methods that are optimized using iterative algorithms. They also represent images using simple single layer features that are either predefined (e.g., wavelets) or learned from the data (e.g., dictionary learning).
, the original signal is decomposed into a predicted signal and a residual signal. The predicted signal is estimated by temporal averaging, while the highly sparse residual signal has a-norm regularization. An iterative feature refinement strategy for CS-MRI was proposed in  to exploit the structural error produced in each iteration. In , the k-space measurements are divided into high and low frequency regions and reconstructed separately. In  the MR image is decomposed into a smooth layer and a detail layer which are estimated using total variation and wavelet regularization separately. In , the low frequency information is estimated using parallel imaging techniques. These methods each employ a fixed transform basis.
Iii Problem Formulation
Exploiting structural regularities in the reconstruction error of CS-MRI is a good approach to compensate for imperfect modeling. Starting with the standard formulation of CS-MRI in Equation 1, we formulate our objective function as
where is an intermediate reconstruction of the MRI. We model this intermediate reconstruction as the summation of a “guidance” image and the error image of the reconstruction ,
Substituting this into Equation 2, we obtain
The guidance image is the reconstructed MRI using any chosen CS-MRI method; thus can be formed using existing software prior to using our proposed method for the final reconstruction. The reconstruction error is between the ground truth and the reconstruction . Since we don’t know this at testing time, we use training data to model this error image with a neural network , where represents the network parameters and is the input to the network. Thus, Equation 4 can be rewritten as
For a new MRI, after obtaining the guidance image (using a pre-existing algorithm) and the mapping
(using a feed-forward neural network trained on data), the proposed framework produces the final output MRI by solving the least square problem of Equation5.
Iv Deep Error Correction Network (DECN)
Following the formulation of our CS-MRI framework above and in Figure 1, we turn to a more detailed discussion of the optimization procedure. We next discuss each module of the proposed Deep Error Correction Network (DECN) framework.
Iv-a Guide Module
With the guide module, we seek a reconstruction of the MRI that approximates the fully-sampled MRI using a standard “off-the-shelf” CS-MRI approach. We denote this as
We illustrate with reconstructions for three CS-MRI methods: TLMRI (transform learning MRI) , PANO (patch-based nonlocal operator)  and GBRWT (graph-based redundant wavelet transform) . The PANO and GBRWT models achieve impressive reconstruction qualities because they use an nonlocal prior and adaptive graph-based wavelet transform to exploit image structures. In TLMRI, the sparsifying transform learning and the reconstruction are performed simultaneously in more efficient way than DLMRI . The three methods represent the state-of-the-art performance in the non-deep CS-MRI models. In Figure 2, we show the reconstructions error for zero-filled (itself a potential reconstruction “algorithm”), TLMRI, PANO and GBRWT on a complexed-valued brain MRI using Cartesian under-sampling. The error display ranges from 0 to 0.2 with normalized data. The parameter setting will be elaborated in the Experiment Section V.
We also consider the deep learning DC-CNN model  as the guide module. We also give the reconstruction error in Figure 2. We observe the zero-filled, TLMRI, PANO, GBRWT and DC-CNN models all suffer the structural reconstruction errors, while the DC-CNN model achieves the highest reconstruction quality with minimal errors because of its powerful model capacity. Another advantage of this CNN model is that, once the network is trained, testing is very fast compared with conventional sparse-regularization CS-MRI models. This is because no iterative algorithm needs to be run for optimization during testing since the operations are a simple feed forward function of the input. We compare the reconstruction time of TLMRI, PANO, GBRWT and DC-CNN for testing for Figure 2 in Table I.
Iv-B Error Reconstruction Module
Using the guidance image , we can train a deep error correction module on the residual. To perform this task, we need access during training to pairs of the true, fully sampled MRI , as well as its reconstruction found by manually undersampling the k-space of this image according to a pre-defined mask and inverting. We then optimize the following objective function over network parameter ,
where indicates the reconstructed MRI using zero-filled and the input to the error correction module is the concatenation of the zero-filled MRI and the guidance MRI . Therefore, the error-correcting network is learning how to map the concatenation of the zero-filled, compressively sensed MRI and the guidance image to the residual of the true MRI using a corresponding off-the-shelf CS-MRI inversion algorithm. Now we give the rationales and explanations for the concatenation operation.
In the CS-MRI inversions, the zero-filled MR images usually serve as the starting point in the iterative optimization. Although the iterative de-aliasing can effectively remove the artifacts and achieve much more pleasing visual quality compared with zero-filled reconstruction, the distortion and information loss is inevitable in the reconstruction. To further illustrate this phenomenon, we compare the pixel-wise reconstruction errors among the zero-filling reconstruction and other reconstruction models of the MR image in Figure 2.
We take the difference between the absolute reconstruction error of zero-filled and the compared CS-MRI methods and only keep the nonnegative values, which can be formulated as
Where the operator set the negative values to zero. We only keep the nonnegative values in the map, which results the filtered difference map. We show the corresponding filtered difference map in figure 3 in the range [0 0.2]. The bright region means the better accuracy of zero-filled reconstruction. We observe the zero-filling reconstruction provide better reconstruction accuracy on some regions, indicating the information loss in the reconstruction occurs.
To alleviate the information loss in the guide module, we introduce the concatenation operation to utilize the information from both the zero-filled MR image and guidance image as the input to the error correction network. In later Experiment Section V, we further validate it by the ablation study.
We again note that the network is paired with a particular inversion algorithm , since each algorithm may have unique and consistent characteristics in the errors they produce. The network can be any deep learning network trained using standard methods.
Iv-C Data Fidelity Module
After the error correction network is trained, for a new undersampled k-space data for which the true is unknown, we use its corresponding guidance image and the approximated reconstructed error to optimize the data fidelity module by solving the following optimization problem
The data fidelity module is utilized in our proposed DECN framework to correct the reconstruction by enforcing greater agreement at the sampled k-space locations. Using the properties of the fast Fourier transform (FFT), we can simplify the optimization by working in the Fourier domain using the common technique described in, e.g.,. The optimal values for in k-space can be found point-wise. This yields the closed-form solution
The regularization parameter is usually set very small in the noise-free environment. We found that worked well in our low-noise experiments.
The proposed DECN model can effectively exploiting the reconstruction residues. In later Experiment Section V, we further validate the network architecture via ablation study on the error correction strategies.
In this section, we present experimental results using complex-valued MRI datasets. The T1 weighted MRI dataset (size ) is acquired on 40 volunteers with total 3800 MR images at Siemens 3.0T scanner with 12 coils using the fast low angle shot (FLASH) sequence (TR/TE = , 220 field of view, slice thickness). The SENSE reconstruction is introduced to compose the gold standard full k-space, which is used to emulate the single-channel MRI. The details can be found in . We randomly select 80 MR images as training set and 20 MR images as testing set. Informed consent was obtained from the imaging subject in compliance with the Institutional Review Board policy. The magnitude of the full-sampled MR image is normalized to unity by dot dividing the image by its largest pixel magnitude.
Under-sampled k-space measurements are manually obtained via Cartesian and Random sampling mask with random phase encodes. Different undersampling ratios are adopted in the experiments.
V-a Network Architecture
|Sampling Pattern||Cartesian Under-sampling||Random Under-sampling|
For the deep guide module (i.e., learning ), we use the CNN architecture called deep cascade CNN (DC-CNN) 
, where the non-adjustable data fidelity layer is also incorporated into the model. This guide module consists of four blocks. Each block is formed by four consecutive convolutional layers with a shortcut and a data fidelity layer. For each convolutional layer, except the last one within a block, there are total of 64 feature maps. We use ReLU
as the activation function.
For the error correction module (i.e., learning ), we adopt the network architecture shown in Figure 1. There are 18 convolutional layers with a skip layer connection as proposed in [8, 9] to alleviate the gradient vanish problem. We again adopt ReLU as the activation function, except for the last layer where the identity function is used to allow negative values. All convolution filters are set to
with stride set to.
V-B Experimental Setup
We train and test the two deep algorithms using Tensorflow
for the Python environment on a NVIDIA GeForce GTX 1080 with 8GB GPU memory. Padding is applied to keep the size of features the same. We use the Xavier method to initialize the network parameters, and we apply ADAM  with momentum. The implementation uses the initialized learning rate , first-order momentum and second momentum . The weight decay regularization parameter is set to . The size of training batch is . We report our performance after training iteration of DC-CNN guide module and iterations of error correction module.
In the guidance module, we implement the state-of-the-art CS-MRI models with the following parameter settings. In TLMRI , we set the data fidelity parameter , the patch size , the number of training signals , the sparsity fraction , the weight on the negative log-determinat+Frobenius norm terms , the patch overlap stride , the DCT matrix is used as initial transform operator, the iterations times for optimization. The above parameter setting follows the advices from the author . In PANO , we use the implementation with parallel computation provided by . The data fidelity parameter is set with zero-filled MR image as initial reference image. The non-local operation is implemented twice to yield the MRI reconstruction. In GBRWT , we set the data fidelity parameter . The Daubechies redundant wavelet sparsity is used as regularization to obtain the reference image. The graph is trained times.
V-C Ablation Study
To validate the architecture of the proposed DECN model, we conduct the ablation study by comparing the DECN framework with other Baseline network architectures in Figure 4, which we refer the model in Figure 4(a) as DECN with no input concatenation and error correction (DECN-NIC-NEC). With the guide module, a later cascaded CNN module learns the mapping from the pre-reconstructed MR image to the full-sampled MR image. Likewise, we name the models in Figure 4(b) (DECN-IC-NEC) and Figure 4(c) (DECN-NIC-EC). By comparing the DECN-NIC-NEC framework with the DECN-IC-NEC framework, we evaluate the benefit brought by the concatenating the zero-filled MR images and corresponding guide MR images as the input to compensate the information loss in the guide module. In Figure 3, we give the illustration the information from zero-filled MR images and guide images can be shared. By comparing the DECN-NIC-NEC framework with the DECN-NIC-EC framework, we evaluate how the error correction strategy improves the reconstruction accuracy compared with simple cascade manner.
We conduct the experiments using PANO with the Cartesian under-sampling mask shown in Figure 6 as the guide module. We give the averaged PSNR (peak signal-to-noise ratio) results in Figure 5. We observe the PANO-DECN-IC-NEC and PANO-DECN-NIC-EC both outperforms the PANO-DECN-NIC-NEC with the similar margins about 0.2dB in PSNR. While the proposed PANO-DECN model with the input concatenation and error correction outperforms the PANO-DECN-NIC-NEC about 0.5 dB in PSNR. The ablation study shows the input concatenation and error correction strategies can effectively improve the model performance in the DECN framework.
We evaluate the proposed DECN framework using PSNR and SSIM (structural similarity index)  as quantitative image quality assessment measures. We give the quantitative reconstruction results of all the test data on different under-sampling patterns and different under-sampling ratios in Table II. We show the Cartesian under-sampling mask in Figure 6(b) and the Random under-sampling mask in Figure 7(b). We observe that DECN improved all off-the-shelf CS-MRI inversion methods on all the under-sampling patterns. Since the Random mask enjoys the more incoherence than the Cartesian mask with the same under-sampling ratio, the CS-MRI achieves better reconstruction quality on the Random masks. Also, we observe the plain DC-CNN model already achieves good reconstruction accuracy, leaving less structural errors for the error correction module, leading to the limited performance improvement about 0.1 dB on the Random and masks. While for other CS-MRI inversions on various sampling patterns, the improvements are at least 1.5dB or even up to 3.5 dB.
In Figure 6, we show reconstruction results and the corresponding error images of an example from the test data on the 1D under-sampling mask. With local magnification on the red box, we observe that by learning the error correction module, the fine details, especially the low-contrast structures are better preserved, leading to a better reconstruction.
In Figure 7, we also compare the MR images produced by the TLMRI, PANO and GBRWT with their DECN counterparts on the 2D under-sampling mask. The subjective comparison DC-CNN and DE-CNN-DECN isn’t included because of the limited improvement. The results are consistent with our observation in Cartesian under-sampling case.
We have proposed a deep error correction framework for the CS-MRI inversion problem. Using any off-the-shelf CS-MRI algorithm to construct a template, or “guide” for the final reconstruction, we use a deep neural network that learns how to correct for errors that typically appear in the chosen algorithm. Experimental results show that the proposed model achieves consistently improves a variety of CS-MRI inversion techniques.
The work is supported in part by National Natural Science Foundation of China under Grants 61571382, 81671766, 61571005, 81671674, U1605252, 61671309 and 81301278, Guangdong Natural Science Foundation under Grant 2015A030313007, Fundamental Research Funds for the Central Universities under Grant 20720160075, 20720150169, CCF-Tencent research fund, Natural Science Foundation of Fujian Province of China (No.2017J01126), and the Science and Technology funds from the Fujian Provincial Administration of Surveying, Mapping, and Geoinformation.
-  M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
-  E. J. Candes, J. Romberg, and T. Tao. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2):489–509, Feb 2006.
-  W. Dong, G. Shi, X. Li, Y. Ma, and F. Huang. Compressive sensing via nonlocal low-rank regularization. IEEE Transactions on Image Processing, 23(8):3618–3632, Aug 2014.
-  D. L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289–1306, April 2006.
-  E. M. Eksioglu. Decoupled algorithm for MRI reconstruction using nonlocal block matching model: BM3D-MRI. Journal of Mathematical Imaging and Vision, 56(3):430–440, Nov 2016.
-  J. A. Fessler. Medical image reconstruction: a brief overview of past milestones and future directions. arXiv preprint arXiv:1707.05927, 2017.
X. Glorot and Y. Bengio.
Understanding the difficulty of training deep feedforward neural
In Y. W. Teh and M. Titterington, editors,
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of
Proceedings of Machine Learning Research, pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
-  K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In , pages 770–778, June 2016.
-  K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630–645. Springer, 2016.
-  J. Huang, S. Zhang, and D. Metaxas. Efficient MR image reconstruction for compressed MR imaging. Medical Image Analysis, 15(5):670 – 679, 2011. Special Issue on the 2010 Conference on Medical Image Computing and Computer-Assisted Intervention.
-  Y. Huang, J. Paisley, Q. Lin, X. Ding, X. Fu, and X. P. Zhang. Bayesian nonparametric dictionary learning for compressed sensing MRI. IEEE Transactions on Image Processing, 23(12):5007–5019, Dec 2014.
-  H. Jung, K. Sung, K. S. Nayak, E. Y. Kim, and J. C. Ye. k-t FOCUSS: A general compressed sensing framework for high resolution dynamic MRI. Magnetic Resonance in Medicine, 61(1):103–116, 2009.
-  D. Kingma and J. Ba. ADAM: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
-  Z. Lai, X. Qu, Y. Liu, D. Guo, J. Ye, Z. Zhan, and Z. Chen. Image reconstruction of compressed sensing MRI using graph-based redundant wavelet transform. Medical Image Analysis, 27(Supplement C):93 – 104, 2016. Discrete Graphical Models in Biomedical Image Analysis.
-  D. Lee, J. Yoo, and J. C. Ye. Deep residual learning for compressed sensing MRI. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pages 15–18, April 2017.
-  Q. Liu, S. Wang, L. Ying, X. Peng, Y. Zhu, and D. Liang. Adaptive dictionary learning in sparse gradient domain for image recovery. IEEE Transactions on Image Processing, 22(12):4652–4663, Dec 2013.
-  M. Lustig, D. Donoho, and J. M. Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine, 58(6):1182–1195, 2007.
-  S. Ma, W. Yin, Y. Zhang, and A. Chakraborty. An efficient algorithm for compressed MR imaging using total variation and wavelets. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, June 2008.
-  V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 807–814, 2010.
-  B. Ning, X. Qu, D. Guo, C. Hu, and Z. Chen. Magnetic resonance image reconstruction using trained geometric directions in 2D redundant wavelets domain and non-convex optimization. Magnetic Resonance Imaging, 31(9):1611 – 1622, 2013.
-  S. Park and J. Park. Compressed sensing MRI exploiting complementary dual decomposition. Medical Image Analysis, 18(3):472 – 486, 2014.
-  X. Qu, D. Guo, B. Ning, Y. Hou, Y. Lin, S. Cai, and Z. Chen. Undersampled MRI reconstruction with patch-based directional wavelets. Magnetic Resonance Imaging, 30(7):964 – 977, 2012.
-  X. Qu, Y. Hou, F. Lam, D. Guo, J. Zhong, and Z. Chen. Magnetic resonance image reconstruction from undersampled measurements using a patch-based nonlocal operator. Medical Image Analysis, 18(6):843 – 856, 2014. Sparse Methods for Signal Reconstruction and Medical Image Analysis.
-  S. Ravishankar and Y. Bresler. MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Transactions on Medical Imaging, 30(5):1028–1041, May 2011.
-  S. Ravishankar and Y. Bresler. Efficient blind compressed sensing using sparsifying transforms with convergence guarantees and application to magnetic resonance imaging. SIAM Journal on Imaging Sciences, 8(4):2519–2557, 2015.
-  J. Schlemper, J. Caballero, J. V. Hajnal, A. Price, and D. Rueckert. A deep cascade of convolutional neural networks for MR image reconstruction. In International Conference on Information Processing in Medical Imaging, pages 647–658. Springer, 2017.
-  K. Sung and B. A. Hargreaves. High-frequency subband compressed sensing MRI using quadruplet sampling. Magnetic Resonance in Medicine, 70(5):1306–1318, 2013.
-  S. Wang, J. Liu, Q. Liu, L. Ying, X. Liu, H. Zheng, and D. Liang. Iterative feature refinement for accurate undersampled MR image reconstruction. Physics in Medicine & Biology, 61(9):3291, 2016.
-  S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang. Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pages 514–517, April 2016.
-  Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, April 2004.
-  J. Yang, Y. Zhang, and W. Yin. A fast alternating direction method for TVL1-L2 signal reconstruction from partial fourier data. IEEE Journal of Selected Topics in Signal Processing, 4(2):288–297, April 2010.
-  Y. Yang, F. Liu, W. Xu, and S. Crozier. Compressed sensing MRI via two-stage reconstruction. IEEE Transactions on Biomedical Engineering, 62(1):110–118, Jan 2015.
-  J. C. Ye, S. Tak, Y. Han, and H. W. Park. Projection reconstruction MR imaging using FOCUSS. Magnetic Resonance in Medicine, 57(4):764–775, 2007.