Towards Ultrafast MRI via Extreme k-Space Undersampling and Superresolution

03/04/2021 ∙ by Aleksandr Belov, et al. ∙ Philips Skoltech 0

We went below the MRI acceleration factors (a.k.a., k-space undersampling) reported by all published papers that reference the original fastMRI challenge, and then considered powerful deep learning based image enhancement methods to compensate for the underresolved images. We thoroughly study the influence of the sampling patterns, the undersampling and the downscaling factors, as well as the recovery models on the final image quality for both the brain and the knee fastMRI benchmarks. The quality of the reconstructed images surpasses that of the other methods, yielding an MSE of 0.00114, a PSNR of 29.6 dB, and an SSIM of 0.956 at x16 acceleration factor. More extreme undersampling factors of x32 and x64 are also investigated, holding promise for certain clinical applications such as computer-assisted surgery or radiation planning. We survey 5 expert radiologists to assess 100 pairs of images and show that the recovered undersampled images statistically preserve their diagnostic value.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

page 6

page 13

page 17

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Magnetic Resonance Imaging (MRI) is a non-invasive imaging modality that offers excellent soft tissue contrast and does not expose the patient to ionizing radiation. The raw MRI signal is recorded in the so-called k

-space, with the digital images being then generated by Fourier transform. The filling of the

k-space typically lasts 15–60 minutes, during which the patient must remain motionless (problematic for children, neurotic or claustrophobic patients). If a patient cannot stay still, a motion artifact will appear on the images, oftentimes demanding a complete re-scan [8]. Furthermore, the long acquisition time limits the applicability of MRI to dynamic imaging of the abdomen or the heart [11, 25] and decreases the throughput of the scanner, leading to higher costs [3].

Contrasting chemicals [5] and physics-based acceleration approaches [20, 24] alleviate the challenge and have been a subject of active research over the last two decades, solidifying the vision of the ‘Ultrafast MRI’ as the ultimate goal. A parallel pursuit towards the same vision is related to compressed sensing [25, 11, 5], where the methods to compensate for the undersampled k-space by proper image reconstruction have been proposed. This direction of research experienced a noticeable resurgence following the publication of the fastMRI benchmarks [29] and the proposal to remove the artifacts resulting from the gaps in the k-space by deep learning (DL) [14]. Given the original settings defined by the challenge, the majority of groups have been experimenting with the reconstruction for the acceleration factors of 2, 4, and 8, with only rare works considering stronger undersampling of 16 and 32.

In this article, we further unholster the arsenal of DL and propose the use of powerful image-to-image translation models, originally developed for natural images, to tackle the poor quality of the underresolved MRI reconstructions. Specifically, we retrain Pix2Pix [12] and SRGAN [15] on the fastMRI data and study their performance alongside the reconstructing U-Net [22] at various downscaling and undersampling factors and sampling patterns. Our best model outperforms state-of-the-art (SOTA) at all conventional acceleration factors and allows to go beyond them to attempt extreme -space undersampling, such as .

2 Related work.

Several DL approaches have already been implemented to accelerate MRI. For instance, [27] proposed an algorithm that combines the optimal cartesian undersampling and MRI reconstruction using U-Net [22], which increased the fixed mask PSNR by dB and SSIM by . The automated transform by manifold approximation (AUTOMAP) [30] learns a transform between k-space and the image domain using fully-connected layers and convolutions layers in the image domain. The main disadvantage of AUTOMAP is a quadratic growth of parameters with the number of pixels in the image. Ref. [10]

presented an approach, based on the interpolation of the missing k-space data using low-rank Hankel matrix completion, that consistently outperforms existing image-domain DL approaches on several masks.

[19] used a cascade of U-Net modules to mimic iterative image reconstruction, resulting in an SSIM of and a PSRN of dB at acceleration on multi-coil data. As the winner of the challenge, this model is considered SOTA.

The most recent works tackle the same challenge by applying parallel imaging [6]

, generative adversarial networks

[2], ensemble learning with priors [16], trajectory optimization [26], and greedy policy search [1] (the only paper we found that considered the factor ). The issues of DL-based reconstruction robustness and the ultimate clinical value have been outlined in multiple works (e.g., [4, 26]), raising a reasonable concern of whether the images recovered by DL preserve their diagnostic value. Henceforth, we decided to engage a team of radiologists to perform such a user study to review our results.

3 Method

3.1 Data description

The Brain fastMRI dataset contains around anonymized MRI volumes provided by the NYU School of Medicine [29]

. We chose only T2 scans (as the largest class), padded, and downsized them to

pixels, discarding the smaller images. Finally, we selected slices of each MRI volume at mid-brain height to obtain similar brain surface area in each image. The resulting slices were split into train / validation / test subsets as 60% / 20% / 20%, making sure that slices from a given patient belong within the same subset. We applied min-max scaling of the intensities using the 2nd and the 98th percentiles.

We also validated our methods on the MRI scans of the knee. The same data preparation resulted in T2 slices, split into the subsets by the same proportion. Synovial liquid [5] noticeably shifted the upper percentiles in the knee data; thus, we normalized the intensities using the distance between the first and the second histogram extrema instead.

3.2 Models

Figure 1: Three possible approaches to downscale and undersample -space. The sampling masks details are given in the supplementary material.

An overview of the different models studied herein is shown in Fig. 1. Using the compressed sensing paradigm [5, 14, 25], we aim to accelerate MRI by sampling low resolution or/and partially filled k-space. The partial filling is accomplished by applying patterned masks on the Fourier transform of the original slices. We used the masks proposed by the fastMRI challenge (a central vertical band of of the value is fixed and the remaining bands are sampled randomly), as well as the radial and the spiral sampling trajectories that are popular in the field of compressed sampling [28]. The three types of masks considered and the effect of them on the images are shown in the supplementary material.

We used Pix2Pix [12] as an image-to-image translation method to correct the artifacts introduced by the k-space undersampling and SRGAN [15] to upscale the low-resolution images to their original size of . Interested in preserving the high-frequency components, we used the MSE of the feature maps obtained with the VGG16 pre-trained on ImageNet 31th layer for the perceptual loss, relying on the reports that convolutional networks pre-trained on ImageNet are still relevant for the perceptual losses of MRI images [13, 18, 21].

We retrained Pix2Pix and SRGAN (SR, for superresolution alone) entirely on the datasets described above. The combinations of SRGAN and U-Net generators (paths 3a-c in Fig. 1) were trained with SRGAN discriminator and a joint back-propagation. We also considered SRGAN for the direct reconstruction without the U-Net (path 2 in Fig. 1). The quality of the produced images was assessed using the MSE, PSNR and SSIM [17] metrics, similarly to SOTA [14, 19].

4 Experiments

We ran extensive series of experiments outlined in Fig. 1. In all SRGAN variants, our models always outperformed bicubic interpolation, which we confirmed for the , , and upscaling factors. We also observed that Pix2Pix models, following the undersampling with the radial and the spiral masks, always outperformed the fastMRI mask. The difference between the radial and the spiral masks, however, was less pronounced (see Tables 1 and 2). All experimental results are shown in the format of , where is the average metric value over the test set, and

is its standard deviation.


Acc. / Mask MSE, PSNR SSIM,
frac. of
k-space

50%
fastMRI 14.39 3.33 28.53 0.96 94.02 3.19
spiral 6.14 2.13 32.37 1.47 96.84 2.22
radial 5.08 1.83 33.20 1.47 97.70 1.74

25%
fastMRI 27.93 6.07 25.64 0.93 90.65 3.95
spiral 13.96 3.81 28.70 1.15 93.56 3.52
radial 11.81 4.62 29.51 1.37 94.46 3.93

12.5%
fastMRI 54.41 12.13 22.75 0.98 85.74 4.21
spiral 28.00 7.67 25.68 1.13 89.45 4.38
radial 21.12 5.45 26.89 1.08 91.23 4.61

6.25%
spiral 45.04 9.89 23.57 1.00 85.93 3.90
radial 42.21 9.78 23.87 1.07 87.12 4.53

3.125%
spiral 69.00 17.48 21.76 1.17 82.37 4.84
radial 65.62 19.70 22.02 1.31 83.06 5.76
Table 1: Pix2Pix model results.

Acc. / Models MSE, PSNR SSIM, frac. of k-space
25%
Bicubic 10.51 4.29 30.16 1.84 96.85 1.51
SRGAN 3.03 1.61 35.72 2.12 98.64 0.93
6.25%
Bicubic 43.94 14.66 23.84 1.58 88.48 3.17
SRGAN 11.69 4.21 29.58 1.49 95.63 2.35
1.5625%
Bicubic 121.59 29.76 19.30 1.21 73.04 4.17
SRGAN 28.76 9.67 24.37 1.13 88.49 3.32

Table 2: SRGAN (SR) model results.

Figure 2: Pix2Pix recovery results for k-space undersampling. The difference with the ground truth (multiplied by 3 for clarity) are shown on the right. Shown in the corner are SSIM and PSNR values. Note the inferior performance of original fastMRI masks.

Figure 2 compares the Pix2Pix reconstruction at undersampling for different sampling patterns. Visually, the reconstructions of all models look clear with sharp anatomic structures. However, the difference images reveal that the finest details are reconstructed best via the radial sampling. This naturally stems from the way the radial mask samples the highest spatial frequencies.

For the combination of architectures, we downscaled the images to pixels before applying k-space undersampling. We then trained SRGAN for both the upscaling and artifact removal or SRGAN+U-Net / U-Net+SRGAN, depending on the order of application of the corresponding generators. Visually, SRGAN alone performed worse than SRGAN+U-Net and U-Net+SRGAN combos, both of which seem to be statistically equivalent (see Tables 3, 4, and Figure 3). A comprehensive presentation of the results obtained at different acceleration factors can be found in the supplementary material.

Mask Model MSE, PSNR SSIM,
fastMRI SRGAN 12.98 3.79 29.04 1.22 94.84 2.55
SRGAN+U-Net 8.81 3.08 30.80 1.44 95.97 2.17
U-Net+SRGAN 9.27 3.28 30.56 1.41 95.69 2.69
spiral SRGAN 7.66 2.94 31.47 1.66 96.28 2.26
SRGAN+U-Net 7.02 2.84 31.88 1.73 96.51 2.06
U-Net+SRGAN 7.09 2.82 31.81 1.65 96.39 2.37
radial SRGAN 6.25 2.63 32.40 1.78 96.95 1.95
SRGAN+U-Net 6.05 2.64 32.58 1.87 97.01 1.87
U-Net+SRGAN 6.13 2.62 32.48 1.75 96.92 2.11
Table 3: Low-Resolution () model with undersampling, total acceleration: , fraction of k-space:

Mask Model MSE, PSNR SSIM,
fastMRI SRGAN 23.07 6.30 26.53 1.17 92.40 2.93
SRGAN+U-Net 15.97 4.77 28.15 1.26 94.00 2.61
U-Net+SRGAN 18.55 5.88 27.51 1.28 93.32 3.27
spiral SRGAN 17.29 5.05 27.79 1.20 93.32 2.88
SRGAN+U-Net 13.61 4.11 28.85 1.26 94.19 2.58
U-Net+SRGAN 13.49 4.18 28.89 1.27 94.08 3.39
radial SRGAN 12.71 4.07 29.18 1.38 94.54 2.79
SRGAN+U-Net 11.44 3.90 29.66 1.46 94.92 2.59
U-Net+SRGAN 11.39 3.92 29.67 1.42 94.79 3.23
Table 4: Low-Resolution () model with undersampling, total acceleration: , fraction of k-space:

Figure 3: Recovery of Low-Resolution k-space (undersampling with the radial mask). The subtraction from the ground truth (multiplied by 3 for clarity and shown on the right) suggests the SRGAN+U-Net combo to be the best model for MRI acceleration.

Our experiments show that undersampling with the radial mask, both for the Pix2Pix models and for the SRGAN+U-Net combo, always outperforms the other masks. Interestingly, SRGAN (SR) yields slightly better scores than the combo methods up to the acceleration (see Figure 7) despite the visual analysis where the combos are somewhat more convincing to the eye of an expert.

We applied the same models to the knee dataset, which, at first, had led to a low image reconstruction quality (both visually and quantitatively). We hypothesized that the low quality of reconstruction could be caused by the differences in the pixel intensities after the normalization, with the same tissue type having been mapped to a different intensity range on two neighboring slices in the volume. Indeed, swapping percentile normalization with the histogram-based normalization yielded more homogeneous results of pixel intensities across the knee slices, which, consequently, boosted the quality of image reconstruction to the level of the results on the brain data (see Figure 5 and 5).

Figure 4: Influence of the normalization method on the SSIM for the knee dataset with 16 acceleration and radial sampling.

Figure 5: Knee slice reconstructions with percentile-based (top) and histogram-based normalization (bottom).
Figure 6: SSIM dependence on the fraction of k-space for several best models, the brain dataset.
Figure 7: SSIM dependence on the fraction of k-space for several best models, the knee dataset.

5 User study

Image quality (IQ) assessment metrics such as SSIM help to estimate the reconstruction quality. However, they may not fully reflect how valuable these reconstructions are from the diagnostic perspective

[7]. To solve this problem, we conducted a user study of reconstruction quality with trained radiologists.

The study was performed on a test subset of the fastMRI dataset. We randomly selected brain and knee slices. Those slices were downscaled with a factor and undersampled by factors of and , using the radial masks. The resulting slices were evaluated for their diagnostic content, compared to their corresponding fully-sampled counterparts (the ground truth).

Five radiologists were involved in the study, using a simple survey tool [23]. The experts were shown the ground truth images and asked to rate the quality of our superresolved reconstructions based on three criteria vital for a typical diagnostic decision-making [5]

: the presence of artifacts, the signal-to-noise ratio (SNR), and the contrast-to-noise ratio (CNR). Each criterion was scored on a 4-point scale as follows: fully acceptable (4), rather acceptable (3), weakly acceptable (2), not acceptable (1).

Figure 8 shows high diagnostic value of the reconstructions, with around 80% of images getting the scores of (3) and (4) for both anatomies.

Figure 8: Diagnostic quality scores from a survey of radiologists based on three IQ parameters for brain and knee anatomies. The majority of votes fall into rather acceptable (3) and fully acceptable (4) categories, showing little discrepancy between anatomies.

6 Discussion and conclusion

We considered powerful image-to-image translation and superresolution models to compensate for the strong undersampling and downscaling of the k-space. Our study comprises a comprehensive evaluation of the undersampling patterns and factors, downscaling, and reconstruction models.

Being more symmetrical and homogeneous at high frequencies, the radial and the spiral undersampling trajectories outperform corresponding original fastMRI sampling masks (for an equivalent fraction of k-space). Unlike them, the radial and the spiral sampling trajectories guarantee that the sampling is omnidirectional, without any given region of the k-space being omitted, including the high-frequency ones. Following the radial or spiral sampling, the acquired data can be translated to the Cartesian grid according to the established apparatus in the compressed sensing [25]. It is natural, thus, to extrapolate our results to the other cutting-edge undersampling trajectories [25, 3, 26], which we anticipate to ‘heal’ similarly well by the Pix2Pix model.

We found that until the acceleration factor, SRGAN (SR) marginally outperforms the SRGAN+U-Net combo, and both of these approaches noticeably outperform Pix2Pix. The absence of significant differences between SRGAN (SR), SRGAN+U-Net, and U-Net+SRGAN at a given fraction of k-space suggests that the latter two can be adopted in general as the go-to acceleration method, allowing for more flexibility in terms of computation efficiency and image quality trade-off (downscaled images require fewer parameters for the final reconstructing generator, which may influence the selection of one model vs. the others for a particular application).

Different applications may indeed require a proper domain adaptation [9]. Herein, e.g.

, the range of intensities in the knee data was noticeably skewed by the joint’s liquids (the synovial fluid), requiring us to consider the histogram normalization approach in the data pre-processing stage (unlike percentile normalization sufficient for the brain data). More complex DL-based domain adaptation methods, translating the images between different MRI modalities and across different scanners

[9, 21], can also be conjoined to the proposed MRI acceleration pipeline and will be the subject of future work. Quantitatively, with the histogram-based normalization, SRGAN (SR) yielded an MSE of , a PSNR of dB, and an SSIM of at acceleration factor, which surpasses SOTA acceleration methods [14, 19].

Using the SRGAN+U-Net combo at acceleration, the reconstruction achieved an MSE of , a PSNR of dB, and an SSIM of . At acceleration, SRGAN+U-Net achieved an MSE of , a PSNR of dB, and an SSIM of on the fastMRI brain data. These measurements correspond to a larger acceleration and a better ultimate image quality than those of the SOTA methods [19]. The quantitative outcome is concordant with the results of our user study, where the differences between the brain and the knee images reflected those in the measured metrics in the radiologists’ evaluation.

In another ongoing survey, we are collecting the ranking votes for the pairs of images at the most extreme k-space undersampling factors generated in our work ( and ). While we do not anticipate those images to be useful for typical clinical examinations, the radiologists still express their enthusiasm because these extreme MRI acceleration factors would be of interest in certain other (non-diagnostic) applications, where the coarse images oftentimes prove sufficient (e.g., surgical planning in the radiological suites or radiation therapy planning in oncological care). We believe these extremely undersampled k-space measurements are bound to improve with time due to the growth of the available data, which will gradually pave the way towards true ultrafast imaging of high diagnostic value.

References

  • [1] T. Bakker, H. van Hoof, and M. Welling (2020) Experimental design for MRI by greedy policy search. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Cited by: §2.
  • [2] Y. Chen, A. G. Christodoulou, Z. Zhou, F. Shi, Y. Xie, and D. Li (2020)

    MRI super-resolution with gan and 3d multi-level densenet: smaller, faster, and better

    .
    External Links: 2003.01217 Cited by: §2.
  • [3] M. S. Cohen and R. M.Weisskoff (1991) Ultra-fast imaging. Magnetic Resonance Imaging 9 (1), pp. 1–37. Cited by: §1, §6.
  • [4] M. Z. Darestani, A. Chaudhari, and R. Heckel (2021) Measuring robustness in deep learning based compressive sensing. External Links: 2102.06103 Cited by: §2.
  • [5] J. F. Debatin and G. C. McKinnon (1998) Ultrafast mri: techniques and applications. Springer, London. External Links: ISBN 978-3-642-80384-0 Cited by: §1, §3.1, §3.2, §5.
  • [6] M. Dieckmeyer, A. Guha Roy, J. Senapati, C. Wachinger, L. Grundl, J. Döpfert, P. Bertran, A. Lemke, C. Zimmer, J. Kirschke, and D. Hedderich (2021-01) Effect of mri acquisition acceleration via compressed sensing and parallel imaging on brain volumetry. Magnetic Resonance Materials in Physics, Biology and Medicine, pp. . External Links: Document Cited by: §2.
  • [7] O. Esteban, R. W. Blair, D. M. Nielson, J. C. Varada, S. Marrett, A. G. Thomas, R. A. Poldrack, and K. J. Gorgolewski (2019) Crowdsourced MRI quality metrics and expert quality annotations for training of humans and machines. Scientific Data 6 (1), pp. 30. External Links: Document, ISSN 2052-4463, Link Cited by: §5.
  • [8] P. F. Ferreira, P. D. Gatehouse, R. H. Mohiaddin, and D. N. Firmin (2013) Cardiovascular magnetic resonance artefacts. Journal of Cardiovascular Magnetic Resonance 15 (2), pp. 41. Cited by: §1.
  • [9] M. Ghafoorian, A. Mehrtash, T. Kapur, N. Karssemeijer, E. Marchiori, M. Pesteie, C. R. Guttmann, F. de Leeuw, C. M. Tempany, B. Van Ginneken, et al. (2017) Transfer learning for domain adaptation in mri: application in brain lesion segmentation. In International conference on medical image computing and computer-assisted intervention, pp. 516–524. Cited by: §6.
  • [10] Y. Han, L. Sunwoo, and J. C. Ye (2020) K-space deep learning for accelerated mri. IEEE Trans. Med. Imaging 39(2), pp. 377–386. Cited by: §2.
  • [11] A. Hsiao, M. Lustig, M. T. Alley, M. Murphy, F. P. Chan, R. J. Herfkens, and S. S. Vasanawala (2012) Rapid pediatric cardiac assessment of flow and ventricular volume with compressed sensing parallel imaging volumetric cine phase-contrast mri. American Journal of Roentgenology 198 (3), pp. 250–259. Cited by: §1, §1.
  • [12] P. Isola, J. Zhu, T. Zhou, and A. A. Efros (2018) Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004v3. Cited by: §1, §3.2.
  • [13] J. Johnson, A. Alahi, and L. Fei-Fei (2016) Perceptual losses for real-time style transfer and super-resolution. External Links: 1603.08155 Cited by: §3.2.
  • [14] F. Knoll, T. Murrell, A. Sriram, N. Yakubova, J. Zbontar, M. Rabbat, A. Defazio, M. J. Muckley, D. K. Sodickson, C. L. Zitnick, and M. P. Recht (2020)

    Advancing machine learning for mr image reconstruction with an open competition: overview of the 2019 fastmri challenge

    .
    arXiv preprint arXiv:2001.02518. Cited by: §1, §3.2, §3.2, §6.
  • [15] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi (2017) Photo-realistic single image super-resolution using a generative adversarial network.

    2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    , pp. 4681–4690.
    Cited by: §1, §3.2.
  • [16] Q. Lyu, H. Shan, and G. Wang (2020) MRI super-resolution with ensemble learning and complementary priors. IEEE Transactions on Computational Imaging 6, pp. 615–624. External Links: ISSN 2573-0436, Link, Document Cited by: §2.
  • [17] J. Nilsson and T. Akenine-Möller (2020) Understanding ssim. arXiv preprint arXiv:2006.13846. Cited by: §3.2, D. Evaluation metrics.
  • [18] A. Panda, R. Naskar, S. Rajbans, and S. Pal (2019) A 3d wide residual network with perceptual loss for brain mri image denoising. In 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Vol. , pp. 1–7. External Links: Document Cited by: §3.2.
  • [19] N. Pezzotti, S. Yousefi, M. S. Elmahdy, J. van Gemert, M. D. Christophe Schülke, T. Nielsen, S. Kastryulin, B. P.F. Lelieveldt, M. J.P. van Osch, E. de Weerdt, and M. Staring (2020) An adaptive intelligence algorithm for undersampled knee mri reconstruction. arXiv preprint arXiv:2004.07339. Cited by: §2, §3.2, §6, §6.
  • [20] S. Prakkamakul, T. Witzel, S. Huang, D. Boulter, M. J. Borja, P. Schaefer, B. Rosen, K. Heberlein, E. Ratai, G. Gonzalez, and O. Rapalino (2016) Ultrafast brain mri: clinical deployment and comparison to conventional brain mri at 3t. Journal of Neuroimaging 26 (5), pp. 503–510. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1111/jon.12365 Cited by: §1.
  • [21] D. Prokopenko, J. V. Stadelmann, H. Schulz, S. Renisch, and D. V. Dylov (2019) Synthetic ct generation from mri using improved dualgan. arXiv preprint arXiv:1909.08942. Cited by: §3.2, §6.
  • [22] E. Schonfeld, B. Schiele, and A. Khoreva (2020) A u-net based discriminator for generative adversarial networks. arXiv preprint arXiv:2002.12655v1. Cited by: §1, §2.
  • [23] M. Tkachenko, M. Malyuk, N. Shevchenko, A. Holmanyuk, and N. Liubimov (2020-2021) Label Studio: data labeling software. External Links: Link Cited by: §5, F. Labeling tool for user study.
  • [24] J. Tsao (2010) Ultrafast imaging: principles, pitfalls, solutions, and applications. Journal of Magnetic Resonance Imaging 32 (2), pp. 252–266. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/jmri.22239 Cited by: §1.
  • [25] S. S. Vasanawala, M. T. Alley, B. A. Hargreaves, R. A. Barth, J. M. Pauly, and M. Lustig (2010) Improved pediatric mr imaging with compressed sensing. Radiology 256 (2), pp. 607–616. Cited by: §1, §1, §3.2, §6.
  • [26] G. Wang, T. Luo, J. Nielsen, D. C. Noll, and J. A. Fessler (2021) B-spline parameterized joint optimization of reconstruction and k-space trajectories (bjork) for accelerated 2d mri. External Links: 2101.11369 Cited by: §2, §6.
  • [27] T. Weiss, S. Vedula, O. Senouf, O. Michailovich, M. Zibulevsky, and A. Bronstein (2019) Joint learning of cartesian undersampling and reconstruction for accelerated mri. arXiv preprint arXiv:1905.09324v1. Cited by: §2.
  • [28] K. L. Wright, J. I. Hamilton, M. A. Griswold, V. Gulani, and N. Seiberlich (2014) Non-cartesian parallel imaging reconstruction. Journal of Magnetic Resonance Imaging 40 (5), pp. 1022–1040. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/jmri.24521 Cited by: §3.2.
  • [29] J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, and et al. (2019) An open dataset and benchmarks for accelerated mri. arXiv preprint arXiv:1811.08839. Cited by: Towards Ultrafast MRI via Extreme k-Space Undersampling and Superresolution, §1, §3.1.
  • [30] B. Zhu, J. Z. Liu, B. R. Rosen, and M. S. Rosen (2017) Image reconstruction by domain transform manifold learning. arXiv preprint arXiv:1704.08841. Cited by: §2.

A. Literature survey

Figure 9: Comparison of acceleration factors and corresponding SSIM metrics of major papers that cite the original fastMRI dataset. Our models are in the top left corner.

B. Image intensity normalization

We used two different normalization methods: percentile normalization and histogram-based normalization.

Percentile normalization

Percentile normalization is a min-max normalization to the and percentiles of an image:

Where is the intensity value of one pixel before normalization, is its intensity value after normalization, and , respectively are the and percentiles of the intensity values for a given slice. After normalization, the values falling outside the range were clipped to those boundaries.

Histogram-based normalization

Histogram-based normalization was performed as follows, for each image:

  1. Obtain the histogram of one slice;

  2. Fit a order polynomial to the histogram;

  3. Obtain the derivative of the polynomial;

  4. Search for zero-crossing of the derivative in order to locate the first minimum and maximum of the histogram;

  5. Center the new intensity range around ;

  6. Define the width of the intensity range as

    where is a scaling coefficient, and is the distance between and ;

  7. Apply min-max normalization.

The order of the polynomial, , as well as the coefficient were obtained empirically.

C. k-space sampling trajectories

We present here the four trajectories used for the sampling of the k-space (Figure 10) as well as examples of images obtained with those trajectories (Figure 11). Interestingly, without the reconstruction, fastMRI mask starts off with a marginally better image quality at low k-space sampling (Figure 13); however, it quickly yields worse quality following the reconstruction.

Figure 10: Typical patterns of the k-space undersampling masks.

Figure 11: Examples of images obtained using a acceleration factor and different k-space undersampling masks.

Figure 12: Examples of images obtained using different acceleration factors and k-space undersampling masks. The red crop shows the absolute difference between the ground truth and the undersampled images.
(a) The brain dataset
(b) The knee dataset
Figure 13: Dependence of the target metrics on the k-space fraction, calculated for the pairs of the ground truth and the undersampled images.

D. Evaluation metrics

To measure the quality of the images, generated using our acceleration methods, we computed the following metrics:

The Mean Square Error (MSE) is defined by:

The Peak Signal-to-Noise Ratio (PSNR) is the ratio between the maximum possible value and the value of the difference between the source and the corrupted image.

where is the maximum possible value of the image.

The Structural Similarity Index Measure (SSIM) is defined by:

where is the average of ; is the average of ;

is the variance of

; is the variance of ; is the covariance of , ; are constants used for computational stabiltiy; is the total intensity range; we set and , as indicated in [17].

E. Combined models with extreme acceleration factors

We present here the detailed results obtained with our different models at all acceleration factors.

Mask Model MSE, PSNR SSIM,
fastmri SRGAN 12.98 3.79 29.04 1.22 94.84 2.55
SRGAN+U-Net 8.81 3.08 30.80 1.44 95.97 2.17
U-Net+SRGAN 9.27 3.28 30.56 1.41 95.69 2.69
spiral SRGAN 7.66 2.94 31.47 1.66 96.28 2.26
SRGAN+U-Net 7.02 2.84 31.88 1.73 96.51 2.06
U-Net+SRGAN 7.09 2.82 31.81 1.65 96.39 2.37
radial SRGAN 6.25 2.63 32.40 1.78 96.95 1.95
SRGAN+U-Net 6.05 2.64 32.58 1.87 97.01 1.87
U-Net+SRGAN 6.13 2.62 32.48 1.75 96.92 2.11
Table 5: Low-Resolution () model with undersampling, total acceleration: , fraction of k-space: .

Mask Model MSE, PSNR SSIM,
fastmri SRGAN 23.07 6.30 26.53 1.17 92.40 2.93
SRGAN+U-Net 15.97 4.77 28.15 1.26 94.00 2.61
U-Net+SRGAN 18.55 5.88 27.51 1.28 93.32 3.27
spiral SRGAN 17.29 5.05 27.79 1.20 93.32 2.88
SRGAN+U-Net 13.61 4.11 28.85 1.26 94.19 2.58
U-Net+SRGAN 13.49 4.18 28.89 1.27 94.08 3.39
radial SRGAN 12.71 4.07 29.18 1.38 94.54 2.79
SRGAN+U-Net 11.44 3.90 29.66 1.46 94.92 2.59
U-Net+SRGAN 11.39 3.92 29.67 1.42 94.79 3.23
Table 6: Low-Resolution () model with undersampling, total acceleration: , fraction of k-space: .
Mask Model MSE, PSNR SSIM,
fastmri SRGAN 49.63 13.10 23.19 1.14 87.19 3.49
SRGAN+U-Net 36.99 9.93 24.47 1.16 89.57 3.10
U-Net+SRGAN 38.72 11.65 24.30 1.26 89.09 3.82
spiral SRGAN 30.89 7.43 25.23 1.04 89.86 3.33
SRGAN+U-Net 23.65 6.13 26.40 1.11 91.43 2.96
U-Net+SRGAN 24.48 6.76 26.27 1.17 91.16 3.68
radial SRGAN 25.29 6.90 26.13 1.16 91.34 3.16
SRGAN+U-Net 20.70 5.91 27.01 1.21 92.40 2.96
U-Net+SRGAN 20.88 6.09 26.98 1.22 92.13 3.62
Table 7: Low-Resolution () model with undersampling, total acceleration: , fraction of k-space: .

Mask Model MSE, PSNR SSIM, spiral SRGAN 46.46 10.66 23.44 1.01 86.80 3.29 SRGAN+U-Net 38.09 9.34 24.32 1.09 88.42 3.16 U-Net+SRGAN 38.76 10.41 24.27 1.16 88.32 3.78 radial SRGAN 42.97 10.13 23.79 1.01 87.71 3.39 SRGAN+U-Net 35.20 8.92 24.67 1.08 89.24 3.19 U-Net+SRGAN 35.64 9.38 24.63 1.13 89.06 3.73

Table 8: Low-Resolution () model with undersampling, total acceleration: , fraction of k-space: .
Acc. / Model MSE, PSNR SSIM,
frac. of
k-space



25%
Pix2pix radial 6.98 2.45 31.80 1.42 95.97 3.12
SRGAN (SR) 3.03 1.61 35.72 2.12 98.64 0.93

12.5%
Pix2pix radial 13.98 4.27 28.72 1.22 93.41 3.96
SRGAN+U-Net radial 6.05 2.64 32.58 1.87 97.01 1.87

6.25%
Pix2pix radial 26.14 7.44 25.99 1.18 90.55 4.20
SRGAN (SR) 11.69 4.21 29.58 1.49 95.63 2.35
U-Net+SRGAN radial 11.39 3.92 29.67 1.42 94.79 3.23

3.125%
Pix2pix radial 54.83 16.45 22.87 1.12 86.39 5.21
SRGAN+U-Net radial 20.70 5.91 27.01 1.21 92.40 2.96

1.5625%
SRGAN (SR) 28.76 9.67 24.37 1.13 88.49 3.32
SRGAN+U-Net radial 35.20 8.92 24.67 1.08 89.24 3.19
Table 9: Comparison of the best models selected based on the SSIM metric for the brain dataset.
Acc. / Model MSE, PSNR SSIM,
frac. of
k-space


25%
Pix2pix radial 27.95 19.64 26.23 2.79 86.69 5.83
SRGAN (SR) 8.84 9.04 32.77 4.89 94.31 3.65

12.5%
Pix2pix radial 42.26 29.35 24.54 2.99 80.97 8.20
SRGAN+U-Net radial 21.58 18.77 27.62 3.23 88.74 5.35

6.25%
Pix2pix radial 58.09 40.44 23.22 3.02 74.87 10.44
SRGAN (SR) 26.02 21.85 27.53 4.23 85.04 7.92
U-Net+SRGAN radial 28.40 22.85 26.53 3.31 83.98 8.42

3.125%
Pix2pix radial 77.72 52.45 21.86 2.84 70.65 11.57
SRGAN+U-Net radial 42.48 35.12 24.97 3.56 80.67 8.90

1.5625%
SRGAN (SR) 45.71 34.51 24.82 3.88 76.95 10.65
SRGAN+U-Net radial 56.72 41.34 23.44 3.19 76.24 10.11
Table 10: Comparison of the best models, selected based on the SSIM metric for the knee dataset.

F. Labeling tool for user study

We used the community version of the open-source labeling tool Label Studio

[23] to perform the user study described in the main text, Section 5.We adjusted the user interface to compare the accelerated (left) and the ground truth (right) images based on tree IQ criteria on a 4-point scale.

Figure 14: User interface of the tool for the user study. It was used to survey five radiologists about the diagnostic value of the reconstructions.

An example of the user interface is presented in figure 14. Experts were asked to rate the quality of the left (accelerated) image compared to the right one (ground truth). The main element of the interface is the panel section with two images for the radiologists to observe side-by-side. Users can zoom in and zoom out the images using a mouse scroll wheel to conveniently examine the regions of interest. While performing the examination, the users provide their answers to the questions below, using mouse or keyboard shortcuts. Once the evaluation is finished, users press submit button and proceed to another pair of images. Users can also skip a pair of images in case if they face difficulties with scoring a particular case. The examination is finished either when the user decides to stop or when there are no pairs of images left in the test subset.

G. Experimental Setup

Specification of dependencies

All models were implemented using Python 3.8 and PyTorch 1.4. Training models and all the experiments were conducted using Nvidia Tesla V100 with 32 GB RAM.

Adam optimizer is used to train the models, with the initial learning rate being equal to

. The size of a batch is 4, and the number of epochs is 100. The combined model contains

M trainable parameters.

Complexity analysis

Training time of the combined models (SRGAN+U-Net, U-Net+SRGAN) are about 60 hours. An average inference time for 100000 slices shown in Table 11.

Training the combined model with the above parameters requires about 11GB of GPU memory. The inference of the combined model requires 2.2GB of GPU memory.

Model Time, ms # of Parameters, M

Pix2pix
9.6 26.1

SRGAN(SR)
7.2 5.7
SRGAN(SR) 6.1 5.9
SRGAN(SR) 5.8 6.2
U-Net+SRGAN 10.7 29.1
SRGAN+U-Net 16.9 29.1

Table 11: An average inference time of the models per slice in milliseconds and the number of trainable parameters.