Compressive sensing (CS) has attracted considerable interests for many researchers in signal and image processing communities [1, 2]. The most attractive aspect of CS is that the sampling and compression are conducted simultaneously, and almost all computational cost is derived from the decoder stage, and therefore, leading to a low computational cost in the encoder stage . Due to the unique advantages of CS, it has been widely used in many important applications, such as magnetic resonance imaging  and single-pixel camera .
Image CS methods can be classified, in general, into two categories: shallow model-based methods[9, 10, 11, 12, 13, 14]
and deep learning based methods[7, 3, 8, 15, 16, 17]. One of the classic shallow approaches is based on image sparsity model, assuming that each image local patch can be encoded as a linear combination of basis elements . Recent works exploited image non-local self-similarity (NSS) prior , by clustering similar patches into groups and modeling them by structural sparsity [11, 19, 13] or low-rankness [6, 14, 20]. However, most of the shallow methods are patch-based, which inevitably lead to recovered images with ringing or blocky artifacts due to naive patch aggregation (See an example in Fig. 1 (b)).
On the other hand, many recent works applied various deep neural networks for image restoration tasks[21, 7, 3, 8, 15, 16, 17, 22]
. Such approaches have demonstrated great potential to learn image properties from training dataset with an end-to-end approach. Popular neural networks structures, such as convolutional neural networks (CNN)
and recurrent neural networks (RNN), have been applied to image restoration tasks achieving state-of-the-art results. Since most of the deep algorithms are supervised, in applications such as remote sensing or biomedical imaging, the model trained on a standard image corpus fails when applied to specific modalities. Furthermore, most of the existing deep methods focused on exploiting image local properties (due to limited receptive field) while largely ignoring NSS, which limit their performance in many image restoration applications.
Bearing the above concerns in mind, we propose a joint low-rank and deep (LRD) image model, which contains a pair of triply complementary priors, namely external and internal, deep and shallow, and local and non-local priors. Based on the LRD model, we propose a novel hybrid plug-and-play (H-PnP) framework for highly effective image CS. To the best of our knowledge, this is the first work to jointly exploit both NSS and deep priors under a unified framework for image restoration. Table 1 depicts the proposed LRD-based H-PnP scheme and some representative of existing reconstruction algorithms with their key attributes.
The main contributions of this paper are summarized as follows:
The proposed LRD model jointly exploits triply-complementary low-rank and deep denoiser priors, to take the advantages of NSS, scalable model richness, as well as good generalizability. In practice, LRD-based method significantly improves the visual quality of the CS reconstructed images (see an example in Fig. 1 (f)).
We propose the H-PnP framework for image CS based on the novel LRD model. To make the optimization tractable, we propose an efficient yet effective algorithm by applying alternating minimizing.
Extensive experimental results demonstrate that the proposed H-PnP based image CS algorithm has superior performance comparing to several popular or state-of-the-art image CS methods.
2 Related Works
In this section, we give a brief introduction on CS, deep PnP model and low-rank image modeling.
Compressive Sensing: The goal of image CS is to reconstruct the high-quality image from its undersampled (lower than Nyquist sampling rate ) measurements obtained by
where denotes the random projection matrix, , and is the additive noise or measurement error. The image CS is an ill-posed inverse problem as the measurement y has much lower dimension than that of the underlying image. Therefore, an effective prior is key to a successful image CS algorithm [10, 8, 11].
Deep Plug-And-Play Methods: Recent works on the plug-and-play (PnP) framework [26, 27, 21] allowed applying the effective image denoiser to solve the general inverse problems, such as image deblurring 28] and computational imaging , etc. By decoupling the problem-specific sensing modality (i.e., the for image CS) from the general image priors, PnP provides a more flexible approach to generalize denoising algorithms to other more sophisticated applications. Very recent works [21, 29, 28, 27] applied state-of-the-art deep denoisers in PnP by solving the following maximum a posteriori (MAP) problem:
where denotes the fidelity term for the inverse problem (i.e., for image CS), and denotes the prior based on certain deep denoiser [21, 22, 27]. is a regularization parameter. By applying highly effective deep priors, the deep PnP approaches have achieved superior results in many image processing applications [21, 29, 28].
Low-Rank Image Modeling: Besides deep image priors, other image properties such as NSS , i.e., image patches are typically similar to other non-local structures within the same image, have been widely utilized for image restoration [11, 19, 6, 20]. Popular NSS-based methods proposed to group the similar patches, and exploit the patch correlation within each group. Specifically, overlapping patches are extracted from the image x. Taking each as the reference patch, its most similar patches are selected to construct each data matrix .
There are different methods to process the constructed data matrices, among which the low-rank (LR) modeling has demonstrated superior performance in many image restoration applications [6, 11, 14, 20]. Comparing to PnP approaches based on deep prior [21, 29, 28, 27], the LR-based methods are unsupervised and typically limited by the model flexibility. Moreover, such methods inevitably produce the ringing artifacts due to the aggregation of overlapping patches.
3 Proposed Method
As mentioned above, NSS-based methods and deep learning based methods have their respective merits and drawbacks. In this section, we propose a general Hybrid PnP (H-PnP) framework by combining the two triply-complementary priors, dubbed the joint low-rank and deep (LRD) prior.
3.1 H-PnP Framework
We now propose a novel H-PnP framework for image CS by incorporating the LRD prior by solving the following optimization problem,
Similar to the deep PnP problem in Eq. (2), the deep prior is applied in the proposed H-PnP scheme (i.e., Eq. (3)). denotes the Frobenius norm, and is the low-rank regularizer with a non-negative weight . The rank penalties are applied to exploit the image self-similarity. denotes the matrix formed by the set of similar patches for each reference patch
. The selected patches are vectorized and formed the columns of the matrix, which is approximated by a corresponding low-rank matrix .
The proposed H-PnP formulation incorporates the LRD prior, which assumes that the underlying image x satisfies both LR and deep priors. Comparing to traditional deep PnP problem of Eq. (2), the proposed H-PnP scheme integrates a pair of triply-complementary priors using one unified the optimization problem.
4 Optimization for the Proposed Model
In this section, we present a highly effective image CS algorithm based on the proposed H-PnP using alternating minimizing. It can be seen that Eq. (3) is a large-scale non-convex optimization problem. To make the optimization tractable, we propose a simple alternating minimizing strategy to solve Eq. (3) for image CS, i.e., the algorithm alternates between solving Eq. (3) for and x, which corresponds to the Low-rank Approximation and Image Update sub-problem, respectively.
4.1 Low-Rank Approximation
For fixed x, we solve Eq. (3) for each by minimizing
In this work, we set the rank penalty to be the weighted nuclear norm, as the corresponding WNNM method  demonstrated superior performance in image restoration amongst other classical CS methods. There exists a closed-form solution to
by applying the singular value decomposition (SVD), with details steps in.
4.2 Image Update
For fixed low-rank matrix , the image x can be updated by solving the following problem,
One can observe that it is quite difficult to solve Eq. (5) directly. In order to facilitate the optimization, we adopt the alternating direction method of multipliers (ADMM) algorithm  to solve x. Specifically, we introduce an auxiliary variable z with the constraint , then Eq. (5) can be rewritten as the following constrained problem,
4.2.1 x Sub-problem
Given the obtained and z, x sub-problem in Eq. (7) is essentially a minimization problem of a strictly convex quadratic function. However, is a random projection matrix without a specific structure in image CS, which is expensive to directly compute matrix inversion. To avoid this issue, a gradient descent method  is applied, i.e.,
where represents the step size, and q is the gradient of the objective, which can be calculated as
where both and are pre-computed and fixed during the iterations.
4.2.2 z Sub-problem
Given x, then z sub-problem in Eq. (8) can be rewritten as
where . From a Bayesian perspective, Eq. (12) is a Gaussian denoising problem for z
by solving a MAP problem, with the corresponding noise standard deviation to be. Accordingly, we denote such denoising problem as
where denotes a Gaussian denoiser based on the specific deep image prior . In general, any deep image Gaussian denoisier can be used as the in Eq. (13). In this paper, we apply a fast and flexible denoising CNN (FFDNet)  for Eq. (13), which is an efficient but effective CNN-based denoisier. Furthermore, FFDNet is capable of dealing with different standard deviations by self-adaption.
Till now, the efficient solution for each separated minimization sub-problem has been achieved, which makes the whole algorithm efficient and effective. After solving the above sub-problems, the complete description of the proposed H-PnP algorithm for image CS is summarized in Algorithm 1. The iterative algorithm in this paper is terminated when , where is a small constant.
5 Experimental Results
We conduct extensive experiments to evaluate the proposed H-PnP algorithm for image CS. Similar to the setups in previous works [8, 11, 10, 32], we simulate the image CS measurements at the block level (with the block size of
) using a Gaussian random projection matrix for each test image. The image CS algorithms are applied to reconstruct the image using the simulated CS measurements. We apply the peak signal to noise ratio (PSNR) to evaluate the CS reconstructed images.
5.1 Implementation and Parameters
We applied the pre-trained FFDNet denoiser  as the deep prior in the proposed H-PnP based image CS algorithm. The main parameters of the proposed method are set as follows. The size of each patch is set to 7
7, the number of patches grouped by KNN operator is= 60, and the size of KNN search window is set to 2020. We set the maximum number of iterations to be = 60. The coefficients , , and are tuned for different sampling ratio in image CS (see the tuning procedure as well as the values for each sampling ratio in our shared code). The source code of the proposed HPnP method for image CS is available at: https://drive.google.com/open?id=1HcgKtj0r5SVpp6yKmRVyI9b8TdXjcT83.
5.2 Comparison with Classical Image CS Methods
We first compare the proposed H-PnP image CS algorithm to the popular or state-of-the-art classic methods, i.e., TV , Rcos , GSR , JASR , TNNM  and WNNM . Amongst them, WNNM is a well-known non-local method which provides the state-of-the-art results for image denoising. We extend the WNNM denoising algorithm  to image CS by applying ADMM  which is similar to what we described in Section 4.2. The extended WNMM for image CS achieves the best results among all classic competing methods. We use the publicly available codes of other competing methods from their official websites with the default parameter settings for all experiments.
We simulate the image CS measurements for all test images using five different sampling ratios, i.e., 0.1, 0.2, 0.3, 0.4 and 0.5. Table 2 lists the average PSNRs over all test images from BSD68 (68 images) , obtained by our proposed H-PnP based image CS algorithm, as well as the six classic competing methods. It is clear that our proposed H-PnP consistently outperforms all competing methods on different sampling ratios for all datasets. On average, our proposed H-PnP enjoys a PSNR gain over TV by 3.50dB, over Rcos by 2.14dB, over GSR by 1.06dB, over JASR by 1.25dB, over TNNM by 0.59dB and over WNNM by 0.54dB. The visual quality comparisons of image 253027 in the case of sampling ratio of 0.1 are shown in Fig. 2. We have magnified a sub-region of each image to compare visual result of each competing method. It can be seen that TV cannot obtain a visual pleasant result. Rcos, GSR, JASR, TNNM and WNNM methods are all prone to produce some undesirable ringing artifacts. By contrast, the proposed H-PnP algorithm not only preserves fine image details, but also removes the visual artifacts significantly.
5.3 Comparison with Deep Image CS Methods
We now compare our proposed H-PnP with deep learning based methods including: SDA , ReconNet , IST-Net , IST-Net , CSNet  and SCSNet  methods. Note that SCSNet exploited a scale CNN that delivers state-of-the-art image CS performance. We follow  to use the images on BSD68  as the test images. The average PSNR results of our proposed H-PnP as well as different deep learning based methods on four sampling ratios are shown in Table 3, where the results of SDA, ReconNet, ISTA-Net and ISTA-Net are from . One can observe that our proposed H-PnP significantly outperforms all deep learning based methods. In particular, the proposed H-PnP achieves 0.76dB gains in average PSNR over SCSNet method. The visual comparisons of image 119082 on BSD68 dataset are shown in Fig. 3. It can be observed that some visual artifacts are still visible in all competing deep learning based methods. By contrast, our proposed H-PnP not only significantly removes undesirable artifacts across all the image, but also preserves large-scale sharp edges and small-scale fine details.
This paper proposed a joint low-rank and deep (LRD) image model, which comprises a pair of triply complementary priors, namely external and internal, deep and shallow, and local and non-local priors. We have then proposed a H-PnP framework based on the LRD model to solve image CS problem along with an alternating minimization method. Experimental results have demonstrated that the proposed H-PnP based image CS algorithm significantly outperforms many state-of-the-art image CS methods. Future work lies in the theoretical analysis of our proposed model and apply it to other image restoration tasks.
-  E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on information theory, vol. 52, no. 2, pp. 489–509, 2006.
-  D. L. Donoho, “Compressed sensing,” IEEE Transactions on information theory, vol. 52, no. 4, pp. 1289–1306, 2006.
-  J. Zhang and B. Ghanem, “Ista-net: Interpretable optimization-inspired deep network for image compressive sensing,” in , 2018, pp. 1828–1837.
-  B. Wen, S. Ravishankar, L. Pfister, and Y. Bresler, “Transform learning for magnetic resonance image reconstruction: From model-based learning to building neural networks,” IEEE Signal Processing Magazine, vol. 37, no. 1, pp. 41–53, 2020.
-  M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE signal processing magazine, vol. 25, no. 2, pp. 83–91, 2008.
-  S. Gu, Q. Xie, D. Meng, W. Zuo, X. Feng, and L. Zhang, “Weighted nuclear norm minimization and its applications to low level vision,” International journal of computer vision, vol. 121, no. 2, pp. 183–208, 2017.
-  K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok, “Reconnet: Non-iterative reconstruction of images from compressively sensed measurements,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 449–458.
-  W. Shi, F. Jiang, S. Liu, and D. Zhao, “Scalable convolutional neural network for image compressed sensing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 12290–12299.
-  J. Zhang, D. Zhao, C. Zhao, R. Xiong, S. Ma, and W. Gao, “Image compressive sensing recovery via collaborative sparsity,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 2, no. 3, pp. 380–391, 2012.
-  C. Li, W. Yin, and Y. Zhang, “User s guide for tval3: Tv minimization by augmented lagrangian and alternating direction algorithms,” CAAM report, vol. 20, no. 46-47, pp. 4, 2009.
-  J. Zhang, D. Zhao, and W. Gao, “Group-based sparse representation for image restoration,” IEEE Transactions on Image Processing, vol. 23, no. 8, pp. 3336–3351, 2014.
-  Z. Zha, X. Yuan, B. Wen, J. Zhou, J. Zhang, and C. Zhu, “A benchmark for sparse coding: When group sparsity meets rank minimization,” IEEE Transactions on Image Processing, vol. 29, pp. 5094–5109, 2020.
-  N. Eslahi and A. Aghagolzadeh, “Compressive sensing image restoration using adaptive curvelet thresholding and nonlocal sparse regularization,” IEEE Transactions on Image Processing, vol. 25, no. 7, pp. 3126–3140, 2016.
-  T. Geng, G. Sun, Y. Xu, and J. He, “Truncated nuclear norm minimization based group sparse representation for image restoration,” SIAM Journal on Imaging Sciences, vol. 11, no. 3, pp. 1878–1897, 2018.
-  X. Yuan and Y. Pu, “Parallel lensless compressive imaging via deep convolutional neural networks,” Optics Express, vol. 26, no. 2, pp. 1962–1977, 2018.
-  A. Mousavi, A. B. Patel, and R. G. Baraniuk, “A deep learning approach to structured signal recovery,” in 2015 53rd annual allerton conference on communication, control, and computing (Allerton). IEEE, 2015, pp. 1336–1343.
-  W. Shi, F. Jiang, S. Liu, and D. Zhao, “Image compressed sensing using convolutional neural network,” IEEE Transactions on Image Processing, vol. 29, pp. 375–388, 2020.
-  M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Transactions on Image processing, vol. 15, no. 12, pp. 3736–3745, 2006.
-  J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, “Non-local sparse models for image restoration,” in 2009 IEEE 12th international conference on computer vision. IEEE, 2009, pp. 2272–2279.
-  Z. Zha, X. Yuan, B. Wen, J. Zhou, J. Zhang, and C. Zhu, “From rank estimation to rank approximation: Rank residual constraint for image restoration,” IEEE Transactions on Image Processing, vol. 29, pp. 3254–3269, 2020.
-  K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep cnn denoiser prior for image restoration,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3929–3938.
-  K. Zhang, W. Zuo, and L. Zhang, “Ffdnet: Toward a fast and flexible solution for cnn-based image denoising,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4608–4622, 2018.
-  J. Zhang, J. Pan, J. Ren, Y. Song, L. Bao, R. W. Lau, and M. H. Yang, “Dynamic scene deblurring using spatially variant recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2521–2529.
J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, and L. Carin,
“Compressive sensing by learning a gaussian mixture model from measurements,”IEEE Transactions on Image Processing, vol. 24, no. 1, pp. 106–119, 2014.
-  D. Liu, B. Wen, Y. Fan, C. Loy, and T. S. Huang, “Non-local recurrent network for image restoration,” in Advances in Neural Information Processing Systems, 2018, pp. 1673–1682.
-  S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, “Plug-and-play priors for model based reconstruction,” in 2013 IEEE Global Conference on Signal and Information Processing. IEEE, 2013, pp. 945–948.
-  D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9446–9454.
-  T. Tirer and R. Giryes, “Image restoration by iterative denoising and backward projections,” IEEE Transactions on Image Processing, vol. 28, no. 3, pp. 1220–1234, March 2019.
-  A. M. Teodoro, J. M. Bioucas-Dias, and M. A. T. Figueiredo, “A convergent image fusion algorithm using scene-adapted gaussian-mixture-based denoising,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 451–463, Jan 2019.
S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al.,
“Distributed optimization and statistical learning via the
alternating direction method of multipliers,”
Foundations and Trends® in Machine learning, vol. 3, no. 1, pp. 1–122, 2011.
-  P. Deift and X. Zhou, “A steepest descent method for oscillatory riemann–hilbert problems. asymptotics for the mkdv equation,” Annals of Mathematics, vol. 137, no. 2, pp. 295–368, 1993.
-  C. Chen, E. W. Tramel, and J. E. Fowler, “Compressed-sensing recovery of images and video using multihypothesis predictions,” in 2011 conference record of the forty fifth asilomar conference on signals, systems and computers (ASILOMAR). IEEE, 2011, pp. 1193–1198.
-  P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 5, pp. 898–916, 2010.