Log In Sign Up

Image Denoising by Gaussian Patch Mixture Model and Low Rank Patches

Non-local self-similarity based low rank algorithms are the state-of-the-art methods for image denoising. In this paper, a new method is proposed by solving two issues: how to improve similar patches matching accuracy and build an appropriate low rank matrix approximation model for Gaussian noise. For the first issue, similar patches can be found locally or globally. Local patch matching is to find similar patches in a large neighborhood which can alleviate noise effect, but the number of patches may be insufficient. Global patch matching is to determine enough similar patches but the error rate of patch matching may be higher. Based on this, we first use local patch matching method to reduce noise and then use Gaussian patch mixture model to achieve global patch matching. The second issue is that there is no low rank matrix approximation model to adapt to Gaussian noise. We build a new model according to the characteristics of Gaussian noise, then prove that there is a globally optimal solution of the model. By solving the two issues, experimental results are reported to show that the proposed approach outperforms the state-of-the-art denoising methods includes several deep learning ones in both PSNR / SSIM values and visual quality.


Good Similar Patches for Image Denoising

Patch-based denoising algorithms like BM3D have achieved outstanding per...

A Non-Local Means Approach for Gaussian Noise Removal from Images using a Modified Weighting Kernel

Gaussian noise removal is an interesting area in digital image processin...

Neither Global Nor Local: A Hierarchical Robust Subspace Clustering For Image Data

In this paper, we consider the problem of subspace clustering in presenc...

PEDENet: Image Anomaly Localization via Patch Embedding and Density Estimation

A neural network targeting at unsupervised image anomaly localization, c...

Patch-based Contour Prior Image Denoising for Salt and Pepper Noise

The salt and pepper noise brings a significant challenge to image denois...

Nonlocal Patches based Gaussian Mixture Model for Image Inpainting

We consider the inpainting problem for noisy images. It is very challeng...

Patch-based Non-Local Bayesian Networks for Blind Confocal Microscopy Denoising

Confocal microscopy is essential for histopathologic cell visualization ...

1 Introduction

As a classical and fundamental problem in low level vision, image denoising has been extensively explored and it is still under investigation as well in methodological aspects. In general, image denoising aims to recover the latent noise-free image from its noise-corrupted version. During the past decades, image denoising has got great development in a variety of models including spatial filtering  [31], total variation [29], wavelet/curvelet transform [14], noise detector [3, 6, 13], sparse representation [4, 12], non-local means and its varieties  [1, 8, 10, 19, 39], deep learning[9, 20, 30, 36, 37], etc.

The landmark work of image denoising is non-local means (NLM) [1] which brings a new era by finding non-local similar patches within the image. Quite a few patch-based denoising methods [8, 10, 12, 19, 32, 37] achieve great success by exploiting the internal self-similarity prior. A quintessential example should be cited that BM3D [10] uses 3D filtering in transformed domain by patch matching, which is quite effective and has been a benchmark in image denoising. Here, patch matching is performed within a relatively large local window instead of globally, and consequently the number of patches may not be sufficient that leads to ringing artifacts. In order to improve the accuracy of patch matching, external natural patch priors are used to regularize the denoising process. Method [7] learns the covariance matrix and the measure of redundancy for each patch based on clustered natural noise-free patch database. Zoran and Weiss [40]

gave high likelihood values for patches sampled from clean images using Gaussian mixture models (GMM), and reconstructed the latent image by maximizing the expected patch log likelihood. Though external patch priors based approaches have achieved remarkable success in image restoration, they are still local and do not make full use of image internal self-similarity. Some methods  

[12, 19] globally search for the most similar patches across the whole image and have demonstrated competitive results. Though global patch matching is to determine enough similar patches, the error rate of patch matching may be higher, which results in destroying image details unsatisfactorily. Methods PCLR [8] and PGPD [35] integrate external patch priors and internal self-similarity into one framework. They learn GMM prior information from noise-free image patches to guide the noisy patches classification and the subsequent low rank subspace learning, which makes the latent patch reconstruction more robust to noise. However, there is no obvious improvement in denoising performance, compared with local patch matching methods. We argue that such utilization of GMM prior information is not effective enough because it neglects the noise of degraded image, which interferes with the selection of similar patches. How to precisely select similar patches is still an open problem.

Besides, low rank matrix approximation (LRMA), which aims to recover the underlying low rank matrix from its degraded observation, appears in a wide range of applications in various fields including computer vision, machine learning, signal processing and bioinformatics. For instance, use of low rank approximation can be found in applications such as face recognition 

[17, 11], background modeling and foreground object detection [19, 21, 15], image alignment [27, 38], video denoising [22, 28] image restoration [8, 33, 12]

, among others. Since direct rank minimization is difficult to solve, the problem is generally relaxed by substitutively minimizing the nuclear norm of the estimated matrix 

[16]. The nuclear norm of a matrix , denoted by

, is defined as the sum of its singular values, i.e.

, where denotes the -th singular value of . The nuclear norm minimization model has been attracting significant attention due to its rapid development in both theory and implementation. Cai et al. [2] proved that the model can be easily solved in a closed form by imposing a soft thresholding operation on the singular values of the observation matrix. Candès et al. [5]

proved that from the noisy input, its intrinsic low-rank reconstruction can be exactly achieved with a high probability by the model. Many state-of-the-art denoising methods  

[18, 34, 25, 12] have been proposed in recent years by utilizing nuclear norm proximal as the key proximal technique. Albeit its success as mentioned before, nuclear norm minimization model still has certain limitations. Gu et al. [19] proposed weighted nuclear norm minimization model trying to achieve a better estimation of the latent data from the corrupted input. The weighted nuclear norm of a matrix is given by , where and is a non-negative weight assigned to . The nuclear norm minimization model provides superior performance and efficiency for matrix completion or impulse noise. However, they ignore the difference between Gaussian noise and impulse noise, then building a low rank model for Gaussian noise is still a worthwhile work.

This article will address two issues: (1) how to improve similar patches matching accuracy; (2) build an appropriate low rank matrix approximation model for Gaussian noise.

Firstly, unlike PCLR [8] and PGPD [35] clustering the noisy patches directly, we first carry out a preliminary denoising of the image by a local patch matching based denoising algorithm and get a preprocessed image, then cluster all preprocessed patches into

classes. If a class contains too many similar patches, we use k-means to divide the big class into some smaller ones. We next record all patches location information. Consequently, in order to classify noisy patches, we replace all preprocessed patches by the corresponding noisy ones in each class according to the location information. Furthermore, the preprocessed image is updated in each iteration in order to improve patch matching accuracy. This idea significantly increases the accuracy of similar patch clustering. Here, we use a local patch matching based denoising method because it matches similar patches with higher precision. Since BM3D 

[10] is the state-of-the-art local patch matching based method and a real-time computing algorithm, we use it as a preprocessing algorithm in this paper. Though some global patch matching based methods outperform BM3D, they introduce some errors for patch matching and the errors can’t be corrected in the future denoising steps. In addition, the methods PCLR [8] and PGPD [35] are iterative algorithms. This means that the methods PCLR [8] and PGPD [35] have already used a global patch matching based algorithm (the -th step denoising) as a preprocessing algorithm for the -th step denoising.

Secondly, according to the characteristics of Gaussian noise, the minimization of i.e. the square of the Frobenius norm of adapts Gaussian noise model. Inspired by [39] and the equation , similar to nuclear norm minimization model [2], we will establish Gaussian nuclear norm minimization model by minimizing and then efficiently solve the problem.

As noted above, the proposed method uses GMM to achieve both local and global patch matching, and the mixture of patches can be further regularized via k-means clustering procedure and low rank minimization. This process may seem complicated, but it is very effective. In the low rank denoising part, we use the soft threshold given by [19] which is good to smooth the image. The proposed model is solved by an alternating minimization procedure and it converges rapidly.

The rest of the paper is organized as follows. In Section 2, we propose our algorithm by using GMM so that both local and global patch matching can be achieved. The complexity and convergence of the algorithm are also been discussed. Section 3 shows experiment results and discussion. Conclusions and future works are given in Section 4.

2 The Proposed Algorithm

Throughout this paper, we denote scalars, vectors and matrices by non boldfaced, boldfaced lower-case, and boldfaced upper-case letters, respectively.

2.1 The low rank regularization model based on GMM prior

The noise-corrupted images are often represented by the following simplified model:


where is noisy image, is latent original image and is additive white Gaussian (AWG) noise matrix of mean

and standard deviation

. Let and be the vectorized image patches which are extracted from the noisy image and latent original image at the -th pixel, respectively. All noisy patches are divided into similar patch groups. Consequently, these similar patches in -th group are collected to form a reshaped matrix , where may be different for each . The noise model (1) can be rewritten in the patch-based representation as follow:


where denote the patch reshaped matrices from the noisy image , original image and noise matrix , respectively.

In this paper, we will introduce preprocessed image based GMM prior low rank regularization model (PG-LR):


where is the Frobenius norm, denotes nuclear norm. is preprocessed image based patch cluster log likelihood term, is the -th patch extracted from preprocessed image , and denotes which class is from. Our model effectively incorporates image internal self-similarity and external patch priors into the low rank denoising model. There are two advantages of such clustering based low rank regularization.

First, the Gaussian patch mixture model based patch clustering can take the advantages of local and global patch matching. Local patch matching is to find similar patches in a large neighborhood which can alleviate noise effect, while the number of patches may not be sufficient. Global patch matching aims to determine enough similar patches but the error rate of patch matching may be higher. The model first use local patch matching method to reduce noise and then use Gaussian patch mixture model to achieve global patch matching. The detail will be discussed in Subsection 2.2.

Second, the nuclear norm minimization model [2] is an outstanding model for removing impulse noise and inpainting, but not adapt to Gaussian noise. Here according to the statistic properties of Gaussian noise and the nuclear norm minimization model [2], we build a new model for Gaussian noise: , where is the number of rows in the matrix . We prove that there is a global optimal solution of the model and the simulation experiment shows that our model is much better than the nuclear norm minimization model [2] for Gaussian noise (see the supplementary material). Subsection 2.3 will give details.

2.2 Improved Gaussian Patch Mixture Model

In this subsection, we first recall GMM model [40]. All noise-free image patches , () are divided into groups with a parameter , , which describes a Gaussian density function parameterized by the mean and covariance matrix . Thus, the probability of each patch could be defined as a weighted sum of -Gaussians:




and is the transpose of , and is the set of parameters with and , .

We investigate the model (4) and discover that is unknown latent image patch and consequently it’s impossible to calculate . In the model (4), it normally takes noisy patch instead of noise-free patch . As a result, the error rate of patch clustering increases significantly due to noise. In this paper, we will solve the problem.

The latent image patch is unknown and the noisy image patch is unfavorable, then we should find another solution. There are quite a few excellent denoising algorithms [1, 8, 10, 19], and a distinguished one is BM3D [10]. BM3D is a real-time computing algorithm and gets favorable restoration results. Furthermore, it matches similar patches in a large neighborhood, which alleviates noise effect for selection of similar patches. We take BM3D as a image preprocessing algorithm and get a preprocessed image that is a rosy estimate of the noise-free one. For this reason, the preprocessed image based patch cluster log likelihood (P-CLL) term is given as follow:


Obviously, our model is feasible and efficient as we know is the pleasurable estimate of . Consequently, we cluster all preprocessed patches into classes by maximizing the P-CLL term. However, the number of image patches varies greatly from class to class. In order to equalize the number of similar patches in each class, we further adjust the classification. If a class doesn’t contain enough similar image patches, it will be merged with the nearest one. If there are excessive patches in a class, k-means is used to divide the large class into some finer ones. Instead of constructing low rank matrices, these preprocessed image patches in each class are used to record patches location information. Finally, all the noisy image patches are clustered by the guide of the location information of the corresponding preprocessed ones. This clustering process is somewhat complex, but the accuracy is undoubtedly higher. As a result, the rank of the created matrix by stacking patches from each class is lower.

Note: why not we use a denoising algorithm based on global patch matching? Jin et al. [23, 24] have proved that non-local means is a local method with a large neighborhood and there is an optimal size of search window. Though global patch matching aims to acquire enough similar patches, the error rate of patch matching is intensely higher because of noise interference. In the image preprocessing, if an image patch is classified into a wrong class, the error will not be corrected in later process. For BM3D [10], the number of patches may be insufficient and the result may not as good as global patch matching based methods (such as WNNM [19], PCLR [8] and PGPD [35]), but the error rate of patch matching is lower. The restoration result will be improved in the future denoising steps but the error will exist forever. Another reason for not using global patch matching based methods is that they run slowly.

2.3 Gaussian nuclear norm minimization for low rank modeling

Consider the singular value decomposition (SVD) of a matrix

of rank :

where and are respectively and matrices with orthonormal columns and the singular values are positive and satisfy that .

Nuclear norm minimization (NNM) [16] is an effective rank minimization. The nuclear norm of a matrix , denoted by , is defined as the sum of its singular values, i.e.


The NNM approach has been attracting significant attention due to its rapid development in both theory and implementation. The nuclear norm proximal (NNP) problem [2] is


where is a positive constant.

Theorem A (Cai et al. [2]) For each , the global optimum of model (8) is


where denotes the -th singular value of and .

Albeit its success as aforementioned, NNM ignores the prior knowledge we often have about singular values of a practical data matrix. More specifically, larger singular values of an input data matrix quantify the information of its underlying principal directions. In order to improve the flexibility of NNM, Gu et al. [19] proposed the weighted nuclear norm. The weighted nuclear norm of a matrix is defined as


where is a non-negative weight assigned to . The weight vector will enhance the representation capability of the original nuclear norm. The weighted nuclear norm proximal (WNNP) problem is given by


Theorem B (Gu et al. [19]) If the weight satisfies , the global optimum of model (11) is


The WNNM model is more difficult to optimize than conventional NNM ones due to the non-convexity of the involved weighted nuclear norm.

In this paper, we consider the proprieties of Gaussian noise. By tradition, the LRMA for Gaussian noise is to minimize . Note that , we adapt NNM model (8) to Gaussian noise by utilizing to take place of . Then we have Gaussian nuclear norm minimization (GNNM):


We prove that the Gaussian nuclear norm minimization problem can be equivalently transformed to a quadratic programming (QP) problem with linear constraints. This allows us to easily reach the global optimum of the original problem by using off-the-shelf convex optimization solvers. In order to solve the problem (13), we introduce the following theorem.

Theorem 1 For each , the global optimum of model (13) is


Proof Considering as a new matrix and using Theorem A, we have that the global optimum of model (13) is , where denotes the -th singular value of . Let and , it is easy to get . Then we have

Therefore, is also the global optimum of model (13).

A image contaminated by AGW noise i.e. model (1) can be recovered by the Theorem 1. We next discuss the parameter . If the standard deviation of AWG noise is , we have the shrinkage amount . According to the AGW noise model (1

), we deduce the relationship between the eigenvalues of the covariance matrix for the noisy observation

and those for its latent clean image . Since

is assumed to follow a Gaussian distribution with zero mean and variance

, the eigenvalue decomposition of the covariance matrix can be expressed as

Then we get the relationship between the eigenvalues of the noisy and clean images:


Combining (14) and the equation above, we have . As a result, model (13) can be rewritten as


For any , in the model (2.1), we have that .

2.4 Optimization and algorithm

We first divide all noise-free image patches into classes and learn the parameter by GMM. Then there are only three unknowns in the proposed model (2.1): the latent image , class label and reshaped low rank matrices , . An alternating minimization procedure is used to solve our model. We start with the preprocessed image . In the -th iteration, the solutions to the alternating minimization scheme are detailed as follows.

(1) Fix , solve for the class label .

For each patch

, we use the probability density function of the learned GMM to calculate the likelihood within each class,


Then the with the maximum probability is assigned to .

(2) Fix , we calculate low rank matrices , , according to


Like Gu et al. [19], we solve the low rank denoising model (2.1) by the formulas as follows:


here is the singular value matrix of , denotes the singular value decomposition (SVD) of , is the number of similar patches, and is the variance of image in the -th iteration.

(3) Fix , , a weighted averaging method is used to reconstruct the estimated image by aggregating all denoised patches. The value of the -th pixel in the reference image is calculated as a weighted average as follows,


where refers to the denoised image intensity of the -th pixel in the -th patch of -th patch group , and the summation is carried out over all patches that overlap with the -th pixel. The empirical weight is given as


is the rank of matrix , is the number of similar patches in the -th class.

In addition, at the beginning of each iteration , the desired denoised image, preprocessed image and standard deviation are updated by


where means the -th iteration, is the iteration regularization parameter, is the noisy image, is the denoised image in the -th iteration, is the noise variance of the input noisy image, and is the scale factor of controlling the re-estimation noise standard deviation . The complete optimization process is shown in Algorithm 1.

Input: Noisy image , , ,
 Intialize: 1) 
     2) BM3D preprocessed image 
1 for    do
2       Interative regularization and by Eqs. (22) and (23), respectively for    do
3             Update  Calculate  via Eq.(17)
4       end for
5      Compute by finding the class that has the largest conditional probability for    do
6             Create by stacking patches from the -th class Sigular value decomposition
7                Get the estimation 
8               via Eq. (19)
9       end for
10      Aggregate  to form denoised image  via Eq. (20) Update by Eq. (24)
11 end for
Output: Restored image 
Algorithm 1 Preprocessed image based GMM prior low rank regularization (PG-LR)

2.5 Complexity and convergence

Suppose that there are patches extracted from the whole image and pixels in each patch. The main computational cost in each iteration is mainly composed of three parts. Firstly, classifying patches into classes costs , where the covariance matrix determinant needs calculations. Next, the computation of the low rank matrix reconstruction by SVD for clusters takes . Finally, averaging all the grouping similar patches needs computations. Thus the computation complexity is and the classification step takes most computations of the algorithm.

(a) (b)
Figure 1: (a) and (b) are graphs of the PSNR and SSIM variation curve with iteration , .

Experience shows our method has a rapid convergence and reaches the optimal solution at the -th or -th iteration although it is non-convex. Figure 1 compares the convergence speed between our algorithm and OLRA [39]. In the comparative experiment, the Cameraman image with intricate details was selected. Figure 1 (a) and (b) describe the variation curve of PSNR and SSIM values, respectively. It’s obvious that our method is always superior than OLRA [39] after the -th iteration. As shown in Table 1, our algorithm is almost the fastest among all low rank based methods.

Image size
BM3D[10] 1.26 3.22
WNNM [19] 117.97 677.67
PCLR [8] 196.11 1090.40
OLRA  [39] 202.20 1014.63
DnCNN [36] 18.83 15.93
FFDNet [37] 16.49 17.23
Ours 129.45 614.10
Table 1: The average computation time(s) for restoring grayscale image with noise level on a desktop (Intel(R) Core(TM) i5-8250 CPU @1.60 GHz) with MATLAB 2019b.
Cameraman Monarch House Peppers Montage Lena Barbara Boat Couple Hill
Figure 2: All test images of the simulation experiment.
Traditional algorithm Deep Learning
PSNR/SSIM BM3D [10] WNNM [19] PCLR [8] OLRA [39] Ours FFDNet [37] DnCNN [36] DnCNN*[36]
25 Cameraman 29.45/0.854 29.63/0.860 29.66/0.863 29.58/0.863 29.90/0.870 30.05/0.877 30.18/0.876 28.87/0.823
Monarch 29.25/0.890 29.85/0.909 29.75/0.908 29.92/0.912 30.26/0.918 30.14/0.917 30.23/0.917 29.26/0.879
House 32.86/0.859 33.23/0.861 32.99/0.859 33.25/0.863 33.50/0.864 33.27/0.862 33.08/0.860 31.65/0.825
Peppers 30.16/0.868 30.40/0.873 30.46/0.875 30.35/0.874 30.76/0.881 30.79/0.884 30.84/0.881 29.49/0.840
Montage 32.37/0.926 32.74/0.933 32.36/0.930 32.26/0.929 32.74/0.934 32.84/0.941 32.95/0.937 30.68/0.879
Lena 32.08/0.861 32.25/0.866 32.17/0.863 32.30/0.867 32.55/0.868 32.59/0.874 32.44/0.869 31.12/0.824
Barbara 30.72/0.874 31.24/0.898 30.65/0.887 31.20/0.897 31.07/0.894 29.98/0.879 30.00/0.878 26.78/0.785
Boat 29.91/0.801 29.83/0.795 30.00/0.800 29.95/0.803 30.34/0.811 30.23/0.812 30.21/0.809 29.45/0.780
Couple 29.72/0.820 29.82/0.820 29.73/0.814 29.77/0.819 30.11/0.826 30.18/0.835 30.12/0.830 29.08/0.788
Hill 29.85/0.775 29.95/0.776 29.84/0.771 29.94/0.778 30.24/0.785 30.04/0.782 30.03/0.780 29.43/0.758
Average 30.64/0.855 30.91/0.860 30.76/0.857 30.85/0.861 31.15/0.865 31.01/0.866 31.01/0.864 29.58/0.818
50 Cameraman 26.12/0.782 26.41/0.785 26.55/0.795 26.35/0.780 26.83/0.794 27.02/0.811 27.00/0.805 25.75/0.704
Monarch 25.82/0.820 26.32/0.835 26.25/0.837 26.21/0.835 26.67/0.849 26.88/0.854 26.76/0.846 25.46/0.767
House 29.69/0.812 30.33/0.823 29.77/0.819 30.19/0.816 30.40/0.818 30.43/0.829 30.01/0.820 27.98/0.711
Peppers 26.68/0.794 26.91/0.801 27.03/0.808 27.06/0.800 27.69/0.820 27.43/0.820 27.29/0.810 25.93/0.724
Montage 27.90/0.861 28.27/0.875 28.20/0.877 28.34/0.872 28.95/0.883 29.18/0.898 28.95/0.882 26.85/0.772
Lena 29.05/0.799 29.25/0.806 29.12/0.801 29.01/0.798 29.68/0.811 29.68/0.823 29.36/0.811 27.78/0.705
Barbara 27.23/0.795 27.79/0.820 27.11/0.798 27.65/0.811 27.90/0.821 26.48/0.779 26.23/0.770 24.08/0.642
Boat 26.78/0.705 26.97/0.708 26.99/0.708 26.84/0.701 27.47/0.721 27.32/0.726 27.19/0.718 26.30/0.660
Couple 26.46/0.707 26.64/0.714 26.55/0.706 26.65/0.714 27.16/0.727 27.07/0.737 26.89/0.724 25.83/0.651
Hill 27.19/0.675 27.33/0.676 27.23/0.672 27.24/0.671 27.85/0.690 27.55/0.688 27.44/0.683 26.55/0.631
Average 27.29/0.775 27.62/0.784 27.48/0.782 27.55/0.780 28.06/0.793 27.90/0.796 27.71/0.787 26.25/0.697
75 Cameraman 24.33/0.734 24.55/0.735 24.74/0.747 24.65/0.741 25.16/0.753 25.29/0.770 25.09/0.759 22.48/0.466
Monarch 23.91/0.756 24.31/0.775 24.28/0.781 24.10/0.776 24.71/0.796 24.99/0.801 24.70/0.783 22.34/0.588
House 27.51/0.765 28.24/0.789 27.57/0.785 28.38/0.795 28.78/0.803 28.43/0.796 27.83/0.776 23.82/0.466
Peppers 24.73/0.737 24.92/0.742 25.06/0.762 24.95/0.748 25.67/0.768 25.39/0.769 25.15/0.752 22.83/0.526
Montage 25.52/0.800 25.72/0.821 25.70/0.827 25.56/0.831 26.57/0.851 26.84/0.857 26.30/0.829 23.36/0.516
Lena 27.26/0.752 27.54/0.766 27.41/0.766 25.96/0.766 28.16/0.776 27.57/0.767 27.97/0.786 24.03/0.471
Barbara 25.12/0.711 25.81/0.749 25.07/0.722 25.68/0.740 26.03/0.753 24.24/0.685 23.62/0.654 21.95/0.462
Boat 25.12/0.641 25.30/0.647 25.29/0.649 25.00/0.634 25.78/0.658 25.64/0.666 25.47/0.655 23.01/0.461
Couple 24.70/0.626 24.86/0.637 24.80/0.629 24.71/0.627 25.43/0.651 25.29/0.662 24.99/0.640 22.87/0.466
Hill 25.68/0.612 25.88/0.619 25.82/0.616 25.59/0.604 26.36/0.625 26.15/0.633 25.95/0.621 23.59/0.454
Average 25.39/0.713 25.71/0.728 25.57/0.728 25.46/0.726 26.27/0.743 26.02/0.742 25.67/0.724 23.03/0.487
100 Cameraman 23.07/0.692 23.36/0.697 23.48/0.715 23.32/0.700 23.96/0.721 23.96/0.740
Monarch 22.52/0.702 22.95/0.726 22.93/0.737 22.86/0.727 23.62/0.758 23.61/0.759
House 25.87/0.720 26.66/0.754 25.97/0.750 26.66/0.755 27.22/0.770 26.91/0.768
Peppers 23.39/0.688 23.45/0.698 23.70/0.725 23.66/0.711 24.45/0.743 23.96/0.733
Montage 23.89/0.747 24.16/0.777 24.23/0.794 23.86/0.783 25.07/0.818 25.12/0.824
Lena 25.95/0.709 26.21/0.726 26.16/0.736 26.96/0.725 27.02/0.741 26.77/0.756
Barbara 23.62/0.643 24.37/0.686 23.65/0.660 24.10/0.672 24.63/0.691 22.82/0.614
Boat 23.97/0.594 24.11/0.598 24.16/0.607 23.83/0.594 24.77/0.618 24.53/0.624
Couple 23.51/0.567 23.56/0.570 23.65/0.575 23.55/0.567 24.37/0.597 24.05/0.600
Hill 24.58/0.565 24.76/0.572 24.83/0.576 24.53/0.562 25.45/0.587 25.15/0.591
Average 24.04/0.663 24.36/0.680 24.28/0.688 24.23/0.680 25.06/0.704 24.69/0.701
Table 2: Comparison in PSNRSSIM values between the state-of-the-art denoising method and ours with noise level . The best PSNRSSIM values in all traditional methods are marked in bold and the best result of each row is marked in red. A transitional method wins in the raw marked in both bold and red. As references we also include the results of DnCNN [36] and FFDNet [37]. Notes: 1. DnCNN* [36] is the result of directly using the DnCNN network in the matlab 2019b deep learning toolbox; 2.Results limited to , as the DnCNN network is not trained beyond that level.
(a) Original (b) Noisy image (c) BM3D[10] 27.51/0.765 (d) WNNM[19] 28.24/0.789 (e) PCLR[8] 27.57/0.785
(f) OLRA[39] 28.38/0.795 (g) FFDNet[37] 28.43/0.796 (h) DnCNN[36] 27.83/0.776 (i) DnCNN*[36] 23.82/0.466 (j) Ours 28.78/0.803
Figure 3: Denoising results on image House by different methods with noise level .
(a) Original (b) Niosy image (c) BM3D[10] 23.07/0.692 (d) WNNM[19] 23.36/0.697
(e) PCLR [8] 23.48/0.715 (f) OLRA[39] 23.32/0.700 (g) FFDNet[37] 23.96/0.740 (h) Ours 23.96/0.721
Figure 4: Denoising results on image Cameraman by different methods with noise level  . There is no restored image of DnCNN in this figure because it is untrained for .

3 Experimental results and discussion

From the BSD dataset [26], 200 training images are uniformly sampled to image patches, where GMM with 250 mixing components and parameter are learned. The related initialization parameters are set to , , and the number of iterations .

The test images are selected from the BSD dataset [26] (see Figure 2). There are five images and five images. Our algorithm PG-LR is compared with several state-of-the-art algorithms, including BM3D [10], WNNM [19], PCLR [8], OLRA [39], DnCNN [36] and FFDNet [37]. All source codes are got from the authors.

Quantitatively, we evaluate the quality of image restoration by the peak signal to noise ratio (PSNR) and structural similarity index (SSIM). The SSIM is defined by Wang et al. 

[33] and PSNR is given by


is the image size, is the original image and is the restored image.

We record the values of PSNR / SSIM of restored image at various Gaussian noise level in Table 2. The table confirms that our method achieves the best results in almost all the cases in both PSNR and SSIM values among the traditional denoising algorithms and is also a little better than some deep learning based methods. Furthermore, our algorithm wins BM3D [10] more than in average when .

For visual image quality, Figures 3 and 4 show that the proposed method yields the best image visual quality in terms of removing noise, preserving edges and maintaining image details. Taking Figure 3 for example, the part zoomed in shows that our proposed method obtains pleasurable result at window frame while BM3D blurs the window. The low rank based algorithms (WNNM [19], PCLR [8] and OLRA [39]) introduce unpleasant artifacts, and the deep learning based methods cut the structure at the lower left corner of the window.

When the noise level is extremely high, the structural characteristics of the image are severely damaged and the restoration task becomes much arduous. In Figure 4, the cameraman’s ear is apparently restored almost completely by our algorithm, while other methods introduce serious artifacts in the hair and destroy the ear. FFDNet [37] sharpens the edges but removes some structures partly such as the details of the camera. Though our approach blurs the edges a little bit, it preserves most of the structures. In summary, our algorithm remains more detail information such as structures and edges than other advanced methods.

In brief, the proposed algorithm has competitive performance of removing noise and remaining details. It outperforms the state-of-the-art methods both quantitatively and in visual quality, including several deep learning methods such as DnCNN [36] and FFDNet [37] .

4 Conclusion

In this paper, we propose a novel method by combining Gaussian patch mixture model and low rank matrix restoration. We conduct extensive experiments at different Gaussian noise levels, showing that the proposed approach is robust and outperforms the state-of-the-art in image denoising. The denoising capability of our proposed method largely benefits from the accuracy of similar patches selection, thereby it makes up for the lack of ability for low rank approximation and maximum expectation algorithms to maintain the details and edges. Future works include: (1) Global patch matching greatly increases the complexity by spreading over all image patches to find their similar ones. How to reduce the computational complexity and ensure great denoising performance is worth discussing. (2) The parameters in the simulation experiment are selected according to experience. The slight changes of a parameter will also make difference. Therefore, how to set an adaptive parameter function is also a point worth studying. (3) In the process of experiments, we find that when performing the classification operation, some classes have only a few similar patches while some others get so many ones. It is rational to set appropriate classes to balance the number of similar patches in each class, thereby enhancing the robustness of the algorithm. (4) The future work also includes extensions to other types of noise such as Poisson noise, mixed noise and unknown type of noise.


Jin has been supported by the National Natural Science Foundation of China (Grant No. 12061052), Natural Science Fund of Inner Mongolia Autonomous Region (Grant No. 2020MS01002), China Scholarship Council for a one year visiting at Ecole normale supérieure Paris-Saclay (No. 201806810001).


  • [1] Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    , volume 2, pages 60–65, 2005.
  • [2] Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956–1982, 2010.
  • [3] Jian-Feng Cai, Raymond H Chan, and Carmine Di Fiore. Minimization of a detail-preserving regularization functional for impulse noise removal. Journal of Mathematical Imaging and Vision, 29(1):79–91, 2007.
  • [4] Jian-Feng Cai, Hui Ji, Zuowei Shen, and Gui-Bo Ye. Data-driven tight frame construction and image denoising. Applied and Computational Harmonic Analysis, 37(1):89–105, 2014.
  • [5] Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright.

    Robust principal component analysis?

    Journal of the ACM (JACM), 58(3):1–37, 2011.
  • [6] Raymond H Chan, Yiqiu Dong, and Michael Hintermuller. An efficient two-phase -tv method for restoring blurred images with impulse noise. IEEE Transactions on Image Processing, 19(7):1731–1739, 2010.
  • [7] Priyam Chatterjee and Peyman Milanfar. Learning denoising bounds for noisy images. In IEEE International Conference on Image Processing (ICIP), pages 1157–1160, 2010.
  • [8] Fei Chen, Lei Zhang, and Huimin Yu. External patch prior guided internal clustering for image denoising. In IEEE International Conference on Computer Vision (ICCV), pages 603–611, 2015.
  • [9] Jingwen Chen, Jiawei Chen, Hongyang Chao, and Yang Ming.

    Image blind denoising with generative adversarial network based noise modeling.

    In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  • [10] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):2080–2095, 2007.
  • [11] Jiayu Dong, Huicheng Zheng, and Lina Lian. Low-rank laplacian-uniform mixed model for robust face recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11889–11898, 2019.
  • [12] Weisheng Dong, Guangming Shi, and Xin Li. Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Transactions on Image Processing, 22(2):700–711, 2012.
  • [13] Yiqiu Dong, Raymond H Chan, and Shufang Xu. A detection statistic for random-valued impulse noise. IEEE Transactions on Image Processing, 16(4):1112–1120, 2007.
  • [14] David L Donoho. De-noising by soft-thresholding. IEEE Transactions on Information Theory, 41(3):613–627, 1995.
  • [15] N Benjamin Erichson, Steven L Brunton, and J Nathan Kutz. Compressed dynamic mode decomposition for background modeling. Journal of Real-Time Image Processing, 16(5):1479–1492, 2019.
  • [16] Sarjoui M Fazel. Matrix rank minimization with applications. 2003.
  • [17] Guangwei Gao, Jian Yang, Xiao-Yuan Jing, Fumin Shen, Wankou Yang, and Dong Yue. Learning robust and discriminative low-rank representations for face recognition with occlusion. Pattern Recognition, 66:129–143, 2017.
  • [18] Qi Ge, Xiao-Yuan Jing, Fei Wu, Zhi-Hui Wei, Liang Xiao, Wen-Ze Shao, Dong Yue, and Hai-Bo Li. Structure-based low-rank model with graph nuclear norm regularization for noise removal. IEEE Transactions on Image Processing, 26(7):3098–3112, 2016.
  • [19] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 2862–2869, 2014.
  • [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 770–778, 2016.
  • [21] Sajid Javed, Arif Mahmood, Thierry Bouwmans, and Soon Ki Jung. Spatiotemporal low-rank modeling for complex scene background initialization. IEEE Transactions on Circuits and Systems for Video Technology, 28(6):1315–1329, 2016.
  • [22] Hui Ji, Chaoqiang Liu, Zuowei Shen, and Yuhong Xu. Robust video denoising using low rank matrix completion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1791–1798, 2010.
  • [23] Qiyu Jin, Ion Grama, Charles Kervrann, and Quansheng Liu. Nonlocal means and optimal weights for noise removal. SIAM Journal on Imaging Sciences, 10(4):1878–1920, 2017.
  • [24] Qiyu Jin, Ion Grama, and Quansheng Liu. Convergence theorems for the non-local means filter. Inverse Problems & Imaging, 12(4):853–881, 2018.
  • [25] Zhouchen Lin, Risheng Liu, and Zhixun Su. Linearized alternating direction method with adaptive penalty for low-rank representation. In Advances In Neural Information Processing Systems (NIPS), pages 612–620, 2011.
  • [26] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE International Conference on Computer Vision (ICCV), volume 2, pages 416–423. IEEE, 2001.
  • [27] Yigang Peng, Arvind Ganesh, John Wright, Wenli Xu, and Yi Ma. Rasl: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2233–2246, 2012.
  • [28] Xutong Ren, Wenhan Yang, Wen-Huang Cheng, and Jiaying Liu. Lr3m: Robust low-light enhancement via low-rank regularized retinex model. IEEE Transactions on Image Processing, 29:5862–5876, 2020.
  • [29] Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992.
  • [30] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  • [31] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In IEEE International Conference on Computer Vision (ICCV), pages 839–846, 1998.
  • [32] Hang Wang, Siyuan Cao, Kangkang Jiang, Hao Wang, and Qingchen Zhang. Seismic data denoising for complex structure using bm3d and local similarity. Journal of Applied Geophysics, 170, 2019.
  • [33] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  • [34] Zhaojun Wu, Qiang Wang, Jing Jin, and Yi Shen.

    Structure tensor total variation-regularized weighted nuclear norm minimization for hyperspectral image mixed denoising.

    Signal Processing, 131:202–219, 2017.
  • [35] Jun Xu, Lei Zhang, Wangmeng Zuo, David Zhang, and Feng Xiangchu. Patch group based nonlocal self-similarity prior learning for image denoising. In IEEE International Conference on Computer Vision (ICCV), pages 244–252, 2015.
  • [36] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
  • [37] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
  • [38] Xiaoqin Zhang, Di Wang, Zhengyuan Zhou, and Yi Ma. Robust low-rank tensor recovery with rectification and alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
  • [39] Yongqin Zhang, Jinsheng Xiao, Jinye Peng, Yu Ding, Jiaying Liu, Zongming Guo, and Xiaopeng Zong. Kernel wiener filtering model with low-rank approximation for image denoising. Information Sciences, 462:402–416, 2018.
  • [40] Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In IEEE International Conference on Computer Vision (ICCV), pages 479–486, 2011.