share
1 Introduction
As a classical and fundamental problem in low level vision, image denoising has been extensively explored and it is still under investigation as well in methodological aspects. In general, image denoising aims to recover the latent noise-free image from its noise-corrupted version. During the past decades, image denoising has got great development in a variety of models including spatial filtering [31], total variation [29], wavelet/curvelet transform [14], noise detector [3, 6, 13], sparse representation [4, 12], non-local means and its varieties [1, 8, 10, 19, 39], deep learning[9, 20, 30, 36, 37], etc.
The landmark work of image denoising is non-local means (NLM) [1] which brings a new era by finding non-local similar patches within the image. Quite a few patch-based denoising methods [8, 10, 12, 19, 32, 37] achieve great success by exploiting the internal self-similarity prior. A quintessential example should be cited that BM3D [10] uses 3D filtering in transformed domain by patch matching, which is quite effective and has been a benchmark in image denoising. Here, patch matching is performed within a relatively large local window instead of globally, and consequently the number of patches may not be sufficient that leads to ringing artifacts. In order to improve the accuracy of patch matching, external natural patch priors are used to regularize the denoising process. Method [7] learns the covariance matrix and the measure of redundancy for each patch based on clustered natural noise-free patch database. Zoran and Weiss [40]
gave high likelihood values for patches sampled from clean images using Gaussian mixture models (GMM), and reconstructed the latent image by maximizing the expected patch log likelihood. Though external patch priors based approaches have achieved remarkable success in image restoration, they are still local and do not make full use of image internal self-similarity. Some methods
[12, 19] globally search for the most similar patches across the whole image and have demonstrated competitive results. Though global patch matching is to determine enough similar patches, the error rate of patch matching may be higher, which results in destroying image details unsatisfactorily. Methods PCLR [8] and PGPD [35] integrate external patch priors and internal self-similarity into one framework. They learn GMM prior information from noise-free image patches to guide the noisy patches classification and the subsequent low rank subspace learning, which makes the latent patch reconstruction more robust to noise. However, there is no obvious improvement in denoising performance, compared with local patch matching methods. We argue that such utilization of GMM prior information is not effective enough because it neglects the noise of degraded image, which interferes with the selection of similar patches. How to precisely select similar patches is still an open problem.Besides, low rank matrix approximation (LRMA), which aims to recover the underlying low rank matrix from its degraded observation, appears in a wide range of applications in various fields including computer vision, machine learning, signal processing and bioinformatics. For instance, use of low rank approximation can be found in applications such as face recognition
[17, 11], background modeling and foreground object detection [19, 21, 15], image alignment [27, 38], video denoising [22, 28] image restoration [8, 33, 12], among others. Since direct rank minimization is difficult to solve, the problem is generally relaxed by substitutively minimizing the nuclear norm of the estimated matrix
[16]. The nuclear norm of a matrix , denoted by, is defined as the sum of its singular values, i.e.
, where denotes the -th singular value of . The nuclear norm minimization model has been attracting significant attention due to its rapid development in both theory and implementation. Cai et al. [2] proved that the model can be easily solved in a closed form by imposing a soft thresholding operation on the singular values of the observation matrix. Candès et al. [5]proved that from the noisy input, its intrinsic low-rank reconstruction can be exactly achieved with a high probability by the model. Many state-of-the-art denoising methods
[18, 34, 25, 12] have been proposed in recent years by utilizing nuclear norm proximal as the key proximal technique. Albeit its success as mentioned before, nuclear norm minimization model still has certain limitations. Gu et al. [19] proposed weighted nuclear norm minimization model trying to achieve a better estimation of the latent data from the corrupted input. The weighted nuclear norm of a matrix is given by , where and is a non-negative weight assigned to . The nuclear norm minimization model provides superior performance and efficiency for matrix completion or impulse noise. However, they ignore the difference between Gaussian noise and impulse noise, then building a low rank model for Gaussian noise is still a worthwhile work.This article will address two issues: (1) how to improve similar patches matching accuracy; (2) build an appropriate low rank matrix approximation model for Gaussian noise.
Firstly, unlike PCLR [8] and PGPD [35] clustering the noisy patches directly, we first carry out a preliminary denoising of the image by a local patch matching based denoising algorithm and get a preprocessed image, then cluster all preprocessed patches into
classes. If a class contains too many similar patches, we use k-means to divide the big class into some smaller ones. We next record all patches location information. Consequently, in order to classify noisy patches, we replace all preprocessed patches by the corresponding noisy ones in each class according to the location information. Furthermore, the preprocessed image is updated in each iteration in order to improve patch matching accuracy. This idea significantly increases the accuracy of similar patch clustering. Here, we use a local patch matching based denoising method because it matches similar patches with higher precision. Since BM3D
[10] is the state-of-the-art local patch matching based method and a real-time computing algorithm, we use it as a preprocessing algorithm in this paper. Though some global patch matching based methods outperform BM3D, they introduce some errors for patch matching and the errors can’t be corrected in the future denoising steps. In addition, the methods PCLR [8] and PGPD [35] are iterative algorithms. This means that the methods PCLR [8] and PGPD [35] have already used a global patch matching based algorithm (the -th step denoising) as a preprocessing algorithm for the -th step denoising.Secondly, according to the characteristics of Gaussian noise, the minimization of i.e. the square of the Frobenius norm of adapts Gaussian noise model. Inspired by [39] and the equation , similar to nuclear norm minimization model [2], we will establish Gaussian nuclear norm minimization model by minimizing and then efficiently solve the problem.
As noted above, the proposed method uses GMM to achieve both local and global patch matching, and the mixture of patches can be further regularized via k-means clustering procedure and low rank minimization. This process may seem complicated, but it is very effective. In the low rank denoising part, we use the soft threshold given by [19] which is good to smooth the image. The proposed model is solved by an alternating minimization procedure and it converges rapidly.
The rest of the paper is organized as follows. In Section 2, we propose our algorithm by using GMM so that both local and global patch matching can be achieved. The complexity and convergence of the algorithm are also been discussed. Section 3 shows experiment results and discussion. Conclusions and future works are given in Section 4.
2 The Proposed Algorithm
Throughout this paper, we denote scalars, vectors and matrices by non boldfaced, boldfaced lower-case, and boldfaced upper-case letters, respectively.
2.1 The low rank regularization model based on GMM prior
The noise-corrupted images are often represented by the following simplified model:
| (1) |
where is noisy image, is latent original image and is additive white Gaussian (AWG) noise matrix of mean
. Let and be the vectorized image patches which are extracted from the noisy image and latent original image at the -th pixel, respectively. All noisy patches are divided into similar patch groups. Consequently, these similar patches in -th group are collected to form a reshaped matrix , where may be different for each . The noise model (1) can be rewritten in the patch-based representation as follow:| (2) |
where denote the patch reshaped matrices from the noisy image , original image and noise matrix , respectively.
In this paper, we will introduce preprocessed image based GMM prior low rank regularization model (PG-LR):
| (3) |
where is the Frobenius norm, denotes nuclear norm. is preprocessed image based patch cluster log likelihood term, is the -th patch extracted from preprocessed image , and denotes which class is from. Our model effectively incorporates image internal self-similarity and external patch priors into the low rank denoising model. There are two advantages of such clustering based low rank regularization.
First, the Gaussian patch mixture model based patch clustering can take the advantages of local and global patch matching. Local patch matching is to find similar patches in a large neighborhood which can alleviate noise effect, while the number of patches may not be sufficient. Global patch matching aims to determine enough similar patches but the error rate of patch matching may be higher. The model first use local patch matching method to reduce noise and then use Gaussian patch mixture model to achieve global patch matching. The detail will be discussed in Subsection 2.2.
Second, the nuclear norm minimization model [2] is an outstanding model for removing impulse noise and inpainting, but not adapt to Gaussian noise. Here according to the statistic properties of Gaussian noise and the nuclear norm minimization model [2], we build a new model for Gaussian noise: , where is the number of rows in the matrix . We prove that there is a global optimal solution of the model and the simulation experiment shows that our model is much better than the nuclear norm minimization model [2] for Gaussian noise (see the supplementary material). Subsection 2.3 will give details.
2.2 Improved Gaussian Patch Mixture Model
In this subsection, we first recall GMM model [40]. All noise-free image patches , () are divided into groups with a parameter , , which describes a Gaussian density function parameterized by the mean and covariance matrix . Thus, the probability of each patch could be defined as a weighted sum of -Gaussians:
| (4) |
where
| (5) |
and is the transpose of , and is the set of parameters with and , .
We investigate the model (4) and discover that is unknown latent image patch and consequently it’s impossible to calculate . In the model (4), it normally takes noisy patch instead of noise-free patch . As a result, the error rate of patch clustering increases significantly due to noise. In this paper, we will solve the problem.
The latent image patch is unknown and the noisy image patch is unfavorable, then we should find another solution. There are quite a few excellent denoising algorithms [1, 8, 10, 19], and a distinguished one is BM3D [10]. BM3D is a real-time computing algorithm and gets favorable restoration results. Furthermore, it matches similar patches in a large neighborhood, which alleviates noise effect for selection of similar patches. We take BM3D as a image preprocessing algorithm and get a preprocessed image that is a rosy estimate of the noise-free one. For this reason, the preprocessed image based patch cluster log likelihood (P-CLL) term is given as follow:
| (6) |
Obviously, our model is feasible and efficient as we know is the pleasurable estimate of . Consequently, we cluster all preprocessed patches into classes by maximizing the P-CLL term. However, the number of image patches varies greatly from class to class. In order to equalize the number of similar patches in each class, we further adjust the classification. If a class doesn’t contain enough similar image patches, it will be merged with the nearest one. If there are excessive patches in a class, k-means is used to divide the large class into some finer ones. Instead of constructing low rank matrices, these preprocessed image patches in each class are used to record patches location information. Finally, all the noisy image patches are clustered by the guide of the location information of the corresponding preprocessed ones. This clustering process is somewhat complex, but the accuracy is undoubtedly higher. As a result, the rank of the created matrix by stacking patches from each class is lower.
Note: why not we use a denoising algorithm based on global patch matching? Jin et al. [23, 24] have proved that non-local means is a local method with a large neighborhood and there is an optimal size of search window. Though global patch matching aims to acquire enough similar patches, the error rate of patch matching is intensely higher because of noise interference. In the image preprocessing, if an image patch is classified into a wrong class, the error will not be corrected in later process. For BM3D [10], the number of patches may be insufficient and the result may not as good as global patch matching based methods (such as WNNM [19], PCLR [8] and PGPD [35]), but the error rate of patch matching is lower. The restoration result will be improved in the future denoising steps but the error will exist forever. Another reason for not using global patch matching based methods is that they run slowly.
2.3 Gaussian nuclear norm minimization for low rank modeling
Consider the singular value decomposition (SVD) of a matrix
of rank :where and are respectively and matrices with orthonormal columns and the singular values are positive and satisfy that .
Nuclear norm minimization (NNM) [16] is an effective rank minimization. The nuclear norm of a matrix , denoted by , is defined as the sum of its singular values, i.e.
| (7) |
The NNM approach has been attracting significant attention due to its rapid development in both theory and implementation. The nuclear norm proximal (NNP) problem [2] is
| (8) |
where is a positive constant.
Theorem A (Cai et al. [2]) For each , the global optimum of model (8) is
| (9) |
where denotes the -th singular value of and .
Albeit its success as aforementioned, NNM ignores the prior knowledge we often have about singular values of a practical data matrix. More specifically, larger singular values of an input data matrix quantify the information of its underlying principal directions. In order to improve the flexibility of NNM, Gu et al. [19] proposed the weighted nuclear norm. The weighted nuclear norm of a matrix is defined as
| (10) |
where is a non-negative weight assigned to . The weight vector will enhance the representation capability of the original nuclear norm. The weighted nuclear norm proximal (WNNP) problem is given by
| (11) |
Theorem B (Gu et al. [19]) If the weight satisfies , the global optimum of model (11) is
| (12) |
The WNNM model is more difficult to optimize than conventional NNM ones due to the non-convexity of the involved weighted nuclear norm.
In this paper, we consider the proprieties of Gaussian noise. By tradition, the LRMA for Gaussian noise is to minimize . Note that , we adapt NNM model (8) to Gaussian noise by utilizing to take place of . Then we have Gaussian nuclear norm minimization (GNNM):
| (13) |
We prove that the Gaussian nuclear norm minimization problem can be equivalently transformed to a quadratic programming (QP) problem with linear constraints. This allows us to easily reach the global optimum of the original problem by using off-the-shelf convex optimization solvers. In order to solve the problem (13), we introduce the following theorem.
Theorem 1 For each , the global optimum of model (13) is
| (14) |
Proof Considering as a new matrix and using Theorem A, we have that the global optimum of model (13) is , where denotes the -th singular value of . Let and , it is easy to get . Then we have
Therefore, is also the global optimum of model (13).
A image contaminated by AGW noise i.e. model (1) can be recovered by the Theorem 1. We next discuss the parameter . If the standard deviation of AWG noise is , we have the shrinkage amount . According to the AGW noise model (1
), we deduce the relationship between the eigenvalues of the covariance matrix for the noisy observation
and those for its latent clean image . Sinceis assumed to follow a Gaussian distribution with zero mean and variance
, the eigenvalue decomposition of the covariance matrix can be expressed as2.4 Optimization and algorithm
We first divide all noise-free image patches into classes and learn the parameter by GMM. Then there are only three unknowns in the proposed model (2.1): the latent image , class label and reshaped low rank matrices , . An alternating minimization procedure is used to solve our model. We start with the preprocessed image . In the -th iteration, the solutions to the alternating minimization scheme are detailed as follows.
(1) Fix , solve for the class label .
For each patch
, we use the probability density function of the learned GMM to calculate the likelihood within each class,
| (17) |
Then the with the maximum probability is assigned to .
(2) Fix , we calculate low rank matrices , , according to
| (18) |
Like Gu et al. [19], we solve the low rank denoising model (2.1) by the formulas as follows:
| (19) | |||
here is the singular value matrix of , denotes the singular value decomposition (SVD) of , is the number of similar patches, and is the variance of image in the -th iteration.
(3) Fix , , a weighted averaging method is used to reconstruct the estimated image by aggregating all denoised patches. The value of the -th pixel in the reference image is calculated as a weighted average as follows,
| (20) |
where refers to the denoised image intensity of the -th pixel in the -th patch of -th patch group , and the summation is carried out over all patches that overlap with the -th pixel. The empirical weight is given as
| (21) |
is the rank of matrix , is the number of similar patches in the -th class.
In addition, at the beginning of each iteration , the desired denoised image, preprocessed image and standard deviation are updated by
| (22) |
| (23) |
| (24) |
where means the -th iteration, is the iteration regularization parameter, is the noisy image, is the denoised image in the -th iteration, is the noise variance of the input noisy image, and is the scale factor of controlling the re-estimation noise standard deviation . The complete optimization process is shown in Algorithm 1.
2.5 Complexity and convergence
Suppose that there are patches extracted from the whole image and pixels in each patch. The main computational cost in each iteration is mainly composed of three parts. Firstly, classifying patches into classes costs , where the covariance matrix determinant needs calculations. Next, the computation of the low rank matrix reconstruction by SVD for clusters takes . Finally, averaging all the grouping similar patches needs computations. Thus the computation complexity is and the classification step takes most computations of the algorithm.
 |
 |
| (a) | (b) |
Experience shows our method has a rapid convergence and reaches the optimal solution at the -th or -th iteration although it is non-convex. Figure 1 compares the convergence speed between our algorithm and OLRA [39]. In the comparative experiment, the Cameraman image with intricate details was selected. Figure 1 (a) and (b) describe the variation curve of PSNR and SSIM values, respectively. It’s obvious that our method is always superior than OLRA [39] after the -th iteration. As shown in Table 1, our algorithm is almost the fastest among all low rank based methods.
| Image size | ||
|---|---|---|
| BM3D[10] | 1.26 | 3.22 |
| WNNM [19] | 117.97 | 677.67 |
| PCLR [8] | 196.11 | 1090.40 |
| OLRA [39] | 202.20 | 1014.63 |
| DnCNN [36] | 18.83 | 15.93 |
| FFDNet [37] | 16.49 | 17.23 |
| Ours | 129.45 | 614.10 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
| Cameraman | Monarch | House | Peppers | Montage | Lena | Barbara | Boat | Couple | Hill |
| Traditional algorithm | Deep Learning | ||||||||
| PSNR/SSIM | BM3D [10] | WNNM [19] | PCLR [8] | OLRA [39] | Ours | FFDNet [37] | DnCNN [36] | DnCNN*[36] | |
| 25 | Cameraman | 29.45/0.854 | 29.63/0.860 | 29.66/0.863 | 29.58/0.863 | 29.90/0.870 | 30.05/0.877 | 30.18/0.876 | 28.87/0.823 |
| Monarch | 29.25/0.890 | 29.85/0.909 | 29.75/0.908 | 29.92/0.912 | 30.26/0.918 | 30.14/0.917 | 30.23/0.917 | 29.26/0.879 | |
| House | 32.86/0.859 | 33.23/0.861 | 32.99/0.859 | 33.25/0.863 | 33.50/0.864 | 33.27/0.862 | 33.08/0.860 | 31.65/0.825 | |
| Peppers | 30.16/0.868 | 30.40/0.873 | 30.46/0.875 | 30.35/0.874 | 30.76/0.881 | 30.79/0.884 | 30.84/0.881 | 29.49/0.840 | |
| Montage | 32.37/0.926 | 32.74/0.933 | 32.36/0.930 | 32.26/0.929 | 32.74/0.934 | 32.84/0.941 | 32.95/0.937 | 30.68/0.879 | |
| Lena | 32.08/0.861 | 32.25/0.866 | 32.17/0.863 | 32.30/0.867 | 32.55/0.868 | 32.59/0.874 | 32.44/0.869 | 31.12/0.824 | |
| Barbara | 30.72/0.874 | 31.24/0.898 | 30.65/0.887 | 31.20/0.897 | 31.07/0.894 | 29.98/0.879 | 30.00/0.878 | 26.78/0.785 | |
| Boat | 29.91/0.801 | 29.83/0.795 | 30.00/0.800 | 29.95/0.803 | 30.34/0.811 | 30.23/0.812 | 30.21/0.809 | 29.45/0.780 | |
| Couple | 29.72/0.820 | 29.82/0.820 | 29.73/0.814 | 29.77/0.819 | 30.11/0.826 | 30.18/0.835 | 30.12/0.830 | 29.08/0.788 | |
| Hill | 29.85/0.775 | 29.95/0.776 | 29.84/0.771 | 29.94/0.778 | 30.24/0.785 | 30.04/0.782 | 30.03/0.780 | 29.43/0.758 | |
| Average | 30.64/0.855 | 30.91/0.860 | 30.76/0.857 | 30.85/0.861 | 31.15/0.865 | 31.01/0.866 | 31.01/0.864 | 29.58/0.818 | |
| 50 | Cameraman | 26.12/0.782 | 26.41/0.785 | 26.55/0.795 | 26.35/0.780 | 26.83/0.794 | 27.02/0.811 | 27.00/0.805 | 25.75/0.704 |
| Monarch | 25.82/0.820 | 26.32/0.835 | 26.25/0.837 | 26.21/0.835 | 26.67/0.849 | 26.88/0.854 | 26.76/0.846 | 25.46/0.767 | |
| House | 29.69/0.812 | 30.33/0.823 | 29.77/0.819 | 30.19/0.816 | 30.40/0.818 | 30.43/0.829 | 30.01/0.820 | 27.98/0.711 | |
| Peppers | 26.68/0.794 | 26.91/0.801 | 27.03/0.808 | 27.06/0.800 | 27.69/0.820 | 27.43/0.820 | 27.29/0.810 | 25.93/0.724 | |
| Montage | 27.90/0.861 | 28.27/0.875 | 28.20/0.877 | 28.34/0.872 | 28.95/0.883 | 29.18/0.898 | 28.95/0.882 | 26.85/0.772 | |
| Lena | 29.05/0.799 | 29.25/0.806 | 29.12/0.801 | 29.01/0.798 | 29.68/0.811 | 29.68/0.823 | 29.36/0.811 | 27.78/0.705 | |
| Barbara | 27.23/0.795 | 27.79/0.820 | 27.11/0.798 | 27.65/0.811 | 27.90/0.821 | 26.48/0.779 | 26.23/0.770 | 24.08/0.642 | |
| Boat | 26.78/0.705 | 26.97/0.708 | 26.99/0.708 | 26.84/0.701 | 27.47/0.721 | 27.32/0.726 | 27.19/0.718 | 26.30/0.660 | |
| Couple | 26.46/0.707 | 26.64/0.714 | 26.55/0.706 | 26.65/0.714 | 27.16/0.727 | 27.07/0.737 | 26.89/0.724 | 25.83/0.651 | |
| Hill | 27.19/0.675 | 27.33/0.676 | 27.23/0.672 | 27.24/0.671 | 27.85/0.690 | 27.55/0.688 | 27.44/0.683 | 26.55/0.631 | |
| Average | 27.29/0.775 | 27.62/0.784 | 27.48/0.782 | 27.55/0.780 | 28.06/0.793 | 27.90/0.796 | 27.71/0.787 | 26.25/0.697 | |
| 75 | Cameraman | 24.33/0.734 | 24.55/0.735 | 24.74/0.747 | 24.65/0.741 | 25.16/0.753 | 25.29/0.770 | 25.09/0.759 | 22.48/0.466 |
| Monarch | 23.91/0.756 | 24.31/0.775 | 24.28/0.781 | 24.10/0.776 | 24.71/0.796 | 24.99/0.801 | 24.70/0.783 | 22.34/0.588 | |
| House | 27.51/0.765 | 28.24/0.789 | 27.57/0.785 | 28.38/0.795 | 28.78/0.803 | 28.43/0.796 | 27.83/0.776 | 23.82/0.466 | |
| Peppers | 24.73/0.737 | 24.92/0.742 | 25.06/0.762 | 24.95/0.748 | 25.67/0.768 | 25.39/0.769 | 25.15/0.752 | 22.83/0.526 | |
| Montage | 25.52/0.800 | 25.72/0.821 | 25.70/0.827 | 25.56/0.831 | 26.57/0.851 | 26.84/0.857 | 26.30/0.829 | 23.36/0.516 | |
| Lena | 27.26/0.752 | 27.54/0.766 | 27.41/0.766 | 25.96/0.766 | 28.16/0.776 | 27.57/0.767 | 27.97/0.786 | 24.03/0.471 | |
| Barbara | 25.12/0.711 | 25.81/0.749 | 25.07/0.722 | 25.68/0.740 | 26.03/0.753 | 24.24/0.685 | 23.62/0.654 | 21.95/0.462 | |
| Boat | 25.12/0.641 | 25.30/0.647 | 25.29/0.649 | 25.00/0.634 | 25.78/0.658 | 25.64/0.666 | 25.47/0.655 | 23.01/0.461 | |
| Couple | 24.70/0.626 | 24.86/0.637 | 24.80/0.629 | 24.71/0.627 | 25.43/0.651 | 25.29/0.662 | 24.99/0.640 | 22.87/0.466 | |
| Hill | 25.68/0.612 | 25.88/0.619 | 25.82/0.616 | 25.59/0.604 | 26.36/0.625 | 26.15/0.633 | 25.95/0.621 | 23.59/0.454 | |
| Average | 25.39/0.713 | 25.71/0.728 | 25.57/0.728 | 25.46/0.726 | 26.27/0.743 | 26.02/0.742 | 25.67/0.724 | 23.03/0.487 | |
| 100 | Cameraman | 23.07/0.692 | 23.36/0.697 | 23.48/0.715 | 23.32/0.700 | 23.96/0.721 | 23.96/0.740 | – | – |
| Monarch | 22.52/0.702 | 22.95/0.726 | 22.93/0.737 | 22.86/0.727 | 23.62/0.758 | 23.61/0.759 | – | – | |
| House | 25.87/0.720 | 26.66/0.754 | 25.97/0.750 | 26.66/0.755 | 27.22/0.770 | 26.91/0.768 | – | – | |
| Peppers | 23.39/0.688 | 23.45/0.698 | 23.70/0.725 | 23.66/0.711 | 24.45/0.743 | 23.96/0.733 | – | – | |
| Montage | 23.89/0.747 | 24.16/0.777 | 24.23/0.794 | 23.86/0.783 | 25.07/0.818 | 25.12/0.824 | – | – | |
| Lena | 25.95/0.709 | 26.21/0.726 | 26.16/0.736 | 26.96/0.725 | 27.02/0.741 | 26.77/0.756 | – | – | |
| Barbara | 23.62/0.643 | 24.37/0.686 | 23.65/0.660 | 24.10/0.672 | 24.63/0.691 | 22.82/0.614 | – | – | |
| Boat | 23.97/0.594 | 24.11/0.598 | 24.16/0.607 | 23.83/0.594 | 24.77/0.618 | 24.53/0.624 | – | – | |
| Couple | 23.51/0.567 | 23.56/0.570 | 23.65/0.575 | 23.55/0.567 | 24.37/0.597 | 24.05/0.600 | – | – | |
| Hill | 24.58/0.565 | 24.76/0.572 | 24.83/0.576 | 24.53/0.562 | 25.45/0.587 | 25.15/0.591 | – | – | |
| Average | 24.04/0.663 | 24.36/0.680 | 24.28/0.688 | 24.23/0.680 | 25.06/0.704 | 24.69/0.701 | – | – | |
 |
 |
 |
 |
 |
| (a) Original | (b) Noisy image | (c) BM3D[10] 27.51/0.765 | (d) WNNM[19] 28.24/0.789 | (e) PCLR[8] 27.57/0.785 |
 |
 |
 |
 |
 |
| (f) OLRA[39] 28.38/0.795 | (g) FFDNet[37] 28.43/0.796 | (h) DnCNN[36] 27.83/0.776 | (i) DnCNN*[36] 23.82/0.466 | (j) Ours 28.78/0.803 |
 |
 |
 |
 |
| (a) Original | (b) Niosy image | (c) BM3D[10] 23.07/0.692 | (d) WNNM[19] 23.36/0.697 |
 |
 |
 |
 |
| (e) PCLR [8] 23.48/0.715 | (f) OLRA[39] 23.32/0.700 | (g) FFDNet[37] 23.96/0.740 | (h) Ours 23.96/0.721 |
3 Experimental results and discussion
From the BSD dataset [26], 200 training images are uniformly sampled to image patches, where GMM with 250 mixing components and parameter are learned. The related initialization parameters are set to , , and the number of iterations .
The test images are selected from the BSD dataset [26] (see Figure 2). There are five images and five images. Our algorithm PG-LR is compared with several state-of-the-art algorithms, including BM3D [10], WNNM [19], PCLR [8], OLRA [39], DnCNN [36] and FFDNet [37]. All source codes are got from the authors.
Quantitatively, we evaluate the quality of image restoration by the peak signal to noise ratio (PSNR) and structural similarity index (SSIM). The SSIM is defined by Wang et al.
[33] and PSNR is given bywhere
is the image size, is the original image and is the restored image.
We record the values of PSNR / SSIM of restored image at various Gaussian noise level in Table 2. The table confirms that our method achieves the best results in almost all the cases in both PSNR and SSIM values among the traditional denoising algorithms and is also a little better than some deep learning based methods. Furthermore, our algorithm wins BM3D [10] more than in average when .
For visual image quality, Figures 3 and 4 show that the proposed method yields the best image visual quality in terms of removing noise, preserving edges and maintaining image details. Taking Figure 3 for example, the part zoomed in shows that our proposed method obtains pleasurable result at window frame while BM3D blurs the window. The low rank based algorithms (WNNM [19], PCLR [8] and OLRA [39]) introduce unpleasant artifacts, and the deep learning based methods cut the structure at the lower left corner of the window.
When the noise level is extremely high, the structural characteristics of the image are severely damaged and the restoration task becomes much arduous. In Figure 4, the cameraman’s ear is apparently restored almost completely by our algorithm, while other methods introduce serious artifacts in the hair and destroy the ear. FFDNet [37] sharpens the edges but removes some structures partly such as the details of the camera. Though our approach blurs the edges a little bit, it preserves most of the structures. In summary, our algorithm remains more detail information such as structures and edges than other advanced methods.
4 Conclusion
In this paper, we propose a novel method by combining Gaussian patch mixture model and low rank matrix restoration. We conduct extensive experiments at different Gaussian noise levels, showing that the proposed approach is robust and outperforms the state-of-the-art in image denoising. The denoising capability of our proposed method largely benefits from the accuracy of similar patches selection, thereby it makes up for the lack of ability for low rank approximation and maximum expectation algorithms to maintain the details and edges. Future works include: (1) Global patch matching greatly increases the complexity by spreading over all image patches to find their similar ones. How to reduce the computational complexity and ensure great denoising performance is worth discussing. (2) The parameters in the simulation experiment are selected according to experience. The slight changes of a parameter will also make difference. Therefore, how to set an adaptive parameter function is also a point worth studying. (3) In the process of experiments, we find that when performing the classification operation, some classes have only a few similar patches while some others get so many ones. It is rational to set appropriate classes to balance the number of similar patches in each class, thereby enhancing the robustness of the algorithm. (4) The future work also includes extensions to other types of noise such as Poisson noise, mixed noise and unknown type of noise.
Acknowledgments
Jin has been supported by the National Natural Science Foundation of China (Grant No. 12061052), Natural Science Fund of Inner Mongolia Autonomous Region (Grant No. 2020MS01002), China Scholarship Council for a one year visiting at Ecole normale supérieure Paris-Saclay (No. 201806810001).
References
-
[1]
Antoni Buades, Bartomeu Coll, and J-M Morel.
A non-local algorithm for image denoising.
In
IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
, volume 2, pages 60–65, 2005. - [2] Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956–1982, 2010.
- [3] Jian-Feng Cai, Raymond H Chan, and Carmine Di Fiore. Minimization of a detail-preserving regularization functional for impulse noise removal. Journal of Mathematical Imaging and Vision, 29(1):79–91, 2007.
- [4] Jian-Feng Cai, Hui Ji, Zuowei Shen, and Gui-Bo Ye. Data-driven tight frame construction and image denoising. Applied and Computational Harmonic Analysis, 37(1):89–105, 2014.
-
[5]
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright.
Robust principal component analysis?
Journal of the ACM (JACM), 58(3):1–37, 2011. - [6] Raymond H Chan, Yiqiu Dong, and Michael Hintermuller. An efficient two-phase -tv method for restoring blurred images with impulse noise. IEEE Transactions on Image Processing, 19(7):1731–1739, 2010.
- [7] Priyam Chatterjee and Peyman Milanfar. Learning denoising bounds for noisy images. In IEEE International Conference on Image Processing (ICIP), pages 1157–1160, 2010.
- [8] Fei Chen, Lei Zhang, and Huimin Yu. External patch prior guided internal clustering for image denoising. In IEEE International Conference on Computer Vision (ICCV), pages 603–611, 2015.
-
[9]
Jingwen Chen, Jiawei Chen, Hongyang Chao, and Yang Ming.
Image blind denoising with generative adversarial network based noise modeling.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. - [10] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):2080–2095, 2007.
- [11] Jiayu Dong, Huicheng Zheng, and Lina Lian. Low-rank laplacian-uniform mixed model for robust face recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11889–11898, 2019.
- [12] Weisheng Dong, Guangming Shi, and Xin Li. Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Transactions on Image Processing, 22(2):700–711, 2012.
- [13] Yiqiu Dong, Raymond H Chan, and Shufang Xu. A detection statistic for random-valued impulse noise. IEEE Transactions on Image Processing, 16(4):1112–1120, 2007.
- [14] David L Donoho. De-noising by soft-thresholding. IEEE Transactions on Information Theory, 41(3):613–627, 1995.
- [15] N Benjamin Erichson, Steven L Brunton, and J Nathan Kutz. Compressed dynamic mode decomposition for background modeling. Journal of Real-Time Image Processing, 16(5):1479–1492, 2019.
- [16] Sarjoui M Fazel. Matrix rank minimization with applications. 2003.
- [17] Guangwei Gao, Jian Yang, Xiao-Yuan Jing, Fumin Shen, Wankou Yang, and Dong Yue. Learning robust and discriminative low-rank representations for face recognition with occlusion. Pattern Recognition, 66:129–143, 2017.
- [18] Qi Ge, Xiao-Yuan Jing, Fei Wu, Zhi-Hui Wei, Liang Xiao, Wen-Ze Shao, Dong Yue, and Hai-Bo Li. Structure-based low-rank model with graph nuclear norm regularization for noise removal. IEEE Transactions on Image Processing, 26(7):3098–3112, 2016.
- [19] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 2862–2869, 2014.
- [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 770–778, 2016.
- [21] Sajid Javed, Arif Mahmood, Thierry Bouwmans, and Soon Ki Jung. Spatiotemporal low-rank modeling for complex scene background initialization. IEEE Transactions on Circuits and Systems for Video Technology, 28(6):1315–1329, 2016.
- [22] Hui Ji, Chaoqiang Liu, Zuowei Shen, and Yuhong Xu. Robust video denoising using low rank matrix completion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1791–1798, 2010.
- [23] Qiyu Jin, Ion Grama, Charles Kervrann, and Quansheng Liu. Nonlocal means and optimal weights for noise removal. SIAM Journal on Imaging Sciences, 10(4):1878–1920, 2017.
- [24] Qiyu Jin, Ion Grama, and Quansheng Liu. Convergence theorems for the non-local means filter. Inverse Problems & Imaging, 12(4):853–881, 2018.
- [25] Zhouchen Lin, Risheng Liu, and Zhixun Su. Linearized alternating direction method with adaptive penalty for low-rank representation. In Advances In Neural Information Processing Systems (NIPS), pages 612–620, 2011.
- [26] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE International Conference on Computer Vision (ICCV), volume 2, pages 416–423. IEEE, 2001.
- [27] Yigang Peng, Arvind Ganesh, John Wright, Wenli Xu, and Yi Ma. Rasl: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2233–2246, 2012.
- [28] Xutong Ren, Wenhan Yang, Wen-Huang Cheng, and Jiaying Liu. Lr3m: Robust low-light enhancement via low-rank regularized retinex model. IEEE Transactions on Image Processing, 29:5862–5876, 2020.
- [29] Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992.
- [30] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
- [31] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In IEEE International Conference on Computer Vision (ICCV), pages 839–846, 1998.
- [32] Hang Wang, Siyuan Cao, Kangkang Jiang, Hao Wang, and Qingchen Zhang. Seismic data denoising for complex structure using bm3d and local similarity. Journal of Applied Geophysics, 170, 2019.
- [33] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
-
[34]
Zhaojun Wu, Qiang Wang, Jing Jin, and Yi Shen.
Structure tensor total variation-regularized weighted nuclear norm minimization for hyperspectral image mixed denoising.
Signal Processing, 131:202–219, 2017. - [35] Jun Xu, Lei Zhang, Wangmeng Zuo, David Zhang, and Feng Xiangchu. Patch group based nonlocal self-similarity prior learning for image denoising. In IEEE International Conference on Computer Vision (ICCV), pages 244–252, 2015.
- [36] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
- [37] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
- [38] Xiaoqin Zhang, Di Wang, Zhengyuan Zhou, and Yi Ma. Robust low-rank tensor recovery with rectification and alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
- [39] Yongqin Zhang, Jinsheng Xiao, Jinye Peng, Yu Ding, Jiaying Liu, Zongming Guo, and Xiaopeng Zong. Kernel wiener filtering model with low-rank approximation for image denoising. Information Sciences, 462:402–416, 2018.
- [40] Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In IEEE International Conference on Computer Vision (ICCV), pages 479–486, 2011.