Multi-channel Weighted Nuclear Norm Minimization for Real Color Image Denoising

05/28/2017 ∙ by Jun Xu, et al. ∙ Xidian University 0

Most of the existing denoising algorithms are developed for grayscale images, while it is not a trivial work to extend them for color image denoising because the noise statistics in R, G, B channels can be very different for real noisy images. In this paper, we propose a multi-channel (MC) optimization model for real color image denoising under the weighted nuclear norm minimization (WNNM) framework. We concatenate the RGB patches to make use of the channel redundancy, and introduce a weight matrix to balance the data fidelity of the three channels in consideration of their different noise statistics. The proposed MC-WNNM model does not have an analytical solution. We reformulate it into a linear equality-constrained problem and solve it with the alternating direction method of multipliers. Each alternative updating step has closed-form solution and the convergence can be guaranteed. Extensive experiments on both synthetic and real noisy image datasets demonstrate the superiority of the proposed MC-WNNM over state-of-the-art denoising methods.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 6

page 7

page 9

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Image denoising is a classical yet fundamental problem for image quality enhancement in computer vision and photography systems. Most of existing denoising algorithms are designed for grayscale images, aiming to recover the clean image

from its noisy observation , where is generally assumed to be additive white Gaussian noise (AWGN). State-of-the-art image denoising methods include sparse representation [1], dictionary learning [2], low-rank approximation [3], non-local self-similarity (NSS) [4] based methods, and the combination of those techniques [2, 1, 5, 6, 7, 8, 3]. Recently, some discriminative denoising methods have also been developed by learning discriminative priors from pairs of clean and noisy images [9, 10, 11, 12].

When the input is a noisy RGB color image, there are mainly three strategies for color image denoising. (1) The first strategy is to apply the grayscale image denoising algorithm to each channel. However, such a straightforward solution will not exploit the spectral correlation among RGB channels, and the denoising performance may not be very satisfying. (2) The second strategy is to transform the RGB image into a less correlated color space, such as YCbCr, and perform denoising in each channel of the transformed space [13, 14]. One representative work along this line is the CBM3D algorithm [14]

. However, the color transform will complicate the noise distribution, and the correlation among color channels is not fully exploited. (3) The third strategy is to perform joint denoising on the RGB channels simultaneously for better use of the spectral correlation. For example, the patches from RGB channels are concatenated as a long vector for processing

[15, 16].

Though joint denoising of RGB channels is a more promising way for color image denoising, it is not a trivial extension from single channel (grayscale image) to multiple channels (color image). The noise in standard RGB (sRGB) space can be approximately modeled as AWGN, but it has different variances for different channels

[17, 18, 19] due to the sensor characteristics and on-board processing steps in digital camera pipelines [19, 20]. This makes the real color image denoising problem much more complex. If the three channels are treated equally in the joint denoising process, false colors or artifacts can be generated [15]. How to account for the different noise characteristics in color channels, and how to effectively exploit the within and cross channel correlation are the key issues for designing a good color image denoising method.

This paper presents a new color image denoising algorithm. Considering that the weighted nuclear norm minimization (WNNM) method [3, 21], which exploits the image NSS property via low rank regularization, has achieved excellent denoising performance on grayscale images, we propose to extend WNNM to real color image denoising. More specifically, we propose a multi-channel WNNM (MC-WNNM) model, which concatenates the patches from RGB channels for rank minimization but introduces a weight matrix to adjust the contributions of the three channels based on their noise levels. The proposed MC-WNNM model no longer has a closed-form solution as in the original WNNM model [21]. We reformulate it into a linear equality-constrained problem with two variables, and solve the relaxed problem under the alternating direction method of multipliers [22] framework. Each variable can be updated with closed-form solutions, and the convergence analysis is given to guarantee a rational termination of the proposed algorithm.

2 Related Work

2.1 Weighted Nuclear Norm Minimization

As a generalization to the nuclear norm minimization (NNM) model [23], the weighted nuclear norm minimization (WNNM) model [3, 21] is described as

(1)

where is the weighted nuclear norm of matrix , () is the weight vector, and is the

th singular value of

. According to the Corollary 1 of [21], if the weights are non-decreasing, the problem (1) has a closed-form solution:

(2)

where

is the singular value decomposition (SVD)

[24] of and is the generalized soft-thresholding operator with weight vector :

(3)

WNNM has demonstrated highly competitive denoising performance on grayscale images. However, if we directly extend it to color image denoising by concatenating the patches from RGB channels, denoising artifacts may happen (please refer to Fig. 1 and the section of experimental results). In this paper, we propose a multi-channel WNNM (MC-WNNM) model for color image denoising, which preserves the power of WNNM and is able to address the noise differences among different channels.

2.2 Real Color Image Denoising

During the last decade, a few methods have been proposed for real color image denoising. Among them, the CBM3D method [14] is a representative one, which first transforms the RGB image into a luminance-chrominance space (e.g., YCbCr) and then applies the benchmark BM3D method [1] to each channel separately. The non-local similar patches are grouped by the luminance channel. In [17]

, Liu et al. proposed the “Noise Level Function” to estimate and remove the noise for each channel in natural images. However, processing each channel separately would often achieve inferior performance to processing the color channels jointly

[15]. Therefore, the methods [25, 26, 16] perform real color image denoising by concatenating the patches of RGB channels into a long vector. However, the concatenation treats each channel equally and ignores the different noise statistics among these channels. The method in [19] models the cross-channel noise in real noisy images as multivariate Gaussian and the noise is removed by the Bayesian non-local means filter [27]. The commercial software Neat Image [28] estimates the noise parameters from a flat region of the given noisy image and filters the noise accordingly. The methods in [19, 28] ignore the non-local self-similarity of natural images [1, 3].

In this paper, we present an effective multi-channel image denoising algorithm, which utilizes the strong low-rank prior of image non-local similar patches, and introduces a weight matrix to balance the multi-channels based on their different noise levels.

3 The Proposed Color Image Denoising Algo- rithm

3.1 The Multi-channel Weighted Nuclear Norm Minimization Model

(a) Clean Red Channel (b) Noisy Red Channel (c) Clean Green Channel (d) Noisy Green Channel
(e) Denoised Red by WNNM (f) Denoised Red by MC-WNNM (g) Denoised Green by WNNM (h) Denoised Green by MC-WNNM
Figure 1: The Red and Green channels of the image “kodim08” from the Kodak PhotoCD Dataset, its synthetic noisy version, and the images recovered by the concatenated WNNM and the proposed MC-WNNM methods.

The color image denoising problem is to recover the clean image from its noisy version , where is the index of R, G, B channels and is the noise in the channel. Patch based image denoising [13, 2, 1, 5, 6, 7, 9, 10, 3, 8, 11, 12] has achieved a great success in the last decade. Given a noisy color image , each local patch of size is extracted and stretched to a patch vector, denoted by , where are the corresponding patches in R, G, B channels. For each local patch , we search the most similar patches to it (including itself) by Euclidean distance in a relatively large local window around it. By stacking the similar patches column by column, we form a noisy patch matrix , where and are the corresponding clean and noise patch matrices.

The noise in standard RGB (sRGB) space could be approximately modeled as additive white Gaussian (AWGN), but noise in different channels has different variances [17, 18, 19]. Therefore, it is problematic to directly apply some grayscale denoising methods to the concatenated vectors or matrices . To better illustrate this point, in Fig. 1, we show a clean image “kodim08” (only the R and G channels are shown due to limit of space), its noisy version generated by adding AWGN to each channel, and the denoised image by applying WNNM [21] to the concatenated patch matrix

. The standard deviations of AWGN added to the R, G, B channels are

, , , respectively. To make WNNM applicable to color image denoising, we set the noise standard deviation as the average deviation of the whole noisy image, i.e., . From Fig. 1, one can see that the concatenated WNNM remains some noise in the R channel while over-smoothing the G channel. This is because it processes R and G channels equally without considering their differences in noise corruption.

Clearly, a more effective color image denoising algorithm should consider the different noise strength in color channels. To this end, we introduce a weight matrix to balance the noise in the RGB channels, and present the following multi-channel WNNM (MC-WNNM) model:

(4)

We follow the method in [21] to set the weight vector on nuclear norm as , where is a small number to avoid zero numerator and is the th singular value of the estimated data matrix at the th iteration. Note that if , the proposed MC-WNNM model will be reduced to the concatenated WNNM model. With an appropriate setting of the weight matrix and a good optimization algorithm, the proposed MC-WNNM model will lead to much better color image denoising results. As shown in Figs. 1(f) and 1(h), MC-WNNM removes clearly the noise in the R channel while preserving textures effectively in the G channel.

3.2 The Setting of Weight Matrix

Let’s denote the noisy patch matrix by , where are sub-matrices of similar patches in R, G, B channels, respectively. The corresponding clean matrix is , where are similarly defined. The weight matrix can be determined under the maximum a-posterior (MAP) estimation framework:

(5)

The log-likelihood term is characterized by the statistics of noise. According to [18]

, we assume that the noise is independent among RGB channels and independently and identically distributed (i.i.d.) in each channel with Gaussian distribution and standard deviations

. There is:

(6)

For the latent data , the small weighted nuclear norm prior is imposed on it, i.e., should be sparsely distributed. We let it be:

(7)

Putting (7) and (6) into (5), we have

(8)

with

(9)

where

is the identity matrix.

Clearly, the weight matrix is diagonal and determined by the noise standard deviation in each channel. The stronger the noise in a channel, the less the contribution that channel should make to the estimation of . Our experimental results (refer to Section 4 please) on synthetic and real noisy images clearly demonstrate the advantages of MC-WNM over WNNM and other methods in color image denoising.

3.3 Model Optimization

The proposed MC-WNNM model does not have an analytical solution. In the WNNM model [21], when the weights assigned on singular values are in a non-descending order, the weighted nuclear norm proximal operator can have a global optimum with closed-form solution. Unfortunately, such a property is not valid for the MC-WNNM model because a weight matrix is assigned to the rows of data matrix . This makes the proposed model more difficult to solve than the original WNNM model.

We employ the variable splitting method [29, 30] to solve the MC-WNNM model. By introducing an augmented variable , the MC-WNNM model can be reformulated as a linear equality-constrained problem with two variables and :

(10)

Since the objective function is separable w.r.t. the two variables, the problem (10) can be solved under the alternating direction method of multipliers (ADMM) [22] framework. The augmented Lagrangian function is:

(11)

where is the augmented Lagrangian multiplier and is the penalty parameter. We initialize the matrix variables , , and

to be zero matrix and

to be a suitable value. Denote by () and the optimization variables and Lagrange multiplier at iteration (), respectively. By taking derivatives of the Lagrangian function w.r.t. and and setting the derivative function to be zero, we can alternatively update the variables as follows:
(1) Update while fixing and :

(12)

This is a standard least squares regression problem with closed-form solution:

(13)

(2) Update while fixing and :

(14)

According to the Theorem 1 in [21], given the SVD of , i.e., , where (without loss of generality, we assume that ), the global optimum of the above problem is , where and is the solution to the following convex optimization problem:

(15)

According to the Remark 1 in [21], the problem above has closed-form solution ():

(16)

where , , is a small number, and is set as by experience in [21].
(3) Update while fixing and :

(17)

(4) Update : , where .

The above alternative updating steps are repeated until the convergence condition is satisfied or the number of iterations exceeds a preset threshold. The convergence condition of the ADMM algorithm is: , , and are simultaneously satisfied, where is a small tolerance number. We summarize the updating procedures in Algorithm 1. The convergence analysis of the proposed Algorithm 1 is given in Theorem 1. Note that since the weighted nuclear norm is non-convex in general, we employ an unbounded sequence of here to make sure that Algorithm 1 converges.

Algorithm 1: Solve MC-WNNM via ADMM
Input: Matrices and , , , ;
Initialization: , ,
       T = False, ;
While (T == false) do
1. Update as
2. Update by solving the problem
  
3. Update as
4. Update ;
5. ;
if (Convergence condition is satisfied) or ()
6. T True
end if
end while
Output: Matrices and .
Theorem 1.

Assume that the weights in are in a non-descending order, the sequences , , and generated in Algorithm 1 satisfy:

(18)
(19)
(20)
Proof.

We give a sketch proof here and detailed proof of this theorem can be found in the supplementary file.

We first prove that the sequence generated by Algorithm 1 is upper bounded. Since is unbounded, i.e., , we can prove that the sequence of Lagrangian function is also upper bounded. Hence, both and are upper bounded. Then is also upper bounded. According to Eq. (17), we can prove that , and (a) is proved. Then we can prove that and hence (b) is proved. Finally, (c) can be proved by checking that where is the SVD of . ∎

3.4 The Denoising Algorithm

Given a noisy color image , suppose that we have extracted local patches and their similar patches.  noisy patch matrices can be formed to estimate the clean matrices . The patches in matrices are aggregated to form the denoised image . To obtain better denoising results, we perform the above denoising procedures for several rounds. The proposed MC-WNNM based color image denoising algorithm is summarized in Algorithm 2.

Algorithm 2: Color Image Denoising by MC-WNNM
Input: Noisy image , noise levels , ;
Initialization: , ;
for do
1. Set ;
2. Extract local patches from ;
for each patch do
3. Search non-local similar patches ;
4. Apply the MC-WNNM model (10) to and
  obtain the estimated ;
end for
5. Aggregate to form the image ;
end for
Output: Denoised image .

3.5 Complexity Analysis

In Algorithm 1 for solving the MC-WNNM model via ADMM, the cost for updating is , while the cost for updating is . The costs for updating and can be ignored. So the overall complexity is , where is the number of iterations. In Algorithm 2 for image denoising, we consider the number of patches extracted from the input noisy image and the number of itertations and ignore the cost for searching similar patches. The overall cost is .

4 Experiments

We evaluate the proposed MC-WNNM method on synthetic and real noisy color images. We compare the proposed method with state-of-the-art denoising methods, including CBM3D [14], MLP [9], WNNM [3], TNRD [11], DnCNN [12] “Noise Clinic” (NC) [25, 26], CC [19], and the commercial software Neat Image (NI) [28]. The Matlab source code of our MC-WNNM algorithm can be downloaded at http://www4.comp.polyu.edu.hk/~cslzhang/code/MCWNNM.zip.

4.1 Experimental Settings

Noise level estimation. For most of the competing denoising algorithms, the standard deviation of noise should be given as a parameter. In synthetic experiments, the noise levels () in R, G ,B channels are assumed to be known. In the case of real noisy images, the noise levels can be estimated via some noise estimation methods [31, 32]. In this paper, we employ the method [32] to estimate the noise level for each color channel.

Noise level of comparison methods. For the CBM3D method [14], a single parameter of noise level should be input. We set the noise level as

(21)

The methods of MLP [9] and TNRD [11] are originally designed for grayscale images. We retrain their models (using the released codes by the authors) at different noise levels from to with a gap of . The denoising on color images is performed by processing each channel with the model trained at the same (or nearest) noise level.

Comparison with WNNM. In order to make a full and fair comparison with the original WNNM method [21], we implement WNNM for color image denoising in three ways. 1) We apply WNNM to each color channel separately with the corresponding noise levels . We call this method “WNNM-1”. 2) We perform WNNM on the concatenated matrix formed by the patches in RGB channels, while the input noise level is computed by Eq. (21). We call this method “WNNM-2”. 3) We set the weight matrix as in the proposed MC-WNNM model, and use our developed algorithm for denoising. We call this method “WNNM-3”.

For a fair comparison, we tune the parameters of WNNM-1, WNNM-2, WNNM-3 and MC-WNNM to achieve their best denoising performance. The detailed parameters are as follows: we set the patch size as , the number of non-local similar patches as , the window size for searching similar patches as . For WNNM-3 and MC-WNNM, the updating parameter is set as . The number of iterations in Algorithm 1 is set as . The number of iterations in Algorithm 2 and the initial penalty parameter will be given in the following sub-sections.

Figure 2: The 15 cropped real noisy images used in [19].

4.2 Experiments on Synthetic Noisy Color Images

We first compare MC-WNNM with the competing denoising methods [14, 9, 11, 12, 25, 28] on the 24 color images from the Kodak PhotoCD Dataset (http://r0k.us/graphics/kodak/). The noisy images are generated by adding AWGN to each of the R, G, B channels, respectively. In the main paper, we report the results by setting . More results with other noise settings can be found in the supplementary file. For WNNM-3 and MC-WNNM, the initial penalty parameter is set as and , respectively. The number of iterations in Algorithm 2 is set as .

The PSNR results by competing methods are listed in Table 1, while the best PSNR result for each image is highlighted in bold. One can see that on all the 24 images, our method achieves the highest PSNR values among the competing methods. On average, MC-WNNM achieves 0.47dB, 0.48dB and 1.09dB improvements over WNNM-1, WNNM-2 and WNNM-3, respectively. For space limitation, we leave the visual comparisons of the synthetic noisy image denoising results in the supplementary file.

Image# CBM3D MLP TNRD DnCNN NI NC WNNM-1 WNNM-2 WNNM-3 MC-WNNM
1 25.24 25.70 25.74 20.47 23.85 24.90 26.01 25.95 25.58 26.66
2 28.27 30.12 30.21 20.47 25.90 25.87 30.08 30.11 29.80 30.20
3 28.81 31.19 31.49 20.53 26.00 28.58 31.58 31.61 31.20 32.25
4 27.95 29.88 29.86 20.47 25.82 25.67 30.13 30.16 29.84 30.49
5 25.03 26.00 26.18 20.52 24.38 25.15 26.44 26.39 25.32 26.82
6 26.24 26.84 26.90 20.66 24.65 24.74 27.39 27.30 26.88 27.98
7 27.88 30.28 30.40 20.52 25.63 27.69 30.47 30.54 29.70 30.98
8 25.05 25.59 25.83 20.57 24.02 25.30 26.71 26.75 25.26 26.90
9 28.44 30.75 30.81 20.50 25.94 27.44 30.86 30.92 30.29 31.49
10 28.27 30.38 30.57 20.52 25.87 28.42 30.65 30.68 29.95 31.26
11 26.95 28.00 28.14 20.52 25.32 24.67 28.19 28.16 27.61 28.63
12 28.76 30.87 31.05 20.60 26.01 28.37 30.97 31.06 30.58 31.48
13 23.76 23.95 23.99 20.52 23.53 22.76 24.27 24.15 23.52 24.89
14 26.02 26.97 27.11 20.51 24.94 25.68 27.20 27.15 26.55 27.57
15 28.38 30.15 30.44 20.71 26.06 28.21 30.52 30.60 30.13 30.81
16 27.75 28.82 28.87 20.52 25.69 26.66 29.27 29.21 29.02 29.96
17 27.90 29.57 29.80 20.56 25.85 28.32 29.78 29.79 29.16 30.40
18 25.77 26.40 26.41 20.53 24.74 25.70 26.63 26.56 26.01 27.22
19 27.30 28.67 28.81 20.53 25.40 26.52 29.19 29.22 28.67 29.57
20 28.96 30.40 30.76 21.44 24.95 25.90 30.79 30.83 29.97 31.07
21 26.54 27.53 27.60 20.51 25.06 26.48 27.80 27.75 27.12 28.34
22 27.05 28.17 28.27 20.51 25.36 26.60 28.21 28.16 27.81 28.64
23 29.14 32.31 32.51 20.54 26.13 23.24 31.89 31.97 31.21 32.34
24 25.75 26.41 26.53 20.59 24.55 25.73 27.10 27.03 26.18 27.59
Average 27.13 28.54 28.68 20.58 25.24 26.19 28.84 28.83 28.22 29.31
Table 1: PSNR(dB) results of different denoising methods on 24 natural color images.

4.3 Experiments on Real Noisy Color Images

We evaluate the proposed method on two real noisy color image datasets, where the images were captured under indoor or outdoor lighting conditions by different types of cameras and camera settings. For WNNM-3 and MC-WNNM, the initial penalty parameter is set as and , respectively. The number of iterations in Algorithm 2 is set as .

The first dataset is provided in [26], which includes 20 real noisy images collected under uncontrolled outdoor environment. Since there is no “ground truth” of the noisy images, the objective measures such as PSNR cannot be computed on this dataset.

The second dataset is provided in [19], which includes noisy images of 11 static scenes. The noisy images were collected under controlled indoor environment. Each scene was shot 500 times under the same camera and camera setting. The mean image of the 500 shots is roughly taken as the “ground truth”, with which the PSNR can be computed. Since the image size is very large (about ) and the 11 scenes share repetitive contents, the authors of [19] cropped 15 smaller images of size for experiments. Fig. 2 shows the contents of these images. Quantitative comparisons on the 15 cropped images will be reported.

4.3.1 Results on Dataset [26]

Since there is no “ground truth” for the real noisy images in dataset [26], we only compare the visual quality of the denoised images by the compared methods. (Note that the method CC [19] is not compared here since its code is not publically available.)

Fig. 3 shows the denoised images of “Dog” by the competing methods. It can be seen that CBM3D, MLP, TRND and WNNM-1 tend to generate some noise caused color artifacts. Besides, WNNM-2 and WNNM-3 tend to over-smooth much the image. These results demonstrate that for color image denoising, neither processing each channel separately nor processing the three channels jointly but ignoring their noise difference is an effective solution. Though NC and NI methods are specifically developed for real color image denoising, their performance is not very satisfactory. In comparison, the proposed MC-WNNM recovers much better the structures and textures (such as the eye area) than the other competing methods. More visual comparisons on this dataset can be found in the supplementary file.

(a) Noisy [26] (b) CBM3D [14] (c) MLP [9] (d) TNRD [11] (e) NI [28]
(f) NC [25, 26] (g) WNNM-1 [3] (h) WNNM-2 [3] (i) WNNM-3 [3] (j) MC-WNNM
Figure 3: Denoised images of the real noisy image “Dog” [26] by different methods. The estimated noise levels of R, G, and B channels are 16.8, 17.0, and 16.6, respectively. The images are better to be zoomed in on screen.
Camera Settings CBM3D MLP TNRD DnCNN NI NC CC WNNM-1 WNNM-2 WNNM-3 MC-WNNM
Canon 5D 39.76 39.00 39.51 37.26 35.68 36.20 38.37 37.51 39.74 39.98 41.13
ISO = 3200 36.40 36.34 36.47 34.13 34.03 34.35 35.37 33.86 35.12 36.65 37.28
36.37 36.33 36.45 34.09 32.63 33.10 34.91 31.43 33.14 34.63 36.52
Nikon D600 34.18 34.70 34.79 33.62 31.78 32.28 34.98 33.46 35.08 35.08 35.53
ISO = 3200 35.07 36.20 36.37 34.48 35.16 35.34 35.95 36.09 36.42 36.84 37.02
37.13 39.33 39.49 35.41 39.98 40.51 41.15 39.86 40.78 39.24 39.56
Nikon D800 36.81 37.95 38.11 35.79 34.84 35.09 37.99 36.35 38.28 38.61 39.26
ISO = 1600 37.76 40.23 40.52 36.08 38.42 38.65 40.36 39.99 41.24 40.81 41.43
37.51 37.94 38.17 35.48 35.79 35.85 38.30 37.15 38.04 38.96 39.55
Nikon D800 35.05 37.55 37.69 34.08 38.36 38.56 39.01 38.60 39.93 37.97 38.91
ISO = 3200 34.07 35.91 35.90 33.70 35.53 35.76 36.75 36.04 37.32 37.30 37.41
34.42 38.15 38.21 33.31 40.05 40.59 39.06 39.73 41.52 38.68 39.39
Nikon D800 31.13 32.69 32.81 29.83 34.08 34.25 34.61 33.29 35.20 34.57 34.80
ISO = 6400 31.22 32.33 32.33 30.55 32.13 32.38 33.21 31.16 33.61 33.43 33.95
30.97 32.29 32.29 30.09 31.52 31.76 33.22 31.98 33.62 34.02 33.94
Average 35.19 36.46 36.61 33.86 35.33 35.65 36.88 35.77 37.27 37.12 37.71
Time 7.8 20.4 6.7 180.3 0.9 18.2 NA 689.1 465.3 198.6 202.9
Table 2: PSNR(dB) results and averaged computational time (s) of different methods on 15 cropped real noisy images used in [19].
(a) Noisy [19]: 35.71dB (b) CBM3D [1, 14]: 37.76dB (c) TNRD [11]: 40.52dB (d) NI [28]: 38.42dB (e) NC [26, 25]: 38.65dB
(f) CC [19]: 40.36dB (g) WNNM-2: 41.24dB (h) WNNM-3: 40.81dB (i) MC-WNNM: 41.43dB (j) Mean Image [19]
Figure 4: Denoised images of a region cropped from the real noisy image “Nikon D800 ISO=1600 2” [19] by different methods. The estimated noise levels of R, G, and B channels are 1.3, 1.1, and 1.4, respectively. The images are better to be zoomed in on screen.

4.3.2 Results on Dataset [19]

As described at the beginning of Section 4.3, there is a mean image for each noisy image in dataset [19], and those mean images can be roughly taken as “ground truth” for quantitative evaluation of denoising algorithms.

The results on PSNR and averaged computational time by competing methods (including CC [19] whose results are copied from [19]) are listed in Table 2. For methods MLP [9] and TNRD [11], both of them achieve the best results when setting the noise level of the trained models at . The highest PSNR results are highlighted in bold. On average, MC-WNNM achieves 1.94dB, 0.44dB, 0.59dB improvements over the three WNNM methods, and significantly outperforms other competing method, including CC [19]. On 10 out of the 15 images, the proposed MC-WNNM achieves the highest PSNR values, while WNNM-2 achieves the highest PSNR results on 3 of 15 images. It should be noted that in the CC method [19], a specific model is trained for each camera and camera setting, while the other methods uses the same model for all cases.

Fig. 4 shows the denoised images of a scene captured by Nikon D800 ISO=1600. (The results of DnCNN and WNNM-1 are not shown here due to the limit of space.) We can see that CBM3D, NI, NC and CC will either remain noise or generate color artifacts, while TNRD, WNNM-2 and WNNM-3 over-smooth the image. In addition, due to treating each channel equally, both the denoised images (Fig. 4(g) and Fig. 4(h)) by WNNM-2 and WNNM-3 have chromatic aberration compared to the mean image (Fig. 4(j)). MC-WNNM results in much better visual quality than other methods. More visual comparisons can be found in the supplementary file.

Comparison on speed. We compare the average computational time (second) of different methods (except CC), which is shown in Table 2. All experiments are run under the Matlab environment on a machine with 3.5GHz CPU and 32GB RAM. The fastest result is highlighted in bold. One can see that Neat Image (NI) is the fastest and costs about 0.9 second, while the proposed MC-WNNM needs 202.9 seconds. Noted that CBM3D, TNRD, and NC are implemented with compiled C++ mex-function and with parallelization, while WNNM, MLP, DnCNN, and the proposed MC-WNNM are implemented purely in Matlab.

5 Conclusion

The real noisy color images have different noise statistics across the R, G, B channels due to digital camera pipelines in CCD or CMOS sensors. This makes the real color image denoising problem more challenging than grayscale image denoising. In this paper, we proposed a novel multi-channel (MC) denoising model to effectively exploit the redundancy across color channels while differentiating their different noise statistics. Specifically, we introduced a weight matrix to the data term in the RGB channel concatenated weighted nuclear norm minimization (WNNM) model, and the resulting MC-WNNM model can process adaptively the different noise in RGB channels. We solved the MC-WNNM model via an ADMM algorithm. Extensive experiments on synthetic and real datasets demonstrated that the proposed MC-WNNM method outperforms significantly the other competing denoising methods.

The proposed MC-WNNM model can be extended in at least two directions. Firstly, it is worthy to investigate new weight matrix beyond the diagonal form, such as the correlation form [33], to further improve the color image denoising performance. Secondly, the proposed MC-WNNM model can be extended for hyperspectral image analysis, which may contain hundreds of bands with complex noise statistics.

References

  • [1] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):2080–2095, 2007.
  • [2] M. Elad and M. Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12):3736–3745, 2006.
  • [3] S. Gu, L. Zhang, W. Zuo, and X. Feng. Weighted nuclear norm minimization with application to image denoising.

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    , pages 2862–2869, 2014.
  • [4] A. Buades, B. Coll, and J. M. Morel. A non-local algorithm for image denoising. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 60–65, 2005.
  • [5] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image restoration. IEEE International Conference on Computer Vision (ICCV), pages 2272–2279, 2009.
  • [6] D. Zoran and Y. Weiss. From learning models of natural image patches to whole image restoration. IEEE International Conference on Computer Vision (ICCV), pages 479–486, 2011.
  • [7] W. Dong, L. Zhang, G. Shi, and X. Li. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, 22(4):1620–1630, 2013.
  • [8] J. Xu, L. Zhang, W. Zuo, D. Zhang, and X. Feng. Patch group based nonlocal self-similarity prior learning for image denoising. IEEE International Conference on Computer Vision (ICCV), pages 244–252, 2015.
  • [9] H. C. Burger, C. J. Schuler, and S. Harmeling.

    Image denoising: Can plain neural networks compete with BM3D?

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2392–2399, 2012.
  • [10] U. Schmidt and S. Roth. Shrinkage fields for effective image restoration. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2774–2781, June 2014.
  • [11] Y. Chen, W. Yu, and T. Pock. On learning optimized reaction diffusion processes for effective image restoration. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5261–5269, 2015.
  • [12] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 2017.
  • [13] S. Roth and M. J. Black. Fields of experts. International Journal of Computer Vision, 82(2):205–229, 2009.
  • [14] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space. IEEE International Conference on Image Processing (ICIP), pages 313–316, 2007.
  • [15] J. Mairal, M. Elad, and G. Sapiro. Sparse representation for color image restoration. IEEE Transactions on Image Processing, 17(1):53–69, 2008.
  • [16] F. Zhu, G. Chen, and P.-A. Heng. From noise modeling to blind image denoising. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  • [17] C. Liu, R. Szeliski, S. Bing Kang, C. L. Zitnick, and W. T. Freeman. Automatic estimation and removal of noise from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):299–314, 2008.
  • [18] B. Leung, G. Jeon, and E. Dubois. Least-squares luma-chroma demultiplexing algorithm for bayer demosaicking. IEEE Transactions on Image Processing, 20(7):1885–1894, 2011.
  • [19] S. Nam, Y. Hwang, Y. Matsushita, and S. J. Kim. A holistic approach to cross-channel image noise modeling and its application to image denoising. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1683–1691, 2016.
  • [20] H. C. Karaimer and M. S. Brown. A software platform for manipulating the camera imaging pipeline. European Conference on Computer Vision (ECCV), October 2016.
  • [21] S. Gu, Q. Xie, D. Meng, W. Zuo, X. Feng, and L. Zhang. Weighted nuclear norm minimization and its applications to low level vision. International Journal of Computer Vision, pages 1–26, 2016.
  • [22] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3(1):1–122, January 2011.
  • [23] J. Cai, E. J. Candès, and Z. Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956–1982, 2010.
  • [24] C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218, 1936.
  • [25] M. Lebrun, M. Colom, and J.-M. Morel. Multiscale image blind denoising. IEEE Transactions on Image Processing, 24(10):3149–3161, 2015.
  • [26] M. Lebrun, M. Colom, and J. M. Morel. The noise clinic: a blind image denoising algorithm. http://www.ipol.im/pub/art/2015/125/. Accessed 01 28, 2015.
  • [27] C. Kervrann, J. Boulanger, and P. Coupé. Bayesian non-local means filter, image redundancy and adaptive dictionaries for noise removal. International Conference on Scale Space and Variational Methods in Computer Vision, pages 520–532, 2007.
  • [28] Neatlab ABSoft. Neat Image. https://ni.neatvideo.com/home.
  • [29] R. Courant. Variational methods for the solution of problems of equilibrium and vibrations. Bull. Amer. Math. Soc., 49(1):1–23, 1943.
  • [30] J. Eckstein and D. P. Bertsekas. On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming, 55(1):293–318, 1992.
  • [31] X. Liu, M. Tanaka, and M. Okutomi. Single-image noise level estimation for blind denoising. IEEE Transactions on Image Processing, 22(12):5226–5237, 2013.
  • [32] G. Chen, F. Zhu, and A. H. Pheng. An efficient statistical method for image noise level estimation. IEEE International Conference on Computer Vision (ICCV), December 2015.
  • [33] N. J. Higham. Computing the nearest correlation matrix: a problem from finance. IMA Journal of Numerical Analysis, 22(3):329, 2002.