Image noise commonly exists in the image acquisition, quantization, transmission and many other processing stages. A digital image contaminated by noises leads to visible loss in image quality and can impact many advanced image processing and computer vision tasks like tracking, recognition, classification, etc. The importance of image denoising is therefore self-explanatory.
Conventional image denoising methods more or less related to filters [NLM0], like moving average filters, Wiener filters, and wavelet filter banks. These filter-based image denoising techniques are commonly of low complexity and can be easily implemented in hardware. However, their performance is not always adequate. With the help of the increased computational capacity powered by digital processors, many advanced denoising techniques are now feasible. Among these techniques, the Non-Local Means (NLM) method [NLM0, NLM1] has attracted significant attention in recent years. The NLM denoises an image pixel as the weighted sum of its noisy neighbors, where each weight reflects the similarity between the local patch centered at the noisy pixel to be denoised and the patch centered at the neighbor pixel. In this way, NLM adapts the denoising process for each pixel and thus outperforms conventional techniques [NLM0].
Many improvements on the original NLM have been proposed in recent years. These discussions mainly focus on three questions: 1) how to pick NLM parameters heuristically or automatically[SURE, CPW]; 2) how to accelerate the NLM or save computations without loss of denoising performance [IINLM, 5710056, 6324435]; and 3) how to adjust the NLM framework to achieve better performance [5356177, 6036056, 6315713, 5548921, multipatch]. In [SURE], the closed-form of Stein’s risk estimator is derived for the NLM and allows prediction of the risk of a denoised image without knowing the clean image. In [IINLM], the integral image scheme is adapted to the NLM and greatly reduces the computational costs. In [multipatch], multiple-patches are discussed to eliminate artifacts in the NLM.
The importance of the center pixel weight (CPW) has been noticed for a long time [NLM0, CPW], and various heuristic weights are designed and used [CPW]. However, the methods proposed are non-ideal as they do not consider all aspects of CPW problem (see details in Section III-A). Thus, new CPWs need to be designed. In this letter, we discuss the CPW problem in NLM and propose new solutions based on the James-Stein estimator [JamesStein]. The rest of the letter is organized as: Sec. II reviews the NLM and related works on CPWs; Sec. III discusses the new formulation of the CPW problem and the new James-Stein type CPWs; Sec. IV shows the experimental comparisons; and we conclude the letter in Sec. V.
Ii A Brief Review
Ii-a The Classic Non-Local Mean Algorithm
Let be a two dimensional clean image defined on the spatial domain , with the th pixel located at the intersection of the th row and the th column. Given one of ’s noisy observation , whose pixels are contaminated by an i.i.d. zero-mean Gaussian noise with a variance of , namely
The classic NLM method [NLM0, NLM1] estimates the clean pixel by using all pixels within a prescribed search region , typically a square or a rectangular region. Specifically, the estimated is the weighted sum of ’s noisy neighbors as
where each weight is computed by quantifying the similarity between two local patches around noisy pixels and as shown in Eq. (3), where is a Gaussian weakly smooth kernel [NLM0], denotes the local patch, typically a square centered at the pixel (for example a denotes a 33 patch). The parameter can be considered as a temperature parameter controlling the behavior of the weight function, namely as , all weights approach to 1; while as , all weights approach to zero.
Ii-B Existing Central Pixel Weights
The CPW in the classic NLM is unitary, because (3) implies for all . However, this unitary CPW is reported not to perform well in many cases [CPW]. Indeed, in the case that is really noisy and has a rare patch (implying that for all non-center weights are very small compared to 1), a unitary CPW means the contribution of the noisy pixel dominates in the denoised pixel , implying a poor performance.
In addition to this CPW, several other CPWs have been proposed and used in the NLM community to enhance performance. These include the zero CPW (Eq. (5)), the Stein CPW (Eq. (6)), the max CPW (Eq. (7)) and the heuristic CPW (Eq. (8)). In the rest of the letter, we use to denote these existing CPWs.
These CPWs can be classified into two groups: global CPWs (Eqs. (4),(5) and (6)) and local CPWs (Eqs. (7) and (8)). The global CPWs use a constant CPW for all pixels, while the local CPWs use different CPWs for pixels. In the next section, we will show that these CPWs fail to take all variables into full consideration and therefore oversimplify the CPW problem.
Iii Shrinkage based Center Pixel Weights
Iii-a The CPW problem in the form of shrinkage estimator
To fully reveal the CPW problem, we separate the contributions of the non-center and of the center pixels in the NLM denoised pixel (Eq. (2))
where is the summation of all non-center weights
and is the denoised pixel by using all non-center weights.
As one can see, a CPW is contained in the coefficients of both and and thus influences the final denoised pixel directly. If we are given an optimal and solve for , then it is clear that the optimal should be a function of , and . In other words, a CPW that does not consider all these variables is incomplete. It is noticeable that the global CPWs and neglects all three, while the local CPWs and neglect .
Let be the fraction () of the contribution of the center pixel in , namely
is then a normalized version of . Consequently, the NLM-CPW problem in (9) can be rewritten as
and is a so-called shrinkage estimator, which improves an existing estimator by using the raw data. In the context of the NLM, the existing estimator is and the raw data is the noisy pixel . The effect of the CPW is to tune the final denoised pixel in somewhere between and , or equivalently to shrink towards to .
Iii-B The James-Stein Center Pixel Weight
One important result in shrinkage estimators is the James-Stein estimator [JamesStein]
. It states that for an unknown parameter vectorand its observations of with the relation,
there exists a James-Stein estimator that shrinks towards an arbitrary vector in the form that
with the coefficient of form (17) [JamesStein].
This James-Stein estimator is a classic solution to minimize the risk of estimation in terms of the mean square error [Stein], where denotes the -norm.
In the context of NLM-CPW problem, the James-Stein based CPW (JSCPW) has the weight of form (17),
where is the number of pixels in the image, and the corresponding new estimator is
Iii-C Local Adapted James-Stein Center Pixel Weights
Although the proposed JSCPW considers all , and , it is still a global CPW, which gives the weight to each pixel unbiasedly. However, we know the denoising process is always biased rather than unbiased for each pixel. For example, pixels in a homogeneous region are commonly better denoised than edge pixels. Therefore, ideally we want a locally adapted CPW for each pixel as in the form of (20). One natural idea is to replace in (20) with , but it does not lead to a stable solution, because is commonly noisy. Alternatively, we view each image block as a small image and thus the JSCPW (17) computed for a local block gives a local CPW adapted to each pixel.
Without loss of generality, say and , so there are two corresponding local image blocks around the th pixel in and , respectively. Given a prescribed local block region , then the local James-Stein center pixel weight (LJSCPW) can be computed as
In this way, we construct a local CPW for each pixel, and thus the denoised pixel by using LJSCPW can be written as
Intuitively, this LJSCPW helps eliminate the influence of remote image pixels and tunes the optimization locally.
The computational cost of the LJSCPW can be only 5 operations/pixel more than the existing CPWs. Specifically, we construct the integral image [IINLM] with 2 operations/pixel for the pixel-wise mean square error between and . Each pixel is the summation of form (21).
This integral image then allows computation of for an arbitrary rectangular with 3 operations/pixel.
In a real case, one may see which conflicts with our assumption that . This case happens when is a slightly denoised version of . So it is reasonable to use rather than , implying . Therefore, we use the positive part of in Eq. (22), where .
Thus ranges from . Because would indicate the raw data is used, i.e. the pixel is not denoised, it may prove useful in some applications to limit to a user-defined value less than unity. In this letter, however, we allow the shrinkage operator to operate over the full range.
Iv Simulation Results
All following simulations are done under the MATLAB r2010a environment. We compare the performance of CPWs i.e. with the proposed James-Stein type CPWs and under the classic NLM framework (only CPW part is changed). In particular, we set searching region to 3131 square and use for in the experiment. All test images 111are available under the page http://www.cs.tut.fi/~foi/GCF-BM3D/BM3D_images.zip as the date of 10/09/2012. are gray-scale images with additive Gaussian noises of . For each test image, we denoise it by using the 200 temperature parameters ranging from 1% to 200% of
. The denoising performance for each CPW scheme is then evaluated by computing its mean and standard deviation in terms of the peak signal noise ratio (PSNR) (dB) defined in Eq. (23).
These results are summarized in Table I. As one can see, compared to the PSNR performance of the zero CPW, the proposed (LJSCPW) and (JSCPW) are the only two in all CPWs that always improve the denoising performance in terms of a higher mean PSNR with a smaller variance, regardless of patch sizes, test images and noise levels. This implies that James-Stein type CPWs are more efficient than other CPWs.
|55 Patch||77 Patch|
We found two general performance trends: 1) as , all CPWs except and converges together; and 2) as increases, the impact of CPWs on denoising results decreases. Fig. 1 illustrates these trends in NLM performance for the cameraman image. The first trend occurs because as , all non-center weights go to 1. The CPWs , , and are then all 1s and thus converge. The second trend is because the noisy image become less effective in providing a good guide in shrinkage when the noise level is high. Thus the performance differences of various CPWs gets smaller as the noise level increases. However, it is noticeable that the proposed locally adapted outperforms other CPWs regardless of chosen temperature parameter , and the NLM performance using the proposed CPWs decays much slower than using other CPWs. Similar behaviors are also observed for other patches sizes and test images. Finally, the NLM denoising results for is given in Fig. 2. It is noticeable that the proposed helps keep weak edges (see the two buildings in the background).
|(b) Noisy image (= 20)||(c) NLM using|
|(e) NLM using||(f) NLM using|
|(h) NLM using||(i) NLM using|
In this letter, we reviewed the CPW problem in NLM and proposed two new solutions JSCPW and LJSCPW based on the James-Stein estimator. We showed that the NLM-CPW problem can be viewed as the well studied statistical shrinkage estimator problem. This novel formulation opens a new door to the CPW problem and allows us to use the James-Stein shrinkage estimator for the NLM-CPW problem directly, that is the global JSCPW. To further enhance denoising performance, we propose a locally adapted James-Stein type CPW for each pixel. In this way, the denoising performance is tuned with respect to each local pixel rather than an entire image. Our experimental results show that the proposed James-Stein type CPWs help the NLM to achieve better overall performance in terms of a higher average PSNR with a more robust performance in terms of a smaller PSNR variance. By using these new CPWs, the NLM algorithm is then less sensitive to the temperature parameter and is better able to retain weak edges.