We consider the standard problem of denoising grayscale images that are corrupted with additive white Gaussian noise. The setting here is that we are given a noisy image
where is some finite rectangular domain of , is the unknown clean image, and are i.i.d. where
is the standard deviation of the Gaussian noise. The focus of this work is the Non-Local Means (NLM) algorithm that was introduced by Buades, Coll, and Morel in[1, 2]. NLM denoises each pixel by taking a weighted average of the neighboring pixels. A key innovation in NLM is that the weights are computed using patches (blocks of neighboring pixels) instead of pixels. More specifically, the denoised image is computed using the formula
where denotes the neighborhood around pixel . The original proposal in  was to set to be the whole image. In practice, one restricts to a sufficiently large window of size centered at , that is, . The weights are set to be
where is a smoothing parameter, and patch is a square neighborhood of the origin of size .
Since its introduction, several variations of NLM have been proposed which can further improve the denoising. For example, various weighting schemes for the center pixel have been proposed in [3, 4, 5]. A weighting scheme based on quadratic programming was proposed in , which exploits the overlapping information between adjacent patches. Further, optimization framework based approaches have been proposed in [7, 8] to improve the performance of NLM. In , the weighting was performed based on the structural similarity indices between patches. In a different direction, it was demonstrated in [10, 11, 12, 13] that the performance (and the speed) of NLM can be improved by computing the weights from low-dimensional projections of the patches.
The primary application of NLM is noise reduction [14, 15]. However, since the inception of NLM, more sophisticated algorithms with superior denoising performances have been proposed [16, 17, 18, 19, 20, 21, 22, 23, 24]. While NLM and its variants are no longer the state-of-the-art in image denoising, they nevertheless continue to be of interest due to their simplicity, decent denoising performance, and the availability of fast implementations. We also note that several researchers have used NLM (either directly or via post-processing) in image restoration applications such as deblurring [25, 26], inpainting , demosaicking 
, and super-resolution[29, 30, 31].
The investigation in this paper was particularly motivated by the work on shape-adaptive patches  and robust regression [33, 34]. In either case, it was demonstrated that the denoising quality of NLM can be improved by adapting the averaging neighborhood at every pixel. While this was done explicitly in  in terms of the patch shape, the adaptation was achieved implicitly in [33, 34] via an optimization. In this paper, we propose to perform the adaptation by explicitly pruning those neighboring pixels whose weights are below a certain global threshold . We use Stein’s unbiased estimator of the mean-squared error to optimally tune . In other words, we optimize the denoising quality by performing a simple one-dimensional optimization for the whole image. As a result, the overall implementation is much faster than those implementations in [32, 33, 34, 35], where one is required to perform an adaptation at every pixel.
The rest of the paper is organized as follows. In Section II, we motivate the main idea using a simple experiment, and then present the proposed denoising algorithm and the implementation details. The denoising results are reported in Section III, where we also compare the proposed method with NLM and its variants.
Ii Pruned Non-Local Means
We now perform some numerical experiments to reflect the motivation that pruning can improve the denoising performance. We consider a scenario where the pixel of interest (POI) is situated closed to a edge, as shown in the example in Figure 1(a). We compute the NLM weights around the POI (marked with a solid black circle) using (3). We expect the weights of neighboring pixels that are on the same side of the edge as the POI to be generally higher than the weights of pixels that are on the other side of the edge. As shown in Figure 1(b) this is indeed the case if the weights are computed from the (unknown) clean edge. Unfortunately, this is not the case if the weights are computed from the noisy edge. As seen in Figure 1(b), the weights of some of the pixels on the left of the edge (same side of the edge as the POI) are smaller than the weights of some of the pixels on the right. This leads to a poor averaging in (2) – pixels from both sides of the edge are mixed. For example, in Figure 1(a), the intensities at the POI before and after adding noise are and . Using NLM, the denoised output is (as a reference, the NLM output using the weights computed from the clean patches is 73.09). On the other hand, notice that if we sort the noisy weights in Figure 1(b), then the majority of the pixels on the left of the edge would have the largest weights. Indeed, if we use the top neighbors (neighbors with largest weights) for the averaging, then the output comes down to (which is even better than that obtained using the clean weights). This suggests a means of improving the quality of denoising, namely, by restricting the averaging in (2) to those neighboring pixels that are assigned top weights.
Based on the above observation, we propose to perform pruning as follows. We fix a threshold , and consider the step function given by
The process of nullifying the weights that are smaller than can be achieved by modifying (1) as:
To avoid introducing unnecessary symbols, we continue to use the same notation for the denoised image as in (2). Notice that the weights defined in (3) take values in ; therefore, the variable in (4). When , none of the weights are pruned, and when is close to unity, a large fraction of the weights are pruned. The optimal setting would be somewhere in between. We can ideally tune by minimizing the mean-squared-error (MSE) given by
where is the size of the image, i.e., the number of pixels in the image. However, since we do not have access to the clean image , we cannot compute the MSE in practice.
This is precisely where Stein’s Unbiased Risk Estimator (SURE) proves to be useful. This is given by
where we recall that is the standard deviation of the additive Gaussian noise in (1).
SURE has the remarkable property that its expected value equals that of the MSE for the noise model in (1) . This makes it an useful surrogate for the MSE – it can be computed without the knowledge of the clean image. The difficulty in applying SURE directly to the transformation in (5) is that (4) is not differentiable and therefore we cannot compute the partial derivatives in (7). This is, however, not a fundamental problem and we can approximate (4) by a differentiable function. For example, we propose to use the following sigmoid function:
In Figure 2, we plot (8) for different values of and compare it with (4) for the same . Ideally, we should use a large to obtain a good approximation. However, as will be evident shortly, the partial derivatives in (7) explode as increases. Following exhaustive simulations, we decided to set . The approximation is reasonable in this case and the derivatives can be computed stably. In other words, we consider the following formula
where . We will refer to (9) as Pruned Non-Local Means (PNLM). We note that the SURE for (2) was originally derived in , and was shown to be close to the MSE. Similarly, we can derive the SURE for (9). In particular, the partial derivative of (9) is given by (cf. Appendix):
The expression of is given in (17) for completeness. A comparison of the MSE and SURE is provided in Figure 3 when . We notice that the gap between MSE and SURE decreases with the increase in image size. Importantly, we note that the minima of MSE and SURE are attained at almost the same .
Iii Experimental Results
The denoising performance of PNLM clearly depends on the choice of . In particular, the optimal (minimizer of MSE) depends on the noise level and the image characteristics. In order to obtain a noise-based rule for setting , we took a large collection of natural images [41, 42] and computed the optimal at various noise levels using exhaustive search. The scatter plot of and the corresponding for some of the images is shown in Figure 4. Also shown is the cubic polynomial that we obtained using least-square fitting:
for other parameter settings. One option is to maintain a lookup table of the optimal rules for selected parameter settings, and then interpolate between them. Instead, we propose to use (12) to initialize a one-dimensional search, irrespective of the parameter setting. Starting from a coarse estimate of , we then use golden section search  to refine the estimate. The complete process is summarized in Algorithm 1. Note that, compared to NLM, we have to perform an additional search in PNLM. Each step of this search requires us to compute (7) and (9) twice. However, the weights in (7) and (9) are computed using the patch distances used in NLM. Therefore, we have to compute the patch distances just once, which happens to be the computation-intensive part of NLM. The iterations in Algorithm 1 are terminated when the change in between successive iterations is within . The whole process is quite fast once the patch distances have been computed.
The denoising performance of PNLM and NLM are compared for different choices of search window and patch size in Table I. We see that PNLM achieves better denoising irrespective of the choice of parameters. We note that the optimal selection of NLM parameters is an altogether different line of work [3, 12]. In this paper, the focus is on post-processing of weights to enhance the robustness of NLM. The idea of pruning remain relevant irrespective of the choice of NLM parameters. For the present discussion, we choose the NLM parameters to be and , as was originally suggested in . As far as the additional parameters and in PNLM are concerned, we set to ensure a reasonable approximation of the step function , and the crucial parameter is set using the process described above.
|7||28.99 / 80.47||32.22 / 84.47|
|10||28.31 / 79.98||31.97 / 84.36|
|12||28.03 / 79.86||31.79 / 84.25|
|15||27.60 / 79.53||31.52 / 84.00|
|20||26.97 / 78.97||31.16 / 83.67|
|7||30.45 / 82.42||32.26 / 84.77|
|10||29.78 / 81.84||32.20 / 84.97|
|12||29.45 / 81.59||32.19 / 85.06|
|15||28.98 / 81.08||32.14 / 85.09|
|20||28.33 / 80.35||32.05 / 85.03|
|7||31.61 / 83.93||31.59 / 84.27|
|10||31.21 / 83.60||31.73 / 84.53|
|12||30.94 / 83.38||31.77 / 84.64|
|15||30.52 / 82.92||31.78 / 84.69|
|20||29.86 / 82.16||31.75 / 84.61|
Iii-B Runtime and Denoising Results
We now present simulation results concerning the run time of the proposed algorithm. In Table II, we compare the computation costs of NLM and PNLM on various image sizes. We see that the computational overhead for PNLM is within of that of NLM. In this context, we note that one can use any of the existing fast algorithms for NLM to speedup the PNLM computation [39, 40]. By using an optimized C implementation of any of these fast algorithms, one can bring down the run-time to within a second for a image.
|NLM||7.7 s||30 s||122 s||490 s|
|PNLM||9.7 s||39 s||157 s||630 s|
We now report some representative denoising results. We compared PNLM with NLM and the following variants of NLM: NLEM , PCA-NLM , Zero-CPW , One-CPW , and Max-CPW . For all experiments, we used the following parameter settings: , and . In Table III and IV, we report the PSNR and SSIM  obtained using various denoising algorithms on the Boat and Hill images at different noise values. We use the standard definition
where MSE is given by (6).
In Table V, we compare the PSNR and SSIM of various denoising algorithms on different natural images, with the noise fixed at . From these results, we see that PNLM consistently outperforms NLM in terms of PSNR and SSIM at all noise levels. For a visual comparison, the denoised images obtained using NLM and PNLM are provided in Figure 5. Notice that the output of PNLM is visibly sharper than that obtained using NLM.
For completeness, we have also compared PNLM with the state-of-the-art denoising methods KSVD  and BM3D . A couple of representative results are provided in Figures 6 and 7. Notice that the PNLM has successfully reduced the gap in PSNR/SSIM between NLM and K-SVD. This is interesting given that K-SVD is much more complex compared to PNLM. However, there is still a significant margin in the denoising performance with respect to BM3D. In this regard, we wish to make a couple of remarks. First, notice that the denoising using PNLM appears to be somewhat better in homogeneous regions compared to both K-SVD and BM3D. In particular, we see “ripples” and “blocking” artifacts in the homogeneous regions for the K-SVD and BM3D outputs. In fact, the overall visual quality of the PNLM output appears to be better than the K-SVD output for both Lena and Barbara, though this fact is not quite captured by the quality metrics. The other remark is that the BM3D algorithm uses block-matching and aggregation in its processing pipeline. Therefore, the present idea of pruning can in principle be incorporated into BM3D. We wish to investigate this in future work.
We demonstrated that the denoising capability of NLM can be significantly improved using pruning, where the pruning is controlled using a single global threshold. The added cost of optimizing the threshold was marginal in comparison to the run-time of NLM. Moreover, we provided several denoising results to demonstrate that pruned NLM outperforms several variants of NLM in terms of PSNR and SSIM. The surprising observation is that, by optimizing a single parameter , we could obtain denoising results that are consistently better than that obtained using sophisticated optimization in [33, 34]
. It would be interesting to see if the present ideas can be extended to the recently-proposed Vector Nonlocal Euclidean Median.
The authors wish to thank the reviewers for their useful comments and suggestions. This work was supported by a Startup Grant from the Indian Institute of Science and EMR Grant SERB/F/6047/2016-2017 from the Department of Science and Technology, Government of India.
where is given by (11). Then, by the quotient rule for derivatives and after some simplification, we obtain
where is given by (11
). Applying chain rule, we have
After performing some reductions, we arrive at (II):
-  Buades, A., Coll, B., and Morel, J. M.: ‘A non-local algorithm for image denoising,’ Jun 2005, vol. 2, pp. 60-65.
-  Buades, A., Coll, B., and Morel, J. M.: ‘A review of image denoising algorithms, with a new one,’ Multiscale Modeling and Simulation, vol. 4, pp. 490-530, 2005.
-  Salmon, J.: ‘On two parameters for denoising with non-local means,’ IEEE Signal Processing Letters, 2010, 17, (3), pp. 269-272.
Duval, V., Aujol, J., and Gousseau, Y.: ‘A bias-variance approach for the nonlocal means,’SIAM Journal on Imaging Sciences, 2011, 4, (2), pp. 760-788.
-  Wu, Y., Tracey, B., Natarajan, P., and Noonan, J. P.: ‘James-Stein type center pixel weights for non-local means image denoising,’ IEEE Signal Processing Letters, 2013, 20, (4), pp. 411-414.
-  Feng, J., Li Song, et al.: ‘An optimized pixel-wise weighting approach for patch-based image denoising,’ IEEE Signal Processing Letters, 2015, 22, (1), 115-119.
-  Zhang, X., Feng, X., and Wang, W.: ‘Two-direction nonlocal model for image denoising,’ IEEE Transactions on Image Processing, 2013, 22, (1), pp. 408-412.
-  Liu, G., Zhong, H., and Jiao, L.: ‘Comparing noisy patches for image denoising: a double noise similarity model,’ IEEE Transactions on Image Processing, 2015, 24, (3), pp. 862-872.
-  Rehman A., and Wang, Z.: ‘SSIM-based non-local means image denoising,’ Proc. IEEE International Conference on Image Processing, 2011, pp. 217-220.
-  Orchard, J., Ebrahimi, M., and Wong, A.: ‘Efficient non-local means denoising using the SVD,’ Proc. IEEE International Conference on Image Processing, 2008, pp. 1732-1735.
-  Tasdizen, T.: ‘Principal neighborhood dictionaries for non-local means image denoising,’ IEEE Transactions on Image Processing, 2009, 18, 12, pp. 2649-2660.
-  Van De Ville, D., and Kocher, M.: ‘Non-local means with dimensionality reduction and SURE-based parameter selection,’ IEEE Transactions on Image Processing, 2011, 20, (9), pp. 2683-2690.
Zuo, Chenglin, et al.: ‘Image denoising using quadtree-based nonlocal means with locally adaptive principal component analysis,’IEEE Signal Processing Letters, 2016, 23, (4), pp. 434-438.
-  Tracey, B. H., and Miller, E. L.: ‘Nonlocal means denoising of ECG signals,’ IEEE Transactions on Biomedical Engineering, 2012, 59, (9), pp. 2383-2386.
-  Yadav, S. K., Sinha, R., and Bora, P. K.: ‘Electrocardiogram signal denoising using non-local wavelet transform domain filtering,’ IET Signal Processing, 2015, 9, (1), pp. 88-96.
-  Elad, M., and Aharon, M.: ‘Image denoising via sparse and redundant representations over learned dictionaries,’ IEEE Transactions on Image Processing, 2006, 15, (12), pp. 3736-3745.
-  Dabov, K., et al.: ‘Image denoising by sparse 3D transform-domain collaborative filtering,’ IEEE Transactions on Image Processing, 2007, 16, (8), pp. 2080-2095.
-  Hossein, T., and Milanfar, P.: ‘Global image denoising,’ IEEE Transactions on Image Processing, 2014, 23, (2), pp. 755-768.
-  Duan, J., et al.: ‘Second order Mumford-Shah model for image denoising,’ Proc. IEEE International Conference on Image Processing, 2015.
-  Yue, H., et al.: ‘Image denoising by exploring external and internal correlations,’ IEEE Transactions on Image Processing, 2015, 24, (6), pp. 1967-1982.
-  Romano, Y., and Elad, M.: ‘Boosting of image denoising algorithms,’ SIAM Journal on Imaging Sciences, 2015, 8, (2), pp. 1187-1219.
-  Dinh, K. Q., Canh, T. N., and Byeungwoo, J.: ‘Color Image Denoising via Cross-Channel Texture Transferring,’ IEEE Transactions on Image Processing,, 2016, 23, (8), pp. 1071-1075.
-  Chen, Yang, et al.: ‘Improving low-dose abdominal CT images by weighted intensity averaging over large-scale neighborhoods,’ European Journal of Radiology, 2011, 80, (2), e42-e49.
-  Chen, Yang, et al.: ‘Thoracic low-dose CT image processing using an artifact suppressed large-scale nonlocal means,’ Physics in Medicine and Biology, 2012, 57, (9), pp. 2667-2688.
-  Chen, F., Xiaojun H., and Chen, W.: ‘Texture-preserving image deblurring,’ IEEE Signal Processing Letters, 2010, 17, (12), pp. 1018-1021.
-  Jung, M., et al.: ‘Nonlocal Mumford-Shah regularizers for color image restoration,’ IEEE Transactions on Image Processing, 2011, 20, (6), pp. 1583-1598.
-  Wong, A., and Orchard, J.: ‘A nonlocal-means approach to exemplar-based inpainting,’ Proc. IEEE International Conference on Image Processing, 2008.
-  Buades, A., et al.: ‘Self-similarity driven color demosaicking,’ IEEE Transactions on Image Processing, 2009, 18, (6), pp. 1192-1202.
-  Protter, M., et al.: ‘Generalizing the nonlocal-means to super-resolution reconstruction,’ IEEE Transactions on Image Processing, 2009, 18, (1), pp. 36-51.
-  Zhang, K., et al.: ‘Single image super-resolution with non-local means and steering kernel regression,’ IEEE Transactions on Image Processing, 2012, 21, (11), pp. 4544-4556.
-  Zhang, K., et al.: ‘Learning multiple linear mappings for efficient single image super-resolution,’ IEEE Transactions on Image Processing, 2015, 24, (3), pp. 846-861.
-  Deledalle, C.-A., Duval, V., and Salmon, J.: ‘Non-local methods with shape-adaptive patches (NLM-SAP),’ Journal of Mathematical Imaging Vision, 2012, 43, (2), pp. 103-120.
-  Chaudhury, K. N., and Singer, A.: ‘Non-local Euclidean medians,’ IEEE Signal Processing Letters, 2012, 19, (11), pp. 745-748.
-  Sun, Z., and Chen, S.: ‘Analysis of non-local Euclidean medians and its improvement,’ IEEE Signal Processing Letters, 2013, 20, (4), pp. 303-306.
-  Chaudhury, K. N., and Singer, A.: ‘Non-local patch regression: Robust image denoising in patch space,’ Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2013, pp. 1345 - 1349.
Stein, C.: ‘Estimation of the mean of a multivariate normal distribution,’The Annals of Statistics, 1981, 9, pp. 1135-1151.
-  Van De Ville, D., and Kocher, M.: ‘Sure-based non-local means,’ IEEE Signal Processing Letters, 2009, 16, (11), pp. 973-976.
-  Kiefer, J.: ‘Sequential minimax search for a maximum,’ Proc. American Mathematical Society, 1953, 4, (3), pp. 502-506.
-  Ghosh, S., and Chaudhury, K. N.: ‘Fast Separable Non-Local Means,’ SPIE Journal of Electronic Imaging, 2016, 25, (2), 023026.
-  Darbon, J., Cunha, A., et al.: ‘Fast nonlocal filtering applied to electron cryomicroscopy,’ Proc. IEEE International Symposium on Biomedical Imaging, 2008, pp. 1331-1334.
-  ‘BM3D Image Database’, http://www.cs.tut.fi/foi/GCF-BM3D.
-  ‘Computer Vision Test Images’, http://www.cs.cmu.edu/cil/v-images.html.
-  Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P.: ‘Image quality assessment: From error visibility to structural similarity,’ IEEE Transactions on Image Processing, 2004, 13, (4), pp. 600-612.
-  Lin C.-Y., Minasian A., Qi X.-J., and Wu H.-T.: ‘Vector nonlocal Euclidean median: principal bundle captures the nature of patch space,’ arXiv:1611.05073, 2016.