Low rank prior and l0 norm to remove impulse noise in images

1 Introduction

Image restoration is a fundamental problem in image processing. We focus on here random-valued impulse noise, often added during image acquisition and transmission. The task is to recover original image from the noisy one by removing noise.

To recover images, one should assume images have some essential properties. Traditional prior assumptions are smoothnesshunt1973application , piecewise constantrudin_nonlinear_1992 , etc. Among many priors developed later, two important priors are patch similaritybuades_review_2006 , and low rank property gu2014weighted ; hu2018note . Patch similarity assumes that there exist many similar patches in images. Low rank property means the matrix composed of similar patches has low rank. Based on patch similarity and a statistics ROADgarnett2005universal for detecting impulse noise, Hu et alhu2016removing propose patch-based weighted filter (PWMF) for removing impulse noise and mixed noise. Since the optimization of the rank is generally a NP-hard problem, many researches use the nuclear norm or the weighted nuclear norm to approximate the rank, such as WNNMgu2014weighted for removing Gaussian noise. But in some cases, the rank optimization has a close-form solution: Hu et alhu2018note propose an exact low rank model for removing Gaussian noise. In jin2018sparse , the authors use the sparse and low rank decomposition of a Hankel structured matrix for impulse noise removal, where $l_{1}$ norm is used for the sparseness, and nuclear norm for approximating rank. In adam2021combined , the authors propose a variational model that uses the nonconvex $l_{p}$ -norm, $0 < p < 1$

for both the data fidelity and a regularization term, combined with a second-order total variation regularization term and an overlapping group sparse regularizer. Recently deep learning is also applied for removing impulse noise, and has reached very good performance

jin2019learning ; wang2020variational .

However, to remove impulse noise, the exact low rank properties are not exploited in the literature as far as we know. In this paper, we will extend the exact low rank modelhu2018note to remove impulse noise. Moreover, according to our computation, $l_{0}$ norm fidelity corresponds to the maximum likelihood estimate under impulse noise, noting that $l_{2}$ norm corresponds to Gaussian noise. Thus we optimize a combination of exact rank and $l_{0}$ norm fidelity, which is solved formally by the alternating direction method of multipliers (ADMM) goldstein2009split . We consider the method as a Plug-and-Play ADMMvenkatakrishnan2013plug , and do not discuss theoretical convergence properties. Since the model is not convex, the results depend on the initial image. We choose the PWMF results as initial images, which can also simplify the model. Experiments show that this algorithm produces very good images, especially for low or medium contrast images.

2 Patch-based low rank method

In hu2018note , Hu et al propose a patch-based low rank method (PLR) for removing Gaussian noisse. Firstly, the noisy image is divided into overlapping patches of size $d \times d$ . For each patch, its $m$ most similar patches located in its $M \times M$

neighbor region are grouped together to form a similarity matrix, each column being a vectorization of a similar patch. The similarity is determined by Euclidean norm. For each similarity matrix

S

, it is denoised by the exact low rank model

min X ∥ S - X ∥_{F}^{2} + m t^{2} Rank (X),

(1)

where the minimum is taken over all the matrices $X$ having the same size as $S$ , $∥ \cdot ∥_{F}$ is the Frobenius norm, and $t^{2}$ represents a threshold parameter( $t^{2} \approx 2 σ^{2}$ ). The solution can be obtained by

^S := P H_{t \sqrt{m}} (Σ) Q^{T},

(2)

where $P$ , $Q$ , and $Σ$ are derived from the SVD of $S$ , that is, $S = P Σ Q^{T}$ , and $H_{t \sqrt{m}}$ is the hard thresholding operator:

H_{t \sqrt{m}} (Σ)_{k k} = {\begin{matrix} Σ_{k k} & if Σ_{k k} > t \sqrt{m}, 0 & otherwise, \end{matrix} k = 1, 2, \dots, d^{2} .

(3)

Finally, each denoised similarity matrix is returned to the original location; the overlapping denoised values for each pixel are aggregated. The parameters are chosen as $d = 7, M = 43, m = 245, t = 1.5 σ$ .

3 Remove impulse noise

The noise model is as follows,

v_{0} = {\begin{matrix} η & with % probalility p u_{0} & with probalility 1 - p \end{matrix}

(4)

where $u_{0}$ is the original image, $v_{0}$ is the observed noisy one, $η$

is independent random noise uniformly distributed on [min

{u_{0}}

, max

{u_{0}}]

, generally taken as [0,255] for 8-bit gray images, and

p (0 < p < 1)

is the proportion of noise, representing the noise level.

As it is known that the fidelity term for Gaussian noise is $l_{2}$ norm, to obtain the fidelity term for impulse noise, we consider the maximum likelihood estimation (MLE) for $u_{0}$ . For simplicity, we utilize the discrete model. That is, for two pixels $u$ and $v$ of the same location of $u_{0}$ and $v$ respectively,

P {v = k} = {\begin{matrix} \frac{p}{256} & k \in {0, 1, 2 \dots 255} / u, 1 - p + \frac{p}{256} & k \in {u} . \end{matrix}

(5)

Given $n$ independent realizations ${v_{1}, v_{2}, \dots, v_{n}}$ of $v$ , the likelihood function is then

L (v_{1}, v_{2}, \dots, v_{n}) = (1 - p + \frac{p}{256})^{N} (\frac{p}{256})^{n - N},

(6)

where $N = n - \sum_{i}^{n} ∥ v_{i} - u ∥_{0}$ is the number $v_{i}$ taking the value $u$ . Recall that

∥ v_{i} - u ∥_{0} = {\begin{matrix} 0 & v_{i} = u, 1 & v_{i} \neq u . \end{matrix} i = 1, 2, \dots, n

(7)

Since $1 - p + \frac{p}{256} > \frac{p}{256}$ , $L (v_{1}, v_{2}, \dots, v_{n})$ is an increasing function of $N$ . That is,

arg max u L (v_{1}, v_{2}, \dots, v_{n}) = arg max u N = arg min u n \sum i ∥ v_{i} - u ∥_{0},

(8)

which means the fidelity term for impulse noise is $l_{0}$ norm.

According to above analysis, the following model is considered to remove impulse noise,

min u μ \sum l R a n k (R_{l} u) + ∥ u - v_{0} ∥_{0},

(9)

where $R_{l} u$ denotes similarity matrices. Since the model is not convex, it is solved formally by ADMM goldstein2009split ; ma2017low .

$u^{k + 1}$	$= arg min u ∥ (u - v_{0}) ∥_{0} + \frac{α}{2} ∥ u - v^{k} - b^{k} ∥_{F}^{2},$	(10)
$v^{k + 1}$	$= arg min v μ \sum l R a n k (R_{l} v) + \frac{α}{2} ∥ u^{k + 1} - v - b^{k} ∥_{F}^{2},$	(11)
$b^{k + 1}$	$= b^{k} + v^{k + 1} - u^{k + 1} .$	(12)

We use PWMF to produce initial image $v^{0}$ for the iteration, since it runs fast and has good results; the initial image $b^{0}$

is the zero matrix. By an element-wise calculation, the minimizer of (

10) has a close form:

u^{k + 1} = {\begin{matrix} v_{0}, if | v^{k} + b^{k} - v_{0} | < \sqrt{2 / α}, v^{k} + b^{k}, else . \end{matrix}

(13)

The $v$ -subproblem (11) is solved as PLR in Section 2, with $u^{k + 1} - b^{k}$ regarded as Gaussian noisy images and $m t^{2} = 2 μ / α$ . Since $v^{0}$ contains little impulse noise, $u^{k + 1} - b^{k}$ are good estimations of original images, which are thus used directly for the selection of similar patches for $v$ -sub problem as PLR.

4 Experiments

To show the performance of the proposed method, we test some classic images, the original clean images of which are displayed in Figure 1

. We use peak signal-to-noise ratio (PSNR) values to measure the quality of denoised images

¯ v

quantitatively, which are defined by PSNR

(¯ v) = 20 {log}_{10} (255 r / ∥ ¯ v - u_{0} ∥_{F})

dB with

r

being the squared root of the number of elements of the image

u_{0}

. For our method, we only need to choose the values of

μ

and

α

, and determine the stop criteria for the iteration. We use PLR with

t = 7.5

in (1) for the solution of (11) assuming

u^{k + 1} - b^{k}

has Gaussian noise level

σ = 5

. We choose

α

empirically by obtaining the highest average PSNR values of tested images, which is taken as

α = 1 / 72

for all the impulse noise levels

p = 0.2, 0.3, 0.4, 0.5

. Comparing (11) and (1), we thus use

μ = m t^{2} α / 2 \approx 95.7

. In addition, the fixed iteration times 50 is used.

The proposed method is compared with some recently published methods, robust ALOHAjin2018sparse , HNHOTV-OGSadam2021combined , and our previous method PWMFhu2016removing . We consider four noise level with $p = 0.2, 0.3, 0.4$ , and 0.5. For PWMF, we use the same parameters as in the paperhu2016removing . For robust ALOHA, since the paperjin2018sparse only considers $p = 0.25$ and $p = 0.4$ , we use the same parameters as $p = 0.25$ when $p = 0.2$ and 0.3, and the same parameters as $p = 0.4$ when $p = 0.4$ and 0.5. For HNHOTV-OGSadam2021combined , according to the paper, we adjust the parameters $λ$ in [43,79], and $ω$ in [4.8, 7.5],which are finally chosen as $λ = 77, ω = 6.2$ for $p = 0.2$ ; $λ = 77, ω = 6.4$ for $p = 0.3$ ; $λ = 75, ω = 6.8$ for $p = 0.4$ , and $λ = 75, ω = 7.2$ for $p = 0.5$ by obtaining highest average PSNR values of tested images.

The PSNR values of different methods are shown in Table 1. It is shown that our method is better than other methods for most of the images, especially for images with weak contrast, such as Lena and House. Robust ALOHAjin2018sparse is especially good for the high contrast image Barbara and is better for the image Cameraman when $p = 0.2$ than other methods, but it is not as good as HNHOTV-OGSadam2021combined or the proposed method in other cases. HNHOTV-OGS is better than the proposed method in only three cases, and the differences are small. The proposed method improves PWMF in all cases. In Figures 2,3,4, and 5, some images are displayed to show the differences of the methods. From these figures, it can be seen that our method improves PWMF: it retains clear image details in textural parts and edges, and also recovers well homogeneous regions. From Figures 2 and 4, it can be observed that robust ALOHA loses more details than our methods, while some noise or artifacts appear in Figures 3 and 5. The image details obtained by HNHOTV-OGS are not as clear as the proposed method either, which can be seen in Figures 3, 4, and 5.

Note that we use the same parameters for all the tested images. Since different images have different structures, the denoising performance can be improved with adaptive parameters, especially for high contrast images, which will be our future work.

Bar	Boa	Cam	Cou	Hill	Hou	Lena	Man	Mon	Pep
$p = 0.2$
28.43	31.86	27.26	31.94	33.59	35.06	35.77	33.07	28.63	32.08
34.79	31.05	28.09	31.06	32.65	34.57	35.36	31.15	28.29	30.10
25.83	32.49	27.13	33.36	33.84	37.24	36.48	32.86	30.29	31.87
29.46	33.68	27.68	33.51	36.26	40.37	38.83	34.11	29.97	33.17
$p = 0.3$
26.33	29.76	25.67	29.69	31.16	32.70	33.61	30.88	26.93	29.84
31.97	29.59	24.42	28.49	29.21	33.05	32.74	29.63	26.23	28.19
24.62	30.31	25.45	30.74	31.28	33.37	33.69	30.87	27.56	29.87
27.1	31.33	25.95	31.20	33.73	37.08	36.38	32.03	28.10	30.84
$p = 0.4$
25.09	27.77	23.74	27.65	29.83	30.93	31.81	29.31	25.06	28.03
29.30	26.84	23.53	26.91	28.59	30.27	30.11	27.24	23.25	26.24
23.60	28.28	23.88	28.69	29.19	30.57	31.39	29.12	25.08	28.16
25.64	28.59	23.76	28.62	32.07	33.75	33.46	29.91	25.62	28.66
$p = 0.5$
24.06	26.57	23.01	26.38	28.52	29.40	30.38	27.99	23.67	26.36
25.61	24.21	20.73	24.63	25.36	26.88	27.70	25.65	21.3	23.48
23.00	26.34	22.66	26.48	27.26	28.77	29.50	27.32	22.99	26.34
24.53	27.07	23.04	27.08	30.12	31.57	31.75	28.42	24.15	26.94

Table 1: PSNR values for removing impulse noise with

p = 0.2, 0.3, 0.4

and 0.5. For each

p

, from top to bottom, the methods are PWMFhu2016removing , robust ALOHAjin2018sparse , HNHOTV-OGSadam2021combined , and the proposed method. From left to right, the images are Barbara, Boats, Cameraman, Couple, Hill, House, Lena, Man, Monarch, Peppers.

Figure 2: Denoised images for $p = 0.2$ . The third and bottom rows show the enlarged versions of the same part of the Peppers images from the first and second rows respectively.

Figure 3: Denoised images for $p = 0.3$ The third and bottom rows show the enlarged versions of the same part of the House images from the first and second rows respectively.

Figure 4: Denoised images for $p = 0.4$ . The third and bottom rows show the enlarged versions of the same part of the Lena images from the first and second rows respectively.

Figure 5: Denoised images for $p = 0.5$ . The third and bottom rows show the enlarged versions of the same part of the Boats images from the first and second rows respectively.

References

(1) B. Hunt, The application of constrained least squares estimation to image restoration by digital computer, IEEE Transactions on Computers, 100 (9) (1973) 805–812.
(2) L. I. Rudin, S. Osher, E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena 60 (1-4) (1992) 259–268.
(3) A. Buades, B. Coll, J. M. Morel, A review of image denoising algorithms, with a new one, Multiscale Model. Simul. 4 (2) (2006) 490–530.
(4)
S. Gu, L. Zhang, W. Zuo, X. Feng, Weighted nuclear norm minimization with application to image denoising, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 2862–2869.
(5) H. Hu, J. Froment, Q. Liu, A note on patch-based low-rank minimization for fast image denoising, J. Visual Commun. Image Representation 50 (2018) 100–110.
(6) R. Garnett, T. Huegerich, C. Chui, W. He, A universal noise removal algorithm with an impulse detector, IEEE Trans. Image Process. 14 (11) (2005) 1747–1754.
(7) H. Hu, B. Li, Q. Liu, Removing mixture of gaussian and impulse noise by patch-based weighted means, J. Sci. Comput. 67 (1) (2016) 103–129.
(8) K. H. Jin, J. C. Ye, Sparse and low-rank decomposition of a hankel structured matrix for impulse noise removal, IEEE Trans. Image Process. 27 (2018) 1448–1461.
(9) T. Adam, R. Paramesran, M. Yin, K. Ratnavelu, Combined higher order non-convex total variation with overlapping group sparsity for impulse noise removal, Multim. Tools Appl. 80 (2021) 18503–18530.
(10) L. Jin, W. Zhang, G. Ma, E. Song, Learning deep cnns for impulse noise removal in images, J. Vis. Commun. Image Represent. 62 (2019) 193–205.
(11) F. Wang, H. Huang, J. Liu, Variational-based mixed noise removal with cnn deep learning regularization, IEEE Trans. Image Process. 29 (2020) 1246–1258.
(12) T. Goldstein, S. Osher, The split bregman method for l1-regularized problems, SIAM J. Imaging Sci. 2 (2) (2009) 323–343.
(13) S. Venkatakrishnan, C. A. Bouman, B. Wohlberg, Plug-and-play priors for model based reconstruction, 2013 IEEE Global Conference on Signal and Information Processing (2013) 945–948.
(14) L. Ma, L. Xu, T. Zeng, Low rank prior and total variation regularization for image deblurring, J. Sci. Comput. 70 (2017) 1336–1357.

Low rank prior and l0 norm to remove impulse noise in images

Sparse + Low Rank Decomposition of Annihilating Filter-based Hankel Matrix for Impulse Noise Removal

Patch-Based Low-Rank Minimization for Image Denoising

Occluded Face Recognition Using Low-rank Regression with Generalized Gradient Direction

Sparse and Low-Rank Covariance Matrices Estimation

1-Bit Matrix Completion under Exact Low-Rank Constraint

Sparse Trace Norm Regularization

Bilinear Parameterization For Differentiable Rank-Regularization

1 Introduction

2 Patch-based low rank method

3 Remove impulse noise

4 Experiments

References

References


Barbara	Boats	Cameraman	Couple	Hill

House	Lena	Man	Monarch	Peppers

Low rank prior and l0 norm to remove impulse noise in images

Related Research

Sparse + Low Rank Decomposition of Annihilating Filter-based Hankel Matrix for Impulse Noise Removal

Patch-Based Low-Rank Minimization for Image Denoising

Occluded Face Recognition Using Low-rank Regression with Generalized Gradient Direction

Sparse and Low-Rank Covariance Matrices Estimation

1-Bit Matrix Completion under Exact Low-Rank Constraint

Sparse Trace Norm Regularization

Bilinear Parameterization For Differentiable Rank-Regularization

1 Introduction

2 Patch-based low rank method

3 Remove impulse noise

4 Experiments

References

References