Coherence-Based Performance Guarantee of Regularized ℓ_1-Norm Minimization and Beyond

I Introduction

THE last decade has seen the burgeoning development of Compressed Sensing (CS) [1, 2] and its wide-spread applications in many fields. At the core of CS is the problem of efficiently recovering a sparse signal from a relatively small number of linear measurements. Mathematically, for any given signal $x \in R^{n}$ , we say that it is sparse if most of its entries are zero. More specifically, if it has at most $k$ non-zero entries, i.e., $∥ x ∥_{0} ≜ | supp (x) | \leq k$ , we call it a $k$ -sparse signal. In standard CS, one usually observes the linear measurements of the sparse signal $x$ via $b = A x$ , where $A \in R^{m \times n}$ ( $m ≪ n$ ) is a given measurement matrix. To recover such a sparse signal, a natural idea is to search the sparsest solution among all the possible solutions. This directly leads to the following $ℓ_{0}$ -norm minimization problem

min x \in R^{n} ∥ x ∥_{0}, s . t . b = A x .

(1)

Unfortunately, this problem is NP-hard in general, and hence it is computationally infeasible. Instead, some algorithms which aims to pursue the suboptimal solutions of (1) were proposed, see, e.g., [3, 4, 5] and their variants [6, 7, 8, 9]. Importantly, many of these algorithms have been proved to perform well under certain conditions.

Besides the above algorithm strategies, there also exist many other efficient approaches [10, 11, 12, 13, 14, 15] which can circumvent the NP-hardness of (1), and a popular one is the constrained $ℓ_{1}$ -norm minimization method which solves

min x \in R^{n} ∥ x ∥_{1}, s . t . b = A x,

(2)

where $∥ \cdot ∥_{1}$ is the $ℓ_{1}$

-norm of vector. Problem (

2) is convex and therefore can be well addressed by many convex optimization softwares. To theoretically investigate the equivalence between (1) and (2), one often adopts the Restricted Isometry Constant (RIC) of matrix with

k

order, denoted by

δ_{k}

, which is defined to be the smallest value of

δ \in (0, 1)

such taht

(1 - δ) ∥ x ∥_{2}^{2} \leq ∥ A x ∥_{2}^{2} \leq (1 + δ) ∥ x ∥_{2}^{2}

for every $k$ -sparse vector $x$ . This notation was first proposed by Candès and Tao in [10], where they have shown that (2) is equivalent to (1) in noiselessly recovering any $k$ -sparse signals when $δ_{k} + δ_{2 k} + δ_{3 k} < 1$ . Subsequently many researchers were committed to improving this condition, see, e.g., [16, 17, 18, 19, 20, 21, 22, 23]. In more general application scenarios of CS, one often wishes to recover the original signal $x$ (may not be exactly sparse) from the noisy observation $b$ with

b = A x + z,

(3)

where $z \in R^{n}$ is the unknown measurement noise/error, which directly leads to the following optimization problem

min x \in R^{n} ∥ x ∥_{1}, s . t . ∥ b - A x ∥_{2} \leq ϵ,

(4)

where $ϵ \geq 0$ . Obviously, (4) will reduces to (2) if one takes $ϵ = 0$ . It should also be noted that the above-mentioned exact recovery conditions are still available to guarantee the robust recovery of signals from (4) in the presence of noise.

On the other hand, when used in many practical applications, particularly the applications where the input data are in large scale, the constrained problem (4) is not always convenient to solve. Instead, one often solves its unconstrained counterpart, i.e., the ( $ℓ_{2}$ -norm) regularized $ℓ_{1}$ -norm minimization problem

min x \in R^{n} ∥ x ∥_{1} + \frac{1}{2 λ} ∥ b - A x ∥_{2}^{2},

(5)

where $λ$ is the nonnegative tradeoff parameter. This problem is also known as the Lasso estimator [24] or the Basis Pursuit DeNoising (BPDN) [25] , and it can also be solved efficiently by many algorithms, see, e.g., [26, 27, 28, 29, 30]. Recently, the relation between (4) and (5) was carefully investigated by Zhang, Yuan and Yin [31] in the context of the non-uniform recovery [32], i.e., the recovery of some specific sparse signals, for example, the sparse signals limited in a specific support. However, when it comes to the uniform recovery [31, 32], i.e., recovering all the (general) sparse signals ¹¹1In these signals, both the number and support of their non-zero entries are not known in advance., it is generally believed that (4) and (5) are not exact equivalents. As early as 2008, Zhu [33] has derived the RIC-based theoretical guarantee for (5), which states that one can robustly recover any $k$ -sparse signal $x$ through (3) with $∥ z ∥_{2} = ϵ$ by using (5) under certain $λ$ , if $A$ obeys $δ_{4 k} + 2 δ_{5 k} < 1$ . However, this work was relatively rarely noted by the researchers. In 2009, Bickel, Ritov and Tsybakov [34] established a RIC-like guarantee for (5) in the presence of the random noise. Recently, some new RIC conditions were obtained to ensure the robust recovery of some unconstrained $ℓ_{1}$ -analysis approaches under the Dantzig Selector (DS) type noise/error (i.e., $∥ A^{T} z ∥_{\infty} \leq ϵ$ ), see [35, 36, 37] for details.

Some random matrices represented by sub-Gaussian random matrices are proved to have a small RIC with overwhelmingly high probability

[32]. However, when used in practical scenarios they often suffer from storage and computation limitations. Moreover, it is also NP-hard in general to find the RIC of any given matrix. To overcome these difficulties, some researchers proposed to reuse the mutual coherence, another powerful tool introduced by Mallat and Zhang [38] in their initial research on matching pursuit. In this paper, denoting by

a_{i} \in R^{m}

the

i

th column of matrix

A

, we shall define the mutual coherence of matrix

A ≜ [a_{1}, a_{2}, \dots, a_{n}]

μ = max 1 \leq i < j \leq n | ⟨ a_{i}, a_{j} ⟩ |,

where we assume that $a_{i}$ obeys $∥ a_{i} ∥_{2} = 1$ for $i = 1, 2, \dots, n$ . Many deterministic measurement matrices in fact are designed according to the mutual coherence. There are many coherence-based theoretical results for (4), see, e.g., [39, 40, 41, 42, 43, 44]. In particular in [44], Cai, Wang and Xu have shown that any signals (not necessary to be $k$ -sparse) can be robustly recovered using (4), if $A$ satisfies $μ < 1 / (2 k - 1)$ , and this condition is also sharp for the noiseless recovery of any $k$ -sparse signals through (2).

As far as we know, the first (mutual) coherence-based result on (5) was given by Fuchs [41] in 2004 under the non-uniform recovery setting, which states that any fixed signal $x$ with $k$ non-zero entries (i.e., $∥ x ∥_{0} = k$ ) can be uniquely recovered from $b = A x$ using (5), if $A$ satisfies $μ < 1 / (2 k - 1)$ and $λ$ in (5) has been taken small enough. Later, Fuchs [45] further investigated (5) for the noisy signal recovery, and shown that if $b$ is observed through (3) with $∥ x ∥_{0} = k$ and $∥ z ∥_{2} \leq ϵ$ , $A$ satisfies $μ \leq c / k$ for certain $c \leq 1 / 2$ , and $x^{♯}$ is assumed to be the optimal solution of (5) under certain $λ$ (related to $μ$ , $c$ and $ϵ$ ), then the support of $x^{♯}$ will be either identical to, or contained in, that of $x$ . Moreover, Fuchs also shown that if similar constraints are imposed on $x$ , $z$ and $λ$ , $x^{♯}$ and $x$ will have their non-zero entries at the same support and with the same signs. Subsequently, Tropp [46] further extended the results in [45] to more general case. In 2010, Ben-Haim, Eldar and Elad [47] revisited (5) under the random noise, and their obtained coherence results have been proved to be better than those induced in [34]. Note that the above coherence results apply only to deal with the signals whose sparsity is known in advance. Recently, using the cumulative coherence [48] tool, Li and Chen [49] established a new uniform recovery condition for (5) to deal with the signal recovery in the presence of noise. Their results shown that if $A$ obeys $μ \leq 1 / (\sqrt{3} (5 k - 2))$ , one can robustly recover any signals corrupted with the DS type noise. However, the noise they considered is based on the DS type noise rather than the often used $ℓ_{2}$ -norm bounded noise, and the recovery condition they obtained still has much room to improve.

In this paper, equipped with the powerful mutual coherence tool, we investigate the performance guarantees of (5) and its some variants. In summary, the contributions of this paper are listed as follows:

We establish a tight uniform recovery condition and two relatively tight error estimates for (5), which are sufficient to guarantee the robust recovery of signals corrupted with the $ℓ_{2}$ -norm bounded noise.
We extend the obtained theoretical results to guarantee the robust recovery of the signals corrupted with the DS type noise and also the structured block-sparse signal recovery in the presence of the bounded noise. To the best of our knowledge, these extended results are established for the first time under the uniform recovery setting.

The remainder of this paper is organized as follows. Section II introduces some notations and preliminaries. In Section III, we present the main results. Section IV shows two extensions. Finally, conclusion and future work are given in Section V.

Ii Notations and Preliminaries

Ii-a Notations

Throughout this paper, we denote $[r] ≜ {1, 2, \dots, r}$ for any given integer $r$ , and $E^{c} = [n] ∖ E$ for any given index set $E \subset [n]$ . We also denote $h_{E}$ as a vector whose entries $(h_{E})_{i} = h_{i}$ for $i \in E$ and 0 otherwise, and $∥ \cdot ∥_{a}^{b} = (∥ \cdot ∥_{a})^{b}$ where $∥ \cdot ∥_{a}$ represents certain norm or quasi-norm. For any signal $x \in R^{n}$ , we denote its best $k$ -term approximate as $x_{[k]}$ , which is defined as

x_{[k]} = arg min ∥ y ∥_{0} \leq k ∥ y - x ∥_{2} .

Besides, for the simplicity of symbol expression we introduce the following two functions

f_{a} (x) = a x^{2} + 3 \sqrt{a} x + 3, g_{a} (x) = 2 a x^{2} + 4 \sqrt{a} x + 1.

Ii-B Three key lemmas

The proof of our main results heavily relies on the following three lemmas. We start with introducing the first lemma, which provides a RIC-like coherence result for any given matrix.

Lemma 1 ([46, 42]).

Assume that the matrix $A \in R^{m \times n}$ is standardized to have unit $ℓ_{2}$ -norm. Then it holds that

(1 - (k - 1) μ) ∥ y ∥_{2}^{2} \leq ∥ A y ∥_{2}^{2} \leq (1 + (k - 1) μ) ∥ y ∥_{2}^{2}

(6)

for all $k$ -sparse signals $y \in R^{n}$ .

Lemma 2.

If $b$ is observed via (3) with $∥ z ∥_{2} \leq ϵ$ , then for any subset $E \subset [n]$ with $| E | = k$ , the optimal solution $x^{♯}$ of (5) satisfies

∥ A h ∥_{2}^{2} - 2 ϵ ∥ A h ∥_{2} \leq

2 λ (∥ h_{E} ∥_{1} - ∥ h_{E^{c}} ∥_{1} + 2 ∥ x_{E^{c}} ∥_{1})

(7)

and

∥ h_{E^{c}} ∥_{1} \leq ∥ h_{E} ∥_{1} + 2 ∥ x_{E^{c}} ∥_{1} + \frac{ϵ}{λ} ∥ A h ∥_{2},

(8)

where $h = x^{♯} - x$ .

Proof:

Since $x^{♯}$ is the optimal solution of (5), we have

∥ x^{♯} ∥_{1} + \frac{1}{2 λ} ∥ b - A x^{♯} ∥_{2}^{2} \leq ∥ x ∥_{1} + \frac{1}{2 λ} ∥ b - A x ∥_{2}^{2},

which is equivalent to

∥ A h ∥_{2}^{2} - 2 ⟨ z, A h ⟩ \leq 2 λ (∥ x ∥_{1} - ∥ x^{♯} ∥_{1}) .

(9)

As to the Left-Hand Side (LHS) of (9), we have

LHS \geq ∥ A h ∥_{2}^{2} - 2 ϵ ∥ A h ∥_{2} .

(10)

As to the Right-Hand Side (RHS) of (9), we know

$RHS =$	$n \sum i = 1 \| x_{i} + h_{i} \| - ∥ x ∥_{1}$
$\geq$	$\sum i \in E \| x_{i} - h_{i} \| + \sum i \in E^{c} \| h_{i} - x_{i} \| - ∥ x ∥_{1}$
$\geq$	$\sum i \in E (\| x_{i} \| - \| h_{i} \|) + \sum i \in E^{c} (\| h_{i} \| - \| x_{i} \|) - (∥ x_{E} ∥_{1} + ∥ x_{E^{c}} ∥_{1})$
$=$	$- ∥ h_{E} ∥_{1} + ∥ h_{E^{c}} ∥_{1} - 2 ∥ x_{E^{c}} ∥_{1} .$	(11)

Therefore combing (9), (10) and (II-B) directly leads to (7), and (8) follows trivially from (7). ∎

Lemma 3.

If the matrix $A \in R^{m \times n}$ is standardized to have unit $ℓ_{2}$ -norm, and obeys

μ < \frac{1}{k - 1}

(12)

for certain integer $k \geq 2$ , then for any vector $h \in R^{n}$ and any subset $E \subset [n]$ with $| E | = k$ , it holds that

∥ h_{E} ∥_{2} \leq α_{1} ∥ A h ∥_{2} + α_{2} ∥ h_{E^{c}} ∥_{1},

(13)

where

α_{1} ≜ \frac{\sqrt{1 + (k - 1) μ}}{1 - (k - 1) μ} ~{}% ~{}and~{}~{} α_{2} ≜ \frac{\sqrt{k} μ}{1 - (k - 1) μ} .

Remark 1.

It is easy to know from Lemma 3 that both $α_{1}$ and $α_{2}$ are two monotone increasing functions on variable $μ$ . Therefore if one restricts $μ < 1 / (2 k - 1)$ , it will be clear that

1 < α_{1} < \sqrt{6}, \sqrt{k} α_{2} < 1 ~{}and~{} \frac{1}{1 - \sqrt{k} α_{2}} < \frac{1}{1 - (2 k - 1) μ} .

(14)

Proof:

The proof is simple. We start with estimating the lower and upper bounds of

ρ ≜ | ⟨ A h, A h_{E} ⟩ | .

First, using Lemma 1, we know

$ρ \geq$	$\| ⟨ A h_{E}, A h_{E} ⟩ \| - \| ⟨ A h_{E^{c}}, A h_{E} ⟩ \|$
$\geq$	$(1 - (k - 1) μ) ∥ h_{E} ∥_{2}^{2} - \| \sum i \in E \sum j \in E^{c} ⟨ a_{i}, a_{j} ⟩ h_{i} h_{j} \|$
$\geq$	$(1 - (k - 1) μ) ∥ h_{E} ∥_{2}^{2} - μ ∥ h_{E} ∥_{1} ∥ h_{E^{c}} ∥_{1}$
$\geq$	$(1 - (k - 1) μ) ∥ h_{E} ∥_{2}^{2} - \sqrt{k} μ ∥ h_{E} ∥_{2} ∥ h_{E^{c}} ∥_{1} .$	(15)

Next we estimate the upper bound of $ρ$ . It follows from 1 that

ρ \leq

∥ A h ∥_{2} ∥ A h_{E} ∥_{2} \leq \sqrt{1 + (k - 1) μ} ∥ h_{E} ∥_{2} ∥ A h ∥_{2} .

(16)

Now, combing (II-B), (16) and the condition (12) directly leads to the desired inequality (13). ∎

Iii Main Results

With preparations above, we now present our main results.

Theorem 1.

For any $b$ observed via (3) with $∥ z ∥_{2} \leq ϵ$ , if the measurement matrix $A$ , which is standardized to have unit $ℓ_{2}$ -norm, satisfies

μ < \frac{1}{2 k - 1}

(17)

for certain integer $k \geq 2$ , then we have

	$∥ A (x^{♯} - x) ∥_{2}$	$\leq C_{1} (α_{1}) ∥ x - x_{[k]} ∥_{1} + C_{2} (α_{1}),$		(18)
	$∥ x^{♯} - x ∥_{2}$	$\leq C_{3} (α_{1}, α_{2}) ∥ x - x_{[k]} ∥_{1} + C_{4} (α_{1}, α_{2}),$

where $x^{♯}$ is the optimal solution of (5), and

	$C_{1} (α_{1})$	$≜ \frac{2 λ}{\sqrt{k} α_{1} λ + ϵ},$
	$C_{2} (α_{1})$	$≜ 2 (\sqrt{k} α_{1} λ + ϵ),$
	$C_{3} (α_{1}, α_{2})$	$≜ \frac{2 \sqrt{k} α_{1} f_{k} (α_{2}) λ + 2 g_{k} (α_{2}) ϵ}{\sqrt{k} (1 - \sqrt{k} α_{2}) (\sqrt{k} α_{1} λ + ϵ)},$
	$C_{4} (α_{1}, α_{2})$	$≜ \frac{\sqrt{k} α_{1} (5 + 2 \sqrt{k} α_{2}) λ + g_{k} (α_{2}) ϵ}{\sqrt{k} (1 - \sqrt{k} α_{2}) λ (\sqrt{k} α_{1} λ + ϵ)^{- 1}} .$

Remark 2.

Theorem 1 shows that one can robustly recover any signals (may not be $k$ -sparse) corrupted with the $ℓ_{2}$ -norm bounded noise, if the measurement matrix $A$ satisfies (17). To the best of our knowledge, we first extend this sharp uniform recovery condition ²²2The sharp condition/bound here and throughout will refer to $μ < 1 / (2 k - 1)$ . It has been shown in [44] that, any $k$ -sparse signal $x$ , without exception, can be exactly recovered from (2) under this sharp condition, and there exists a matrix $A$ with $m < n$ obeying $u = 1 / (2 k - 1)$ , and two nonzero $k$ -sparse vectors $˜ x$ and $ˆ x$ with disjoint supports such that $A ˜ x \neq A ˆ x$ . derived by Cai, Wang and Xu in [44] for the constrained problem (2) to its unconstrained counterpart, i.e., the unconstrained problem (5). Similar to [36, 37], if we associate $ϵ$ with $λ$ , e.g., setting $ϵ = λ$ , then we get a special case of Theorem 1, and one can find this result in Corollary 1. In Remark 3, we will analyze the tightness of these two error estimates under the setting of $ϵ = λ$ . Besides, it is also very easy to induce some other special cases of Theorem 1 to cope with several different sparse recovery tasks. For examples, one can consider the robust recovery of any exactly sparse signals, i.e., setting the original signals $x$ to be exactly $k$ -sparse. The detailed analysis of these cases will become very similar to that of Corollary 1.

Corollary 1.

Assume that $b$ is observed via (3) with $∥ z ∥_{2} \leq λ$ . If the measurement matrix $A$ is standardized to have unit $ℓ_{2}$ -norm, and also satisfies (17) for certain integer $k \geq 2$ , then we have

	$∥ A (x^{♯} - x) ∥_{2}$	$\leq {ˆ C}_{1} ∥ x - x_{[k]} ∥_{1} + {ˆ C}_{2} λ,$
	$∥ x^{♯} - x ∥_{2}$	$\leq {ˆ C}_{3} ∥ x - x_{[k]} ∥_{1} + {ˆ C}_{4} λ,$

where $x^{♯}$ is the optimal solution of (5), and

	${ˆ C}_{1}$	$= \frac{2}{\sqrt{k} α_{1} + 1},$
	${ˆ C}_{2}$	$= 2 (\sqrt{k} α_{1} + 1),$
	${ˆ C}_{3}$	$= \frac{2 \sqrt{k} α_{1} f_{k} (α_{2}) + 2 g_{k} (α_{2})}{\sqrt{k} (1 - \sqrt{k} α_{2}) (\sqrt{k} α_{1} + 1)},$
	${ˆ C}_{4}$	$= \frac{\sqrt{k} α_{1} (5 + 2 \sqrt{k} α_{2}) + g_{k} (α_{2})}{\sqrt{k} (1 - \sqrt{k} α_{2}) (\sqrt{k} α_{1} + 1)^{- 1}} .$

Remark 3.

Due to the existence of $α_{1}$ and $α_{2}$ , the coefficients ${ˆ C}_{1}, \dots, {ˆ C}_{4}$ are not convenient to be analyzed. Fortunately, based on the previous estimates for $α_{1}$ and $α_{2}$ , i.e., (14), we can give a rough but simple estimate for each ${ˆ C}_{i}$ . Specifically,

	${ˆ C}_{1} <$	$\frac{2}{\sqrt{k} + 1} < \frac{2}{\sqrt{k}},$
	${ˆ C}_{2} <$	$2 (\sqrt{6 k} + 1) \leq 2 (\sqrt{6} + 1) \sqrt{k},$
	${ˆ C}_{3} <$	$\frac{14 (\sqrt{6 k} + 1)}{\sqrt{k} (\sqrt{k} + 1) (1 - \sqrt{k} α_{2})} < \frac{14 \sqrt{6}}{\sqrt{k} (1 - (2 k - 1) μ)},$
	${ˆ C}_{4} <$	$\frac{7 (\sqrt{6 k} + 1)^{2}}{\sqrt{k} (1 - \sqrt{k} α_{2})} < \frac{7 (\sqrt{6} + 1)^{2} \sqrt{k}}{1 - (2 k - 1) μ} .$

These upper bound estimates of coefficients make our recovery error, denoted by RE, have the form of

RE \leq {¯ ¯¯ ¯ C}_{1} \frac{∥ x - x_{[k]} ∥_{1}}{\sqrt{k}} + {¯ ¯¯ ¯ C}_{2} \sqrt{k} λ,

(19)

where RE stands for $∥ A (x^{♯} - x) ∥_{2}$ or $∥ x^{♯} - x ∥_{2}$ , and ${¯ ¯¯ ¯ C}_{1}$ and ${¯ ¯¯ ¯ C}_{2}$ depend only on the value of $1 - (2 k - 1) μ$ , which characterizes the gap between the coherence of the selected measurement matrix $A$ and its sharp bound. This result also coincides with the ones obtained in [35, 36, 37, 49] for (5) in form. However, one should note that the authors in these literature focus on sparse recovery corrupted with the DS type noise, which is totally different from ours. Despite this, our obtained upper bound estimates to some degree are still better than theirs since a much tighter (or sharp) recovery condition is used. What’s more, some coefficients in these estimates can be further improved if one optimizes some inequalities used to prove Theorem 1.

Proof:

We first assume that $μ < 1 / (k - 1)$ for certain integer $k \geq 2$ and denote

E = supp (x_{[k]}) ~{}~{}and~{}~{} h = x^{♯} - x,

then using Lemma 2, Lemma 3, we have

$∥ A h ∥_{2}^{2} - 2 ϵ ∥ A h ∥_{2} \leq$	$2 λ (∥ h_{E} ∥_{1} - ∥ h_{E^{c}} ∥_{1} + 2 ∥ x_{E^{c}} ∥_{1})$
$\leq$	$2 \sqrt{k} λ (α_{1} ∥ A h ∥_{2} + α_{2} ∥ h_{E^{c}} ∥_{1}) - 2 λ ∥ h_{E^{c}} ∥_{1} + 4 λ ∥ x_{E^{c}} ∥_{1}$
$=$	$2 \sqrt{k} α_{1} λ ∥ A h ∥_{2} + 4 λ ∥ x_{E^{c}} ∥_{1} - 2 (1 - \sqrt{k} α_{2}) λ ∥ h_{E^{c}} ∥_{1},$	(20)

where we used $∥ h_{E} ∥_{1} \leq \sqrt{k} ∥ h_{E} ∥_{2}$ . We can known from the condition (17) that

	$1 - \sqrt{k} α_{2}$	$= 1 - \frac{k μ}{1 - (k - 1) μ}$
		$> 1 - \frac{k / (2 k - 1)}{1 - (k - 1) / (2 k - 1)} = 0.$

Therefore we can further write (III) as

∥ A h ∥_{2}^{2} - 2 (ϵ + \sqrt{k} α_{1} λ) ∥ A h ∥_{2} - 4 λ ∥ x_{E^{c}} ∥_{1} \leq 0.

which implies that

	$∥ A h ∥_{2} \leq$	$(\sqrt{k} α_{1} λ + ϵ) + \sqrt{(\sqrt{k} α_{1} λ + ϵ)^{2} + 4 λ ∥ x_{E^{c}} ∥_{1}}$
	$\leq$	$(\sqrt{k} α_{1} λ + ϵ) + \sqrt{{(\sqrt{k} α_{1} λ + ϵ + \frac{2 λ ∥ x_{E^{c}} ∥_{1}}{\sqrt{k} α_{1} λ + ϵ})}_{1}^{2}}$
	$=$	$\frac{2 λ}{\sqrt{k} α_{1} λ + ϵ} ∥ x_{E^{c}} ∥_{1} + 2 (\sqrt{k} α_{1} λ + ϵ) .$

This completes (18). Based on (18), (8) and (13), we have

$∥ h_{E} ∥_{2} \leq$	$α_{1} ∥ A h ∥_{2} + α_{2} (∥ h_{E} ∥_{1} + 2 ∥ x_{E^{c}} ∥_{1} + \frac{ϵ}{λ} ∥ A h ∥_{2})$
$=$	$\frac{α_{1} λ + α_{2} ϵ}{λ} ∥ A h ∥_{2} + \sqrt{k} α_{2} ∥ h_{E} ∥_{2} + 2 α_{2} ∥ x_{E^{c}} ∥_{1}$
$\leq$	$\frac{α_{1} λ + α_{2} ϵ}{(1 - \sqrt{k} α_{2}) λ} ∥ A h ∥_{2} + \frac{2 α_{2}}{1 - \sqrt{k} α_{2}} ∥ x_{E^{c}} ∥_{1}$
$\leq$	$\frac{α_{1} λ + α_{2} ϵ}{(1 - \sqrt{k} α_{2}) λ} (\frac{2 λ}{\sqrt{k} α_{1} λ + ϵ} ∥ x_{E^{c}} ∥_{1} + 2 \sqrt{k} α_{1} λ + 2 ϵ) + \frac{2 α_{2}}{1 - \sqrt{k} α_{2}} ∥ x_{E^{c}} ∥_{1}$
$=$	$\frac{2 α_{1} (1 + \sqrt{k} α_{2}) λ + 4 α_{2} ϵ}{(1 - \sqrt{k} α_{2}) (\sqrt{k} α_{1} λ + ϵ)} ∥ x_{E^{c}} ∥_{1} + \frac{2 (α_{1} λ + α_{2} ϵ) (\sqrt{k} α_{1} λ + ϵ)}{(1 - \sqrt{k} α_{2}) λ} .$	(21)

Besides, using (III) together with (8) and (18) again, we can estimate $∥ h_{E^{c}} ∥_{1}$ as

∥ h_{E^{c}} ∥_{1} \leq \frac{4}{1 - \sqrt{k} α_{2}} ∥ x_{E^{c}} ∥_{1} + \frac{2 (\sqrt{k} α_{1} λ + ϵ)^{2}}{(1 - \sqrt{k} α_{2}) λ} .

(22)

On the other hand, let $E_{1}$ be the index set of the $k$ largest entries of $h_{E^{c}}$ . Then we know from [36] that

∥ h_{E^{c}} ∥_{2} \leq ∥ h_{E_{1}} ∥_{2} + \frac{∥ h_{E^{c}} ∥_{1}}{2 \sqrt{k}} .

(23)

Similarly, using Lemma 3 again on index $E_{1}$ , we also have

∥ h_{E_{1}} ∥_{2} \leq α_{1} ∥ A h ∥_{2} + α_{2} ∥ h_{E_{1}^{c}} ∥_{1} .

This, together with (18) and (III), directly leads to

	$∥ h_{E_{1}} ∥_{2} \leq$	$α_{1} (\frac{2 λ}{\sqrt{k} α_{1} λ + ϵ} ∥ x_{E^{c}} ∥_{1} + 2 \sqrt{k} α_{1} λ + 2 ϵ) + α_{2} (∥ h_{E} ∥_{1} + ∥ h_{E^{c}} ∥_{1})$
	$\leq$	$\frac{2 α_{1} (1 + k (α_{2})^{2}) λ + 4 \sqrt{k} (α_{2})^{2} ϵ}{(1 - \sqrt{k} α_{2}) (\sqrt{k} α_{1} λ + ϵ)} ∥ x_{E^{c}} ∥_{1} + \frac{2 (α_{1} λ + \sqrt{k} (α_{2})^{2} ϵ)}{(1 - \sqrt{k} α_{2}) λ (\sqrt{k} α_{1} λ + ϵ)^{- 1}} + α_{2} ∥ h_{E^{c}} ∥_{1} .$		(24)

Now, combining (III), (22) and (III), we can estimate $∥ h ∥_{2}$ as follows:

	$∥ h ∥_{2} \leq$	$∥ h_{E} ∥_{2} + ∥ h_{E^{c}} ∥_{2}$
	$\leq$	$∥ h_{E} ∥_{2} + ∥ h_{E_{1}} ∥_{2} + \frac{∥ h_{E^{c}} ∥_{1}}{2 \sqrt{k}}$
	$\leq$	$\frac{2 \sqrt{k} α_{1} f_{k} (α_{2}) λ + 2 g_{k} (α_{2}) ϵ}{\sqrt{k} (1 - \sqrt{k} α_{2}) (\sqrt{k} α_{1} λ + ϵ)} ∥ x_{E^{c}} ∥_{1} + \frac{\sqrt{k} α_{1} (5 + 2 \sqrt{k} α_{2}) λ + 2 g_{k} (α_{2}) ϵ}{\sqrt{k} (1 - \sqrt{k} α_{2}) λ (\sqrt{k} α_{1} λ + ϵ)^{- 1}},$

which completes the proof. ∎

Iv Extensions

In this section, two extensions of Theorem 1 are discussed. They include extending Theorem 1 to guarantee the robust recovery of signals from a DS regularized $ℓ_{1}$ -norm minimization model in the presence of the DS type noise, and that of the structured block-sparse signals from two regularized mixed $ℓ_{2} / ℓ_{1}$ -norm minimization models in the presence of the bounded noise. We start with introducing the DS regularized $ℓ_{1}$ -norm minimization model for signal recovery in the presence of the DS type noise.

Iv-a Robust recovery via a DS regularized $ℓ_{1}$ -norm minimization

The research on the DS type noise was initiated by Candès and Tao in [50], which aims at recovering the signals corrupted with the DS type noise by solving the following constrained problem

min x \in R^{n} ∥ x ∥_{1}, s . t . ∥ A^{T} (b - A x) ∥_{\infty} \leq ϵ .

(25)

Many remarkable results on this problem have been obtained over the past decade, see, e.g., [34, 18, 20, 21, 22, 23, 44]. Similar to the relation of (4) and (5), a closely related problem to (25) is the following DS regularized $ℓ_{1}$ -norm minimization problem:

min x \in R^{n} ∥ x ∥_{1} + \frac{1}{2 λ} ∥ A^{T} (b - A x) ∥_{\infty}^{2} .

(26)

Inspired by Theorem 1, we also establish a uniform recovery condition and two relatively tight error estimates for (26) to guarantee the robust signal recovery in the presence of such kind of noise, see Theorem 2 for details. This new theorem as well as Theorem 1, to the best of our knowledge, first extends the sharp uniform recovery condition obtained in [44] for (2) to its two unconstrained variants, i.e., (5) and (26), to deal with the signals corrupted with the $ℓ_{2}$ -norm bounded noise and the DS type noise, respectively. In what follows, we present this theorem.

Theorem 2.

For any $b$ observed via (3) with $∥ A^{T} z ∥_{2} \leq ϵ$ , if the measurement matrix $A$ , whose columns are standardized to have unit $ℓ_{2}$ -norm, satisfies

μ < \frac{1}{2 k - 1}

(27)

for certain integer $k \geq 2$ , then we have

	$∥ A^{T} A (x^{♯} - x) ∥_{\infty}$	$\leq C_{1} (α_{1}) ∥ x - x_{[k]} ∥_{1} + C_{2} (α_{1}),$
	$∥ x^{♯} - x ∥_{2}$	$\leq C_{3} (α_{1}, α_{2}) ∥ x - x_{[k]} ∥_{1} + C_{4} (α_{1}, α_{2}),$

where $x^{♯}$ here denotes the optimal solution of (26).

Remark 4.

In general, it is usually suggested to recover the signals corrupted with the DS type noise using the constrained problem (25), see, e.g., [50, 21, 22, 23]. Recently, some researchers proposed to deal with such kind of noise using the unconstrained problem (5), see, e.g., [35, 36, 37, 49], and they also developed a series of recovery conditions and error estimates to realize the robust recovery from (5). However, these results are far from the best. Take for example the mutual coherence condition ³³3Their original condition in fact is obtained under the cumulative coherence notation. However, whether this condition is sharp was not discussed by the authors. recently obtained in [49], which takes the form of

μ < \frac{2}{\sqrt{3} (5 k - 2)} .

(28)

Obviously (28) is rigorously included in our sharp condition (27). In the aspect of algorithm implementation, since (26) is convex, many convex optimization softwares are available to solve it efficiently. Besides, compared to the regularization term (i.e., the second term of the objective function) in (5), the one in (26) is non-smooth and thus non-differentiable. However, if one solves (26) using some non-gradient algorithms, such as the alternating direction method and multipliers [29, 51], (26) is still comparable to (5) in terms of the algorithmic complexity.

Proof:

The proof is very similar to that of Theorem 1, and hence we here only present some technique differences. Our proof also relies on Lemma 1 and the variants of Lemma 2 and Lemma 3. One should keep in mind that the term $∥ A h ∥_{2}$ will be replaced by $∥ A^{T} A h ∥_{\infty}$ . Specifically, (7), (8) and (13) are replaced in order by the following inequalities

	$∥ A^{T} A h ∥_{\infty}^{2} - 2 ϵ ∥ A^{T} A h ∥_{\infty} \leq$	$2 λ (∥ h_{E} ∥_{1} - ∥ h_{E^{c}} ∥_{1} + 2 ∥ x_{E^{c}} ∥_{1}),$
	$∥ h_{E^{c}} ∥_{1} \leq$	$∥ h_{E} ∥_{1} + 2 ∥ x_{E^{c}} ∥_{1} + \frac{ϵ}{λ} ∥ A^{T} A h ∥_{\infty},$
	$∥ h_{E} ∥_{2} \leq$	$α_{1} ∥ A^{T} A h ∥_{\infty} + α_{2} ∥ h_{E^{c}} ∥_{1} .$

These, as well as the skills in proving Theorem 1, are sufficient to prove Theorem 2. ∎

Iv-B Structure block-sparse recovery

Our Theorem 1 can still be extended to guarantee the robust recovery of the structured block-sparse signals. Such a kind of signals (data) arise in many applications [52, 53, 54]. We assume w.l.o.g. that there are $l$ blocks with block size $d = n / l$ in signal $x \in R^{n}$ , and then we can write any signal $x \in R^{n}$ as

x=[x1,⋯,xd(x[1])T,xd+1,⋯,x2d\emph(x[2])T,⋯,xN−d+1,⋯,xn\emph(x[l])T]T

where $x [i] \in R^{d}$ denotes the $i$ th block sub-vector of $x$ . If $x$ has at most $k$ non-zero blocks, i.e., $∥ x ∥_{2, 0} \leq k$ , we refer to such a vector $x$ as block $k$ -sparse signal. Naturally, a block $k$ -sparse signals will reduce the traditional $k$ -sparse signal if one takes $d = 1$ . Accordingly, we can also write any matrix $A \in R^{m \times n}$ as

A = [a_{1}, \dots, a_{d}      A [1], a_{d + 1}, \dots, a_{2 d}      A [2], \dots, a_{N - d + 1}, \dots, a_{n}      A [l]]

where $A [i] \in R^{m \times d}$ denotes the $i$ th block sub-matrix of $A$ . To recover such a structured block-sparse signal, Eldar amd Mishali [55] proposed solving the following mixed $ℓ_{2} / ℓ_{1}$ -norm minimization problem:

min x \in R^{n} ∥ x ∥_{2, 1}, s . t . b = A x,

(29)

where $∥ x ∥_{2, 1} ≜ \sum_{i = 1}^{l} ∥ x [i] ∥_{2}$ , and they also derived a block-RIC recovery condition for (29). More improved block-RIC conditions can be found in [56, 57, 58, 59]. As early as 2010, Eldar, Kuppinger and Bölcskei [60] have generalized the traditional mutual coherence to the block setting, and show that any block $k$ -sparse signal $x$ can be exactly recovered via (29) if $A$ obeys

μ_{B} \leq \frac{1 - (d - 1) ν}{(2 k - 1) d},

(30)

where $μ_{B}$ and $ν$ are called block coherence and sub-coherence, respectively, and they are defined as

μ_{B} = max 1 \leq i < j \leq l ∥ (A [i])^{T} A [j] ∥_{2} ~{}~{}% and~{}~{} ν = max 1 \leq i \leq l μ (A [i]) .

Obviously, (30) will reduce to (17) if one lets the block sub-matrix $A [i]$ be orthonormal ⁴⁴4 This means that $(A [i])^{T} A [i] = I_{d}$ , where $I_{d}$ stands for a $d \times d$ identity matrix. for all $i \in [l]$ , namely, $ν = 0$ , and also sets the block size $d = 1$ , see, e.g., [61, 62] for more discussion on block coherence and its related theoretical investigation. Equipped with the block coherence, we here consider extending Theorem 1 to guarantee the robust recovery of such structured block-sparse signals corrupted with the bounded noise by solving the following unconstrained problem

min x \in R^{n} ∥ x ∥_{2, 1} + \frac{1}{2 λ} ∥ b - A x ∥_{2}^{2} .

(31)

This problem sometimes is called the group Lasso [63], and it can also be viewed as the block (group) extension of (5). One can find our second extension of Theorem 1 as follows.

Theorem 3.

For any $b$ observed via (3) with $∥ z ∥_{2} \leq ϵ$ , if the measurement matrix $A$ , whose every block sub-matrix $A [i]$ for $i \in [l]$ is orthonormal, satisfies

μ_{B} < \frac{1}{(2 k - 1) d}

(32)

for certain integer $k \geq 2$ , then we have

	$∥ A (x^{♯} - x) ∥_{2}$	$\leq C_{1} (β_{1}) ∥ x - x_{{k}} ∥_{2, 1} + C_{2} (β_{1}),$
	$∥ x^{♯} - x ∥_{2}$	$\leq C_{3} (β_{1}, β_{2}) ∥ x - x_{{k}} ∥_{2, 1} + C_{4} (β_{1}, β_{2}),$

where $x^{♯}$ denotes the optimal solution of (31), $x_{{k}}$ denotes the best $k$ -block approximate of $x$ , defined as

x_{{k}} = arg min ∥ y ∥_{2, 0} \leq k ∥ y - x ∥

Coherence-Based Performance Guarantee of Regularized ℓ_1-Norm Minimization and Beyond

Authors

The block mutual coherence property condition for signal recovery

Deterministic Analysis of Weighted BPDN With Partially Known Support Information

Uncertainty relations and sparse signal recovery

Convex Reconstruction of Structured Matrix Signals from Linear Measurements (I): Theoretical Results

Recovery Guarantees for Compressible Signals with Adversarial Noise

The springback penalty for robust signal recovery

Maximum-A-Posteriori Signal Recovery with Prior Information: Applications to Compressive Sensing

I Introduction

Ii Notations and Preliminaries

Ii-a Notations

Ii-B Three key lemmas

Lemma 1 ([46, 42]).

Lemma 2.

Proof:

Lemma 3.

Remark 1.

Proof:

Iii Main Results

Theorem 1.

Remark 2.

Corollary 1.

Remark 3.

Proof:

Iv Extensions

Iv-a Robust recovery via a DS regularized $ℓ_{1}$ -norm minimization

Theorem 2.

Remark 4.

Proof:

Iv-B Structure block-sparse recovery

Theorem 3.

Coherence-Based Performance Guarantee of Regularized ℓ_1-Norm Minimization and Beyond

Authors

Related Research

The block mutual coherence property condition for signal recovery

Deterministic Analysis of Weighted BPDN With Partially Known Support Information

Uncertainty relations and sparse signal recovery

Convex Reconstruction of Structured Matrix Signals from Linear Measurements (I): Theoretical Results

Recovery Guarantees for Compressible Signals with Adversarial Noise

The springback penalty for robust signal recovery

Maximum-A-Posteriori Signal Recovery with Prior Information: Applications to Compressive Sensing

This week in AI

I Introduction

Ii Notations and Preliminaries

Ii-a Notations

Ii-B Three key lemmas

Lemma 1 ([46, 42]).

Lemma 2.

Proof:

Lemma 3.

Remark 1.

Proof:

Iii Main Results

Theorem 1.

Remark 2.

Corollary 1.

Remark 3.

Proof:

Iv Extensions

Iv-a Robust recovery via a DS regularized ℓ1-norm minimization

Theorem 2.

Remark 4.

Proof:

Iv-B Structure block-sparse recovery

Theorem 3.

Iv-a Robust recovery via a DS regularized $ℓ_{1}$ -norm minimization