Coherence-Based Performance Guarantee of Regularized ℓ_1-Norm Minimization and Beyond

12/10/2018
by   Wendong Wang, et al.
Southwest University
0

In this paper, we consider recovering the signal x∈R^n from its few noisy measurements b=Ax+z, where A∈R^m× n with m≪ n is the measurement matrix, and z∈R^m is the measurement noise/error. We first establish a coherence-based performance guarantee for a regularized ℓ_1-norm minimization model to recover such signals x in the presence of the ℓ_2-norm bounded noise, i.e., z_2≤ϵ, and then extend these theoretical results to guarantee the robust recovery of the signals corrupted with the Dantzig Selector (DS) type noise, i.e., A^Tz_∞≤ϵ, and the structured block-sparse signal recovery in the presence of the bounded noise. To the best of our knowledge, we first extend nontrivially the sharp uniform recovery condition derived by Cai, Wang and Xu (2010) for the constrained ℓ_1-norm minimization model, which takes the form of μ<1/2k-1, where μ is defined as the (mutual) coherence of A, to two unconstrained regularized ℓ_1-norm minimization models to guarantee the robust recovery of any signals (not necessary to be k-sparse) under the ℓ_2-norm bounded noise and the DS type noise settings, respectively. Besides, a uniform recovery condition and its two resulting error estimates are also established for the first time to our knowledge, for the robust block-sparse signal recovery using a regularized mixed ℓ_2/ℓ_1-norm minimization model, and these results well complement the existing theoretical investigation on this model which focuses on the non-uniform recovery conditions and/or the robust signal recovery in presence of the random noise.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/11/2020

The block mutual coherence property condition for signal recovery

Compressed sensing shows that a sparse signal can stably be recovered fr...
03/03/2019

Deterministic Analysis of Weighted BPDN With Partially Known Support Information

In this paper, with the aid of the powerful Restricted Isometry Constant...
11/09/2018

Uncertainty relations and sparse signal recovery

This chapter provides a principled introduction to uncertainty relations...
10/19/2019

Convex Reconstruction of Structured Matrix Signals from Linear Measurements (I): Theoretical Results

We investigate the problem of reconstructing n-by-n structured matrix si...
07/15/2019

Recovery Guarantees for Compressible Signals with Adversarial Noise

We provide recovery guarantees for compressible signals that have been c...
10/13/2021

The springback penalty for robust signal recovery

We propose a new penalty, named as the springback penalty, for construct...
02/15/2018

Maximum-A-Posteriori Signal Recovery with Prior Information: Applications to Compressive Sensing

This paper studies the asymptotic performance of maximum-a-posteriori es...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

THE last decade has seen the burgeoning development of Compressed Sensing (CS) [1, 2] and its wide-spread applications in many fields. At the core of CS is the problem of efficiently recovering a sparse signal from a relatively small number of linear measurements. Mathematically, for any given signal , we say that it is sparse if most of its entries are zero. More specifically, if it has at most non-zero entries, i.e., , we call it a -sparse signal. In standard CS, one usually observes the linear measurements of the sparse signal via , where () is a given measurement matrix. To recover such a sparse signal, a natural idea is to search the sparsest solution among all the possible solutions. This directly leads to the following -norm minimization problem

(1)

Unfortunately, this problem is NP-hard in general, and hence it is computationally infeasible. Instead, some algorithms which aims to pursue the suboptimal solutions of (1) were proposed, see, e.g., [3, 4, 5] and their variants [6, 7, 8, 9]. Importantly, many of these algorithms have been proved to perform well under certain conditions.

Besides the above algorithm strategies, there also exist many other efficient approaches [10, 11, 12, 13, 14, 15] which can circumvent the NP-hardness of (1), and a popular one is the constrained -norm minimization method which solves

(2)

where is the

-norm of vector. Problem (

2) is convex and therefore can be well addressed by many convex optimization softwares. To theoretically investigate the equivalence between (1) and (2), one often adopts the Restricted Isometry Constant (RIC) of matrix with order, denoted by , which is defined to be the smallest value of such taht

for every -sparse vector . This notation was first proposed by Candès and Tao in [10], where they have shown that (2) is equivalent to (1) in noiselessly recovering any -sparse signals when . Subsequently many researchers were committed to improving this condition, see, e.g., [16, 17, 18, 19, 20, 21, 22, 23]. In more general application scenarios of CS, one often wishes to recover the original signal (may not be exactly sparse) from the noisy observation with

(3)

where is the unknown measurement noise/error, which directly leads to the following optimization problem

(4)

where . Obviously, (4) will reduces to (2) if one takes . It should also be noted that the above-mentioned exact recovery conditions are still available to guarantee the robust recovery of signals from (4) in the presence of noise.

On the other hand, when used in many practical applications, particularly the applications where the input data are in large scale, the constrained problem (4) is not always convenient to solve. Instead, one often solves its unconstrained counterpart, i.e., the (-norm) regularized -norm minimization problem

(5)

where is the nonnegative tradeoff parameter. This problem is also known as the Lasso estimator [24] or the Basis Pursuit DeNoising (BPDN) [25] , and it can also be solved efficiently by many algorithms, see, e.g., [26, 27, 28, 29, 30]. Recently, the relation between (4) and (5) was carefully investigated by Zhang, Yuan and Yin [31] in the context of the non-uniform recovery [32], i.e., the recovery of some specific sparse signals, for example, the sparse signals limited in a specific support. However, when it comes to the uniform recovery [31, 32], i.e., recovering all the (general) sparse signals 111In these signals, both the number and support of their non-zero entries are not known in advance., it is generally believed that (4) and (5) are not exact equivalents. As early as 2008, Zhu [33] has derived the RIC-based theoretical guarantee for (5), which states that one can robustly recover any -sparse signal through (3) with by using (5) under certain , if obeys . However, this work was relatively rarely noted by the researchers. In 2009, Bickel, Ritov and Tsybakov [34] established a RIC-like guarantee for (5) in the presence of the random noise. Recently, some new RIC conditions were obtained to ensure the robust recovery of some unconstrained -analysis approaches under the Dantzig Selector (DS) type noise/error (i.e., ), see [35, 36, 37] for details.

Some random matrices represented by sub-Gaussian random matrices are proved to have a small RIC with overwhelmingly high probability

[32]. However, when used in practical scenarios they often suffer from storage and computation limitations. Moreover, it is also NP-hard in general to find the RIC of any given matrix. To overcome these difficulties, some researchers proposed to reuse the mutual coherence, another powerful tool introduced by Mallat and Zhang [38] in their initial research on matching pursuit. In this paper, denoting by the th column of matrix , we shall define the mutual coherence of matrix as

where we assume that obeys for . Many deterministic measurement matrices in fact are designed according to the mutual coherence. There are many coherence-based theoretical results for (4), see, e.g., [39, 40, 41, 42, 43, 44]. In particular in [44], Cai, Wang and Xu have shown that any signals (not necessary to be -sparse) can be robustly recovered using (4), if satisfies , and this condition is also sharp for the noiseless recovery of any -sparse signals through (2).

As far as we know, the first (mutual) coherence-based result on (5) was given by Fuchs [41] in 2004 under the non-uniform recovery setting, which states that any fixed signal with non-zero entries (i.e., ) can be uniquely recovered from using (5), if satisfies and in (5) has been taken small enough. Later, Fuchs [45] further investigated (5) for the noisy signal recovery, and shown that if is observed through (3) with and , satisfies for certain , and is assumed to be the optimal solution of (5) under certain (related to , and ), then the support of will be either identical to, or contained in, that of . Moreover, Fuchs also shown that if similar constraints are imposed on , and , and will have their non-zero entries at the same support and with the same signs. Subsequently, Tropp [46] further extended the results in [45] to more general case. In 2010, Ben-Haim, Eldar and Elad [47] revisited (5) under the random noise, and their obtained coherence results have been proved to be better than those induced in [34]. Note that the above coherence results apply only to deal with the signals whose sparsity is known in advance. Recently, using the cumulative coherence [48] tool, Li and Chen [49] established a new uniform recovery condition for (5) to deal with the signal recovery in the presence of noise. Their results shown that if obeys , one can robustly recover any signals corrupted with the DS type noise. However, the noise they considered is based on the DS type noise rather than the often used -norm bounded noise, and the recovery condition they obtained still has much room to improve.

In this paper, equipped with the powerful mutual coherence tool, we investigate the performance guarantees of (5) and its some variants. In summary, the contributions of this paper are listed as follows:

  • We establish a tight uniform recovery condition and two relatively tight error estimates for (5), which are sufficient to guarantee the robust recovery of signals corrupted with the -norm bounded noise.

  • We extend the obtained theoretical results to guarantee the robust recovery of the signals corrupted with the DS type noise and also the structured block-sparse signal recovery in the presence of the bounded noise. To the best of our knowledge, these extended results are established for the first time under the uniform recovery setting.

The remainder of this paper is organized as follows. Section II introduces some notations and preliminaries. In Section III, we present the main results. Section IV shows two extensions. Finally, conclusion and future work are given in Section V.

Ii Notations and Preliminaries

Ii-a Notations

Throughout this paper, we denote for any given integer , and for any given index set . We also denote as a vector whose entries for and 0 otherwise, and where represents certain norm or quasi-norm. For any signal , we denote its best -term approximate as , which is defined as

Besides, for the simplicity of symbol expression we introduce the following two functions

Ii-B Three key lemmas

The proof of our main results heavily relies on the following three lemmas. We start with introducing the first lemma, which provides a RIC-like coherence result for any given matrix.

Lemma 1 ([46, 42]).

Assume that the matrix is standardized to have unit -norm. Then it holds that

(6)

for all -sparse signals .

Lemma 2.

If is observed via (3) with , then for any subset with , the optimal solution of (5) satisfies

(7)

and

(8)

where .

Proof:

Since is the optimal solution of (5), we have

which is equivalent to

(9)

As to the Left-Hand Side (LHS) of (9), we have

(10)

As to the Right-Hand Side (RHS) of (9), we know

(11)

Therefore combing (9), (10) and (II-B) directly leads to (7), and (8) follows trivially from (7). ∎

Lemma 3.

If the matrix is standardized to have unit -norm, and obeys

(12)

for certain integer , then for any vector and any subset with , it holds that

(13)

where

Remark 1.

It is easy to know from Lemma 3 that both and are two monotone increasing functions on variable . Therefore if one restricts , it will be clear that

(14)
Proof:

The proof is simple. We start with estimating the lower and upper bounds of

First, using Lemma 1, we know

(15)

Next we estimate the upper bound of . It follows from 1 that

(16)

Now, combing (II-B), (16) and the condition (12) directly leads to the desired inequality (13). ∎

Iii Main Results

With preparations above, we now present our main results.

Theorem 1.

For any observed via (3) with , if the measurement matrix , which is standardized to have unit -norm, satisfies

(17)

for certain integer , then we have

(18)

where is the optimal solution of (5), and

Remark 2.

Theorem 1 shows that one can robustly recover any signals (may not be -sparse) corrupted with the -norm bounded noise, if the measurement matrix satisfies (17). To the best of our knowledge, we first extend this sharp uniform recovery condition 222The sharp condition/bound here and throughout will refer to . It has been shown in [44] that, any -sparse signal , without exception, can be exactly recovered from (2) under this sharp condition, and there exists a matrix with obeying , and two nonzero -sparse vectors and with disjoint supports such that . derived by Cai, Wang and Xu in [44] for the constrained problem (2) to its unconstrained counterpart, i.e., the unconstrained problem (5). Similar to [36, 37], if we associate with , e.g., setting , then we get a special case of Theorem 1, and one can find this result in Corollary 1. In Remark 3, we will analyze the tightness of these two error estimates under the setting of . Besides, it is also very easy to induce some other special cases of Theorem 1 to cope with several different sparse recovery tasks. For examples, one can consider the robust recovery of any exactly sparse signals, i.e., setting the original signals to be exactly -sparse. The detailed analysis of these cases will become very similar to that of Corollary 1.

Corollary 1.

Assume that is observed via (3) with . If the measurement matrix is standardized to have unit -norm, and also satisfies (17) for certain integer , then we have

where is the optimal solution of (5), and

Remark 3.

Due to the existence of and , the coefficients are not convenient to be analyzed. Fortunately, based on the previous estimates for and , i.e., (14), we can give a rough but simple estimate for each . Specifically,

These upper bound estimates of coefficients make our recovery error, denoted by RE, have the form of

(19)

where RE stands for or , and and depend only on the value of , which characterizes the gap between the coherence of the selected measurement matrix and its sharp bound. This result also coincides with the ones obtained in [35, 36, 37, 49] for (5) in form. However, one should note that the authors in these literature focus on sparse recovery corrupted with the DS type noise, which is totally different from ours. Despite this, our obtained upper bound estimates to some degree are still better than theirs since a much tighter (or sharp) recovery condition is used. What’s more, some coefficients in these estimates can be further improved if one optimizes some inequalities used to prove Theorem 1.

Proof:

We first assume that for certain integer and denote

then using Lemma 2, Lemma 3, we have

(20)

where we used . We can known from the condition (17) that

Therefore we can further write (III) as

which implies that

This completes (18). Based on (18), (8) and (13), we have

(21)

Besides, using (III) together with (8) and (18) again, we can estimate as

(22)

On the other hand, let be the index set of the largest entries of . Then we know from [36] that

(23)

Similarly, using Lemma 3 again on index , we also have

This, together with (18) and (III), directly leads to

(24)

Now, combining (III), (22) and (III), we can estimate as follows:

which completes the proof. ∎

Iv Extensions

In this section, two extensions of Theorem 1 are discussed. They include extending Theorem 1 to guarantee the robust recovery of signals from a DS regularized -norm minimization model in the presence of the DS type noise, and that of the structured block-sparse signals from two regularized mixed -norm minimization models in the presence of the bounded noise. We start with introducing the DS regularized -norm minimization model for signal recovery in the presence of the DS type noise.

Iv-a Robust recovery via a DS regularized -norm minimization

The research on the DS type noise was initiated by Candès and Tao in [50], which aims at recovering the signals corrupted with the DS type noise by solving the following constrained problem

(25)

Many remarkable results on this problem have been obtained over the past decade, see, e.g., [34, 18, 20, 21, 22, 23, 44]. Similar to the relation of (4) and (5), a closely related problem to (25) is the following DS regularized -norm minimization problem:

(26)

Inspired by Theorem 1, we also establish a uniform recovery condition and two relatively tight error estimates for (26) to guarantee the robust signal recovery in the presence of such kind of noise, see Theorem 2 for details. This new theorem as well as Theorem 1, to the best of our knowledge, first extends the sharp uniform recovery condition obtained in [44] for (2) to its two unconstrained variants, i.e., (5) and (26), to deal with the signals corrupted with the -norm bounded noise and the DS type noise, respectively. In what follows, we present this theorem.

Theorem 2.

For any observed via (3) with , if the measurement matrix , whose columns are standardized to have unit -norm, satisfies

(27)

for certain integer , then we have

where here denotes the optimal solution of (26).

Remark 4.

In general, it is usually suggested to recover the signals corrupted with the DS type noise using the constrained problem (25), see, e.g., [50, 21, 22, 23]. Recently, some researchers proposed to deal with such kind of noise using the unconstrained problem (5), see, e.g., [35, 36, 37, 49], and they also developed a series of recovery conditions and error estimates to realize the robust recovery from (5). However, these results are far from the best. Take for example the mutual coherence condition 333Their original condition in fact is obtained under the cumulative coherence notation. However, whether this condition is sharp was not discussed by the authors. recently obtained in [49], which takes the form of

(28)

Obviously (28) is rigorously included in our sharp condition (27). In the aspect of algorithm implementation, since (26) is convex, many convex optimization softwares are available to solve it efficiently. Besides, compared to the regularization term (i.e., the second term of the objective function) in (5), the one in (26) is non-smooth and thus non-differentiable. However, if one solves (26) using some non-gradient algorithms, such as the alternating direction method and multipliers [29, 51], (26) is still comparable to (5) in terms of the algorithmic complexity.

Proof:

The proof is very similar to that of Theorem 1, and hence we here only present some technique differences. Our proof also relies on Lemma 1 and the variants of Lemma 2 and Lemma 3. One should keep in mind that the term will be replaced by . Specifically, (7), (8) and (13) are replaced in order by the following inequalities

These, as well as the skills in proving Theorem 1, are sufficient to prove Theorem 2. ∎

Iv-B Structure block-sparse recovery

Our Theorem 1 can still be extended to guarantee the robust recovery of the structured block-sparse signals. Such a kind of signals (data) arise in many applications [52, 53, 54]. We assume w.l.o.g. that there are blocks with block size in signal , and then we can write any signal as

where denotes the th block sub-vector of . If has at most non-zero blocks, i.e., , we refer to such a vector as block -sparse signal. Naturally, a block -sparse signals will reduce the traditional -sparse signal if one takes . Accordingly, we can also write any matrix as

where denotes the th block sub-matrix of . To recover such a structured block-sparse signal, Eldar amd Mishali [55] proposed solving the following mixed -norm minimization problem:

(29)

where , and they also derived a block-RIC recovery condition for (29). More improved block-RIC conditions can be found in [56, 57, 58, 59]. As early as 2010, Eldar, Kuppinger and Bölcskei [60] have generalized the traditional mutual coherence to the block setting, and show that any block -sparse signal can be exactly recovered via (29) if obeys

(30)

where and are called block coherence and sub-coherence, respectively, and they are defined as

Obviously, (30) will reduce to (17) if one lets the block sub-matrix be orthonormal 444 This means that , where stands for a identity matrix. for all , namely, , and also sets the block size , see, e.g., [61, 62] for more discussion on block coherence and its related theoretical investigation. Equipped with the block coherence, we here consider extending Theorem 1 to guarantee the robust recovery of such structured block-sparse signals corrupted with the bounded noise by solving the following unconstrained problem

(31)

This problem sometimes is called the group Lasso [63], and it can also be viewed as the block (group) extension of (5). One can find our second extension of Theorem 1 as follows.

Theorem 3.

For any observed via (3) with , if the measurement matrix , whose every block sub-matrix for is orthonormal, satisfies

(32)

for certain integer , then we have

where denotes the optimal solution of (31), denotes the best -block approximate of , defined as