I Introduction
Approximate message-passing (AMP) [1] is a low-complexity algorithm of signal recovery in compressed sensing [2, 3]. When the sensing matrix has independent and identically distributed (i.i.d.) zero-mean sub-Gaussian elements [4, 5], AMP was proved to be asymptotically Bayes-optimal in a certain region of the compression rate. However, AMP fails to converge when the sensing matrix is ill-conditioned [6] or has non-zero mean [7].
To solve this convergence issue of AMP, orthogonal AMP (OAMP) [8] and vector AMP (VAMP) [9] were proposed. OAMP and VAMP are equivalent to each other. The Bayes-optimal version of OAMP/VAMP was originally proposed by Opper and Winther [10]. OAMP/VAMP was proved to be asymptotically Bayes-optimal when the sensing matrix is orthogonally invariant [9, 11]
. However, OAMP/VAMP has high complexity unless the singular-value decomposition (SVD) of the sensing matrix can be computed efficiently.
This letter proposes a novel message-passing (MP) algorithm that solves both the convergence issue of AMP and the complexity issue of OAMP/VAMP. The proposed MP uses the same matched filter as AMP while OAMP/VAMP utilizes a linear minimum mean-square error (LMMSE) filter. Furthermore, it performs the so-called Onsager correction via a convolution of messages in all preceding iterations while AMP and OAMP/VAMP have correction terms that depend only on messages in the latest iteration. Thus, the proposed MP is called convolutional AMP (CAMP).
The tap coefficients in the convolution are determined so as to realize the asymptotic Gaussianity of the estimation errors of CAMP. For that purpose, they are defined such that a general error model proposed in
[12] contains the error model of CAMP asymptotically. Since the asymptotic Gaussianity in the general error model has been proved for any orthogonally invariant sensing matrix [12], the estimation errors of CAMP are asymptotically Gaussian-distributed. Numerical simulations for ill-conditioned sensing matrices show that CAMP can achieve performance comparable to OAMP/VAMP in spite of complexity comparable to AMP.
Ii Measurement Model
Consider the -dimensional linear measurements of an unknown -dimensional sparse signal vector ,
(1) |
In (1), denotes a known sensing matrix. The vector is additive white Gaussian noise (AWGN) with covariance . The triple
is independent random variables. For simplicity, the signal vector
is assumed to have i.i.d. elements with zero mean and unit variance. Furthermore, the power normalization
is assumed.An important assumption is the right-orthogonal invariance of : In the SVD , the orthogonal matrix is independent of and Haar-distributed [13]. This class of matrices contains zero-mean i.i.d. Gaussian matrices.
As an additional technical assumption, the empirical eigenvalue distribution of
converges almost surely to a deterministic distribution with a compact support in the large system limit, in which and tend to infinity while the compression rate is kept . Let denote theth moment of the empirical eigenvalue distribution,
(2) |
with . The technical assumption implies that any moment converges almost surely in the large system limit. In particular, the power normalization implies in the large system limit.
Iii Convolutional AMP
Iii-a Algorithm
The so-called Onsager correction is used to guarantee the asymptotic Gaussianity of the estimation errors before thresholding in each iteration of MP. The Onsager correction in AMP depends only on a message in the latest iteration. While AMP is a low-complexity algorithm, the Onsager correction in AMP fails to guarantee the asymptotic Gaussianity, with the only exception of zero-mean i.i.d. sensing matrices [4, 5].
The proposed CAMP has Onsager correction applicable to all right-orthogonally invariant sensing matrices. The correction term is a convolution of messages in all preceding iterations. Thus, the proposed MP is called convolutional AMP.
Let denote an estimator of in iteration of CAMP. The estimator is recursively given by
(3) |
(4) |
(5) |
with . In the CAMP, are a sequence of Lipschitz-continuous thresholding functions. For any function , represents the element-wise application of to a vector , i.e. . The notional convention is used in (4). The notation denotes the arithmetic mean of the elements of . The CAMP reduces to conventional AMP in the case of and for all .
To define the tap coefficients in the CAMP, consider a discrete-time dynamical system ,
(6) |
(7) |
(8) | |||||
where denotes the th moment (2) of the empirical eigenvalue distribution of .
In a practical implementation, the moment sequence should be replaced by the asymptotic one in the large system limit. This replacement implies that the complexity to compute can be independent of the system size if the asymptotic eigenvalue distribution of has a closed-form expression.
AMP | OAMP/VAMP | CAMP |
---|---|---|
The computational complexity of the CAMP, AMP, and OAMP/VAMP is compared in Table I. The complexity of AMP is dominated by matrix-vector multiplication. The first term for OAMP/VAMP is the worst-case complexity of the SVD of . The second term for the CAMP is due to computation of the Onsager correction term. As long as the number of iterations is much smaller than and , the complexity of the CAMP is comparable to that of AMP.
Iii-B State Evolution
The tap coefficients in the CAMP have been determined so as to guarantee the asymptotic Gaussianity of the estimation errors. The author [12] proposed a general error model and used state evolution (SE) to prove that the estimation error before thresholding is asymptotically Gaussian-distributed in the general error model. To prove the asymptotic Gaussianity of the estimation error before the thresholding , thus, it is sufficient to show that the error model of the CAMP is included into the general error model.
Let denote the estimation error after the thresholding. According to the definition of the general error model [12], define , , and
(9) |
for , where is an abbreviation of given in (5). Then, satisfies the following equation:
(10) | |||||
with for all .
Proof:
For and , define
(13) |
When holds for all and , the general error model in [12] includes the error model of the CAMP. The following theorem implies that the inclusion is correct in the large system limit. Thus, the asymptotic Gaussianity of the estimation errors is guaranteed in the CAMP.
Theorem 1
For all and , the almost sure convergence holds in the large system limit.
Proof:
The proof is by induction to show
-
,
-
the almost sure convergence of to a constant,
-
Let and for . depends on and only through .
According to [12, Theorem 1], the statement 2) follows from the statement 1). Thus, we only focus on the first and last statements. For , we use (10) to obtain , because of and .
For some , assume the three statements for all and . We shall prove the first and last statements for .
We first prove the statement 3). For and , we use (10) to obtain
(14) |
(15) |
where we have used the second induction hypothesis. Similarly, for we have
(16) |
From the last induction hypothesis, we can define for . Using (14) and this change of variables yields
(17) |
(18) | |||||
for , with . Since the right-hand sides (RHSs) depend on and only through , we find that the statement 3) holds for , and can re-write the left-hand sides of (17) and (18) as and , respectively.
Iii-C Closed-Form Solution
The sequence may be computed by solving the discrete-time dynamical systems (6)–(8) numerically when the moment sequence is given. However, it is possible to obtain a closed-form solution of the tap coefficients via the -transform of the asymptotic eigenvalue distribution of [14], given by
(20) |
Theorem 2
Let denote the generating function of with respect to , defined as
(21) |
Then, is implicitly given by in the large system limit, with
(22) |
Proof:
Define the generating function of as
(23) |
Theorem 2 follows from the following closed-form expression of :
(24) |
By definition, is a polynomial of and . Thus, the numerator of (24) must be zero when the denominator is zero. The point given in (22) is a zero of the denominator for any . Thus, we let the numerator at be zero to obtain . Thus, we arrive at Theorem 2.
To complete the proof of Theorem 2, we shall prove (24). We first derive a closed-form expression of (21), given by
(25) |
Substituting (8) into the last term on the RHS of (25) yields
(26) |
For the first three terms, we have
(27) |
for
. Since the Z-transform of convolution is the product of Z-transforms, the last term reduces to
(28) |
Similarly, for the fourth term we have
(29) |
Using these results, as well as (6) and (7), we obtain the closed-form expression
(30) |
The following corollary implies that the CAMP reduces to conventional AMP when the sensing matrix has i.i.d. Gaussian elements with mean proportional to . Thus, the CAMP has no ability to handle this non-zero mean case.
Corollary 1
If has independent Gaussian elements with mean and variance for any , the CAMP is equivalent to conventional AMP.
Proof:
The following corollary is utilized in numerical simulations.
Corollary 2
If is orthogonally invariant and has non-zero singular values satisfying condition number , , and , then holds for all , with
(35) |
(36) |
Iv Numerical Simulation
The CAMP is compared to AMP and OAMP/VAMP in terms of the mean-square error (MSE) in signal recovery. As an example of ill-conditioned sensing matrices in Corollary 2, is considered for , with denoting the th singular value in Corollary 2. The rows of are selected uniformly and randomly from the rows of the Hadamard orthogonal matrix.
We assume the Bernoulli-Gaussian (BG) prior: Each signal element takes
with probability
. Otherwise, it is sampled from the zero-mean Gaussian distribution with variance . We use the soft thresholding [1](39) |
For the sensing matrix in Corollary 2, we have no SE results of the CAMP or AMP for designing the threshold . Thus, the threshold is fixed to a constant over all iterations, which was optimized via an exhaustive search.
[width=]fig1.eps
Figure 1 shows the MSEs of the CAMP, AMP, and OAMP/VAMP estimated from independent trials. The CAMP outperforms AMP and achieves the MSEs comparable to OAMP/VAMP. The inferior performance of AMP is due to a bad convergence property of AMP. Using a large threshold improves the convergence property. Exhaustive search of implied that larger thresholds are required for AMP to converge than for the other algorithms. Thus, we conclude that CAMP improves the convergence property of AMP.
References
- [1] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,” Proc. Nat. Acad. Sci., vol. 106, no. 45, pp. 18 914–18 919, Nov. 2009.
- [2] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289–1306, Apr. 2006.
- [3] E. J. Candés, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb. 2006.
- [4] M. Bayati and A. Montanari, “The dynamics of message passing on dense graphs, with applications to compressed sensing,” IEEE Trans. Inf. Theory, vol. 57, no. 2, pp. 764–785, Feb. 2011.
-
[5]
M. Bayati, M. Lelarge, and A. Montanari, “Universality in polytope phase transitions and message passing algorithms,”
Ann. Appl. Probab., vol. 25, no. 2, pp. 753–822, Apr. 2015. - [6] S. Rangan, P. Schniter, A. Fletcher, and S. Sarkar, “On the convergence of approximate message passing with arbitrary matrices,” IEEE Trans. Inf. Theory, vol. 65, no. 9, pp. 5339–5351, Sep. 2019.
- [7] F. Caltagirone, L. Zdeborová, and F. Krzakala, “On convergence of approximate message passing,” in Proc. 2014 IEEE Int. Symp. Inf. Theory, Honolulu, HI, USA, Jul. 2014, pp. 1812–1816.
- [8] J. Ma and L. Ping, “Orthogonal AMP,” IEEE Access, vol. 5, pp. 2020–2033, Jan. 2017.
- [9] S. Rangan, P. Schniter, and A. K. Fletcher, “Vector approximate message passing,” IEEE Trans. Inf. Theory, vol. 65, no. 10, pp. 6664–6684, Oct. 2019.
- [10] M. Opper and O. Winther, “Expectation consistent approximate inference,” J. Mach. Learn. Res., vol. 6, pp. 2177–2204, Dec. 2005.
- [11] K. Takeuchi, “Rigorous dynamics of expectation-propagation-based signal recovery from unitarily invariant measurements,” IEEE Trans. Inf. Theory, vol. 66, no. 1, pp. 368–386, Jan. 2020.
- [12] ——, “A unified framework of state evolution for message-passing algorithms,” in Proc. 2019 IEEE Int. Symp. Inf. Theory, Paris, France, Jul. 2019, pp. 151–155.
- [13] F. Hiai and D. Petz, The Semicircle Law, Free Random Variables, and Entropy. Amer. Math. Soc., 2000.
- [14] A. M. Tulino and S. Verdú, Random Matrix Theory and Wireless Communications. Hanover, MA USA: Now Publishers Inc., 2004.