Convolutional Approximate Message-Passing

by   Keigo Takeuchi, et al.

This letter proposes a novel message-passing algorithm for signal recovery in compressed sensing. The proposed algorithm solves the disadvantages of approximate message-passing (AMP) and orthogonal/vector AMP, and realizes their advantages. AMP converges only in a limited class of sensing matrices while it has low complexity. Orthogonal/vector AMP requires a high-complexity matrix inversion while it is applicable for a wide class of sensing matrices. The key feature of the proposed algorithm is the so-called Onsager correction via a convolution of messages in all preceding iterations while the conventional message-passing algorithms have correction terms that depend only on messages in the latest iteration. Thus, the proposed algorithm is called convolutional AMP (CAMP). Ill-conditioned sensing matrices are simulated as an example in which the convergence of AMP is not guaranteed. Numerical simulations show that CAMP can improve the convergence property of AMP and achieve high performance comparable to orthogonal/vector AMP in spite of low complexity comparable to AMP.



page 1

page 2

page 3

page 4


Bayes-Optimal Convolutional AMP

This paper proposes Bayes-optimal convolutional approximate message-pass...

On the Convergence of Orthogonal/Vector AMP: Long-Memory Message-Passing Strategy

Orthogonal/vector approximate message-passing (AMP) is a powerful messag...

Precoding via Approximate Message Passing with Instantaneous Signal Constraints

This paper proposes a low complexity precoding algorithm based on the re...

Universality of Linearized Message Passing for Phase Retrieval with Structured Sensing Matrices

In the phase retrieval problem one seeks to recover an unknown n dimensi...

Sparse Bayesian Learning Using Approximate Message Passing with Unitary Transformation

Sparse Bayesian learning (SBL) can be implemented with low complexity ba...

A Low Complexity MAP Detector for OTFS Modulation in Logarithmic Domain

Orthogonal time-frequency space (OTFS) has been confirmed to take advant...

Approximate Message Passing with Restricted Boltzmann Machine Priors

Approximate Message Passing (AMP) has been shown to be an excellent stat...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Approximate message-passing (AMP) [1] is a low-complexity algorithm of signal recovery in compressed sensing [2, 3]. When the sensing matrix has independent and identically distributed (i.i.d.) zero-mean sub-Gaussian elements [4, 5], AMP was proved to be asymptotically Bayes-optimal in a certain region of the compression rate. However, AMP fails to converge when the sensing matrix is ill-conditioned [6] or has non-zero mean [7].

To solve this convergence issue of AMP, orthogonal AMP (OAMP) [8] and vector AMP (VAMP) [9] were proposed. OAMP and VAMP are equivalent to each other. The Bayes-optimal version of OAMP/VAMP was originally proposed by Opper and Winther [10]. OAMP/VAMP was proved to be asymptotically Bayes-optimal when the sensing matrix is orthogonally invariant [9, 11]

. However, OAMP/VAMP has high complexity unless the singular-value decomposition (SVD) of the sensing matrix can be computed efficiently.

This letter proposes a novel message-passing (MP) algorithm that solves both the convergence issue of AMP and the complexity issue of OAMP/VAMP. The proposed MP uses the same matched filter as AMP while OAMP/VAMP utilizes a linear minimum mean-square error (LMMSE) filter. Furthermore, it performs the so-called Onsager correction via a convolution of messages in all preceding iterations while AMP and OAMP/VAMP have correction terms that depend only on messages in the latest iteration. Thus, the proposed MP is called convolutional AMP (CAMP).

The tap coefficients in the convolution are determined so as to realize the asymptotic Gaussianity of the estimation errors of CAMP. For that purpose, they are defined such that a general error model proposed in

[12] contains the error model of CAMP asymptotically. Since the asymptotic Gaussianity in the general error model has been proved for any orthogonally invariant sensing matrix [12]

, the estimation errors of CAMP are asymptotically Gaussian-distributed. Numerical simulations for ill-conditioned sensing matrices show that CAMP can achieve performance comparable to OAMP/VAMP in spite of complexity comparable to AMP.

Ii Measurement Model

Consider the -dimensional linear measurements of an unknown -dimensional sparse signal vector ,


In (1), denotes a known sensing matrix. The vector is additive white Gaussian noise (AWGN) with covariance . The triple

is independent random variables. For simplicity, the signal vector

is assumed to have i.i.d. elements with zero mean and unit variance. Furthermore, the power normalization

is assumed.

An important assumption is the right-orthogonal invariance of : In the SVD , the orthogonal matrix is independent of and Haar-distributed [13]. This class of matrices contains zero-mean i.i.d. Gaussian matrices.

As an additional technical assumption, the empirical eigenvalue distribution of

converges almost surely to a deterministic distribution with a compact support in the large system limit, in which and tend to infinity while the compression rate is kept . Let denote the

th moment of the empirical eigenvalue distribution,


with . The technical assumption implies that any moment converges almost surely in the large system limit. In particular, the power normalization implies in the large system limit.

Iii Convolutional AMP

Iii-a Algorithm

The so-called Onsager correction is used to guarantee the asymptotic Gaussianity of the estimation errors before thresholding in each iteration of MP. The Onsager correction in AMP depends only on a message in the latest iteration. While AMP is a low-complexity algorithm, the Onsager correction in AMP fails to guarantee the asymptotic Gaussianity, with the only exception of zero-mean i.i.d. sensing matrices [4, 5].

The proposed CAMP has Onsager correction applicable to all right-orthogonally invariant sensing matrices. The correction term is a convolution of messages in all preceding iterations. Thus, the proposed MP is called convolutional AMP.

Let denote an estimator of in iteration  of CAMP. The estimator is recursively given by


with . In the CAMP, are a sequence of Lipschitz-continuous thresholding functions. For any function , represents the element-wise application of to a vector , i.e. . The notional convention is used in (4). The notation denotes the arithmetic mean of the elements of . The CAMP reduces to conventional AMP in the case of and for all .

To define the tap coefficients in the CAMP, consider a discrete-time dynamical system ,


where denotes the th moment (2) of the empirical eigenvalue distribution of .

In a practical implementation, the moment sequence should be replaced by the asymptotic one in the large system limit. This replacement implies that the complexity to compute can be independent of the system size if the asymptotic eigenvalue distribution of has a closed-form expression.

TABLE I: Complexity in and the number of iterations .

The computational complexity of the CAMP, AMP, and OAMP/VAMP is compared in Table I. The complexity of AMP is dominated by matrix-vector multiplication. The first term for OAMP/VAMP is the worst-case complexity of the SVD of . The second term for the CAMP is due to computation of the Onsager correction term. As long as the number of iterations  is much smaller than and , the complexity of the CAMP is comparable to that of AMP.

Iii-B State Evolution

The tap coefficients in the CAMP have been determined so as to guarantee the asymptotic Gaussianity of the estimation errors. The author [12] proposed a general error model and used state evolution (SE) to prove that the estimation error before thresholding is asymptotically Gaussian-distributed in the general error model. To prove the asymptotic Gaussianity of the estimation error before the thresholding , thus, it is sufficient to show that the error model of the CAMP is included into the general error model.

Let denote the estimation error after the thresholding. According to the definition of the general error model [12], define , , and


for , where is an abbreviation of given in (5). Then, satisfies the following equation:


with for all .


From the definitions of and , we use the SVD to have


Left-multiplying (4) by and substituting (1) and (11), we obtain


Using (9) and the definitions of and , we arrive at (10).

For and , define


When holds for all and , the general error model in [12] includes the error model of the CAMP. The following theorem implies that the inclusion is correct in the large system limit. Thus, the asymptotic Gaussianity of the estimation errors is guaranteed in the CAMP.

Theorem 1

For all and , the almost sure convergence holds in the large system limit.


The proof is by induction to show

  1. ,

  2. the almost sure convergence of to a constant,

  3. Let and for . depends on and only through .

According to [12, Theorem 1], the statement 2) follows from the statement 1). Thus, we only focus on the first and last statements. For , we use (10) to obtain , because of and .

For some , assume the three statements for all and . We shall prove the first and last statements for .

We first prove the statement 3). For and , we use (10) to obtain


where we have used the second induction hypothesis. Similarly, for we have


From the last induction hypothesis, we can define for . Using (14) and this change of variables yields


for , with . Since the right-hand sides (RHSs) depend on and only through , we find that the statement 3) holds for , and can re-write the left-hand sides of (17) and (18) as and , respectively.

Finally, we prove the statement 1). It is sufficient to prove and for all and . The proof is by induction. For , we have . Comparing (6) and yields .

For , we use (17), , and to obtain . Furthermore, we use (7), (17), and to find .

Assume and for all , and prove and . For the former statement, we use (18) and the induction hypotheses and to obtain


For the latter statement, we use (8), (18), and the induction hypothesis to find . Thus, and hold for all and . In other words, we have proved the statement 1).

Iii-C Closed-Form Solution

The sequence may be computed by solving the discrete-time dynamical systems (6)–(8) numerically when the moment sequence is given. However, it is possible to obtain a closed-form solution of the tap coefficients via the -transform of the asymptotic eigenvalue distribution of  [14], given by

Theorem 2

Let denote the generating function of with respect to , defined as


Then, is implicitly given by in the large system limit, with


Define the generating function of as


Theorem 2 follows from the following closed-form expression of :


By definition, is a polynomial of and . Thus, the numerator of (24) must be zero when the denominator is zero. The point given in (22) is a zero of the denominator for any . Thus, we let the numerator at be zero to obtain . Thus, we arrive at Theorem 2.

To complete the proof of Theorem 2, we shall prove (24). We first derive a closed-form expression of (21), given by


Substituting (8) into the last term on the RHS of (25) yields


For the first three terms, we have



. Since the Z-transform of convolution is the product of Z-transforms, the last term reduces to


Similarly, for the fourth term we have


Using these results, as well as (6) and (7), we obtain the closed-form expression


The closed-form expression (24) follows from (30). Using the -transform (20) yields


where we have used obtained from Theorem 1. Applying obtained from (6) and solving , we arrive at (24).

The following corollary implies that the CAMP reduces to conventional AMP when the sensing matrix has i.i.d. Gaussian elements with mean proportional to . Thus, the CAMP has no ability to handle this non-zero mean case.

Corollary 1

If has independent Gaussian elements with mean and variance for any , the CAMP is equivalent to conventional AMP.


The R-transform  [14, Section 2.4.2] of the asymptotic eigenvalue distribution of is given by


Using Theorem 2 and the following relationship between the and R transforms:


we obtain


where is given by (22). Substituting (22) and solving , we arrive at .

From the definition (21), we find and for all . This implies that the update rule (4) reduces to that corresponding to conventional AMP.

The following corollary is utilized in numerical simulations.

Corollary 2

If is orthogonally invariant and has non-zero singular values satisfying condition number , , and , then holds for all , with


Since holds for all , we use (20) and for any to find


where the second equality follows from for all . Using Theorem 2 yields


It is an exercise to confirm that the generating function of in Corollary 2 is equal to the sum of the second and last terms. Thus, Corollary 2 holds.

Iv Numerical Simulation

The CAMP is compared to AMP and OAMP/VAMP in terms of the mean-square error (MSE) in signal recovery. As an example of ill-conditioned sensing matrices in Corollary 2, is considered for , with denoting the th singular value in Corollary 2. The rows of are selected uniformly and randomly from the rows of the Hadamard orthogonal matrix.

We assume the Bernoulli-Gaussian (BG) prior: Each signal element takes

with probability

. Otherwise, it is sampled from the zero-mean Gaussian distribution with variance . We use the soft thresholding [1]


For the sensing matrix in Corollary 2, we have no SE results of the CAMP or AMP for designing the threshold . Thus, the threshold is fixed to a constant over all iterations, which was optimized via an exhaustive search.


Fig. 1: MSE versus the condition number  for signal density , iterations, , , and  dB.

Figure 1 shows the MSEs of the CAMP, AMP, and OAMP/VAMP estimated from independent trials. The CAMP outperforms AMP and achieves the MSEs comparable to OAMP/VAMP. The inferior performance of AMP is due to a bad convergence property of AMP. Using a large threshold improves the convergence property. Exhaustive search of implied that larger thresholds are required for AMP to converge than for the other algorithms. Thus, we conclude that CAMP improves the convergence property of AMP.


  • [1] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,” Proc. Nat. Acad. Sci., vol. 106, no. 45, pp. 18 914–18 919, Nov. 2009.
  • [2] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289–1306, Apr. 2006.
  • [3] E. J. Candés, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb. 2006.
  • [4] M. Bayati and A. Montanari, “The dynamics of message passing on dense graphs, with applications to compressed sensing,” IEEE Trans. Inf. Theory, vol. 57, no. 2, pp. 764–785, Feb. 2011.
  • [5]

    M. Bayati, M. Lelarge, and A. Montanari, “Universality in polytope phase transitions and message passing algorithms,”

    Ann. Appl. Probab., vol. 25, no. 2, pp. 753–822, Apr. 2015.
  • [6] S. Rangan, P. Schniter, A. Fletcher, and S. Sarkar, “On the convergence of approximate message passing with arbitrary matrices,” IEEE Trans. Inf. Theory, vol. 65, no. 9, pp. 5339–5351, Sep. 2019.
  • [7] F. Caltagirone, L. Zdeborová, and F. Krzakala, “On convergence of approximate message passing,” in Proc. 2014 IEEE Int. Symp. Inf. Theory, Honolulu, HI, USA, Jul. 2014, pp. 1812–1816.
  • [8] J. Ma and L. Ping, “Orthogonal AMP,” IEEE Access, vol. 5, pp. 2020–2033, Jan. 2017.
  • [9] S. Rangan, P. Schniter, and A. K. Fletcher, “Vector approximate message passing,” IEEE Trans. Inf. Theory, vol. 65, no. 10, pp. 6664–6684, Oct. 2019.
  • [10] M. Opper and O. Winther, “Expectation consistent approximate inference,” J. Mach. Learn. Res., vol. 6, pp. 2177–2204, Dec. 2005.
  • [11] K. Takeuchi, “Rigorous dynamics of expectation-propagation-based signal recovery from unitarily invariant measurements,” IEEE Trans. Inf. Theory, vol. 66, no. 1, pp. 368–386, Jan. 2020.
  • [12] ——, “A unified framework of state evolution for message-passing algorithms,” in Proc. 2019 IEEE Int. Symp. Inf. Theory, Paris, France, Jul. 2019, pp. 151–155.
  • [13] F. Hiai and D. Petz, The Semicircle Law, Free Random Variables, and Entropy.   Amer. Math. Soc., 2000.
  • [14] A. M. Tulino and S. Verdú, Random Matrix Theory and Wireless Communications.   Hanover, MA USA: Now Publishers Inc., 2004.