An Analysis of State Evolution for Approximate Message Passing with Side Information

I Introduction

High-dimensional linear regression is a well-studied model that has been used in many applications including compressed sensing

[1], imaging[2]

, and machine learning and statistics

[3]. The unknown signal

x \in R^{n}

is viewed through the linear model:

y = A x + w,

(1)

where $y \in R^{m}$ are the measurements, $A \in R^{m \times n}$ is a known measurement matrix, and $w \in R^{m}$

is measurement noise. The goal is to estimate the unknown signal

x

having knowledge only of the noisy measurements

y

and the measurement matrix

A

. When the problem is under-determined (i.e.,

m < n

), in order for reconstruction to be successful, it is necessary to exploit structural or probabilistic characteristics of the input signal

x

. Often a prior distribution on the input signal

x

is assumed, and in this case approximate message passing (AMP) algorithms[1] can be used for the reconstruction task.

AMP [1, 4] is a class of low-complexity algorithms for efficiently solving high-dimensional regression tasks (1

). AMP works by iteratively generating estimates of the unknown input vector,

x

, using a possibly non-linear denoiser function tailored to any prior knowledge about

x

. One favorable feature of AMP is that under some technical conditions on the measurement matrix

A

and

x

, the observations at each iteration of the algorithm are almost surely equal in distribution to

x

plus independent and identically distributed (i.i.d.) Gaussian noise in the large system limit.

AMP with Side Information (AMP-SI): In information theory [5], when different communication systems share side information (SI), overall communication can become more efficient. Recently [6, 7], a novel algorithmic framework, referred to as AMP-SI, has been introduced for incorporating SI into AMP for high-dimensional regression tasks (1). AMP-SI has been empirically demonstrated to have good reconstruction quality and is easy to use. For example, we have proposed to use AMP-SI for channel estimation in emerging millimeter wave communication systems [8], where the time dynamics of the channel structure allow previous channel estimates to be used as SI when estimating the current channel structure [7].

We model each entry of the observed SI, denoted by $˜ x \in R^{n}$ , as depending statistically on the corresponding entry of the unknown signal $x$

through some joint probability density function (pdf),

f (X, ˜ X)

. AMP-SI uses a conditional denoiser,

η_{t} : R^{2} \to R

, to incorporate SI,

η_{t} (a, b) = E [X | X + λ_{t} Z = a, ˜ X = b] .

(2)

The AMP-SI algorithm iteratively updates estimates of the input signal $x$ : let $x^{0} = 0$ , the all-zeros vector, then

	$r^{t}$	$= y - A x^{t} + \frac{r^{t - 1}}{δ} ⟨ η_{t - 1}^{'} (x^{t - 1} + A^{T} r^{t - 1}, ˜ x) ⟩,$		(3)
	$x^{t + 1}$	$= η_{t} (x^{t} + A^{T} r^{t}, ˜ x),$		(4)

where $x^{t} \in R^{n}$ is the estimate of input signal at iteration t, and the denoiser in (2) is applied entry-wise to vector inputs. The derivative $η_{t}^{'} (s, .) = \frac{\partial}{\partial s} η_{t} (s, .)$ is with respect to the first input, and $⟨ v ⟩$ is the empirical average of a vector $v$ , i.e., $⟨ v ⟩ = \frac{1}{n} \sum_{i = 1}^{n} v_{i}$ . Using the denoiser in (2), the AMP-SI algorithm (3)-(4) provides the minimum mean squared error (MMSE) estimate of the signal when SI $˜ X$ is available [6].

State Evolution (SE): It has been proven that the performance of AMP, as measured, for example, by the normalized squared $ℓ_{2}$ -error $\frac{1}{n} | | x^{t} - x | |_{2}^{2}$ between the estimate $x^{t}$ and true signal $x$ , can be accurately predicted by a scalar recursion referred as SE[9, 10] when the measurement matrix A is i.i.d. Gaussian under various assumptions on the elements of the signal. The SE equation for AMP-SI is as follows. Assume the entries of the noise $w$ are i.i.d. $\sim f (W)$ with $σ_{w}^{2} = E [W^{2}]$ , and let $λ_{0} = σ_{w}^{2} + E [X^{2}] / δ$ . Then for $t \geq 0$ ,

λ_{t}^{2} = σ_{w}^{2} + \frac{1}{δ} E [{(η_{t - 1} (X + λ_{t - 1} Z, ˜ X) - X)}_{t - 1}^{2}],

(5)

where $(X, ˜ X) \sim f (X, ˜ X)$ are independent of $Z \sim N (0, 1)$ , where we use $N (μ, σ^{2})$

to denote a Gaussian distribution with mean

μ

and variance

σ^{2}

Considering AMP-SI (3)-(4), however, we cannot directly apply the existing AMP theoretical results [9, 10], as the conditional denoiser (2) depends on the index $i$ through the SI, meaning that different scalar denoisers will be used at different indices within the AMP-SI iterations. Recent results [11], however, extend the asymptotic SE analysis to a larger class of possible denoisers, allowing, for example, each element of the input to use a different non-linear denoiser as is the case in AMP-SI. We employ these results to rigorously relate the SE presented in (5) to the AMP-SI algorithm in (3)-(4).

Related Work:

While integrating SI into reconstruction algorithms is not new, AMP-SI introduces a unified framework within AMP supporting arbitrary signal and SI dependencies. Prior work using SI has been either heuristic, limited to specific applications, or outside the AMP framework.

For example, Wang and Liang [12] integrate SI into AMP for a specific signal prior density, but the method is difficult to apply to other signal models. Ziniel and Schniter [13] develop an AMP-based reconstruction algorithm for a time-varying signal model based on Markov processes for the support and amplitude. This signal model is easily incorporated into the AMP-SI framework as discussed in the analysis of the birth-death-drift model of [6, 7]. Manoel et al. implement an AMP-based algorithm in which the input signal is repeatedly reconstructed in a streaming fashion, and information from past reconstruction attempts is aggregated into a prior, thus improving ongoing reconstruction results [14]. This reconstruction scheme resembles that of AMP-SI, in particular when the Bernoulli-Gaussian model is used (see Section II-B).

Contribution and Outline: Ma et al. use numerical experiments to show that SE (5) accurately tracks the performance of AMP-SI (3)-(4) [7], as was shown rigorously for standard AMP. Ma et al. conjecture that rigorous theoretical guarantees can be given for AMP-SI as well [7]. In this work, we analyze AMP-SI performance when the input signal and SI are drawn i.i.d. according to a general pdf $f (X, ˜ X)$ obeying some finite moment conditions, the AMP-SI denoiser (2) is Lipschitz, and the measurement matrix $A$ is i.i.d. Gaussian.

In Section II, we give the main results, examples for various signal and SI models, and numerical experiments comparing the empirical performance of AMP-SI and the SE predictions. The proof of our main theorem is provided in Section III.

Ii Main Results

Ii-a Main Theorem

Our main result provides AMP-SI performance guarantees when considering pseudo-Lipschitzloss functions, which we define in the following.

Definition II.1.

Pseudo-Lipschitz functions [9]: For $k \in N_{> 0}$ and any $n \in N_{> 0}$ , a function $ϕ : R^{n} \to R$ is pseudo-Lipschitz of order $k$ if there exists a constant $L$ , referred to as the pseudo-Lipschitz constant of $ϕ$ , such that for any $x, y \in R^{n}$ ,

| ϕ (x) - ϕ (y) | \leq L (1 + \frac{| | x | |^{k - 1}}{\sqrt{n}} + \frac{| | y | |^{k - 1}}{\sqrt{n}}) \frac{| | x - y | |}{\sqrt{n}} .

(6)

For $k = 1$ , this definition coincides with the standard definition of a Lipschitz function. Throughout this work, $| | \cdot | |$ denotes the Euclidean norm.

We are now ready to state our main result. Throughout the paper we let $p =$ denote convergence in probability.

Theorem II.1.

For any order $k$ pseudo-Lipschitz functions $ϕ : R^{2} \to R$ and $ψ : R^{3} \to R$ , assume the following.

The measurement matrix $A$ has i.i.d. Gaussian entries with mean $0$ and variance $1 / m$ .
The noise $w$ is i.i.d. $\sim f (W)$ with finite $E [| W |^{k}]$ .
The signal and SI $(X, ˜ X)$ are sampled i.i.d. from $f (X, ˜ X)$ with finite $E [| X |^{k}]$ , finite $E [| ˜ X |^{k}]$ , and finite $E [| X ˜ X |]$ .
For $t \geq 0$ , the denoisers $η_{t} (\cdot, \cdot)$ defined in (2) are Lipschitz continuous, meaning for scalars $a_{1}, a_{2}, b_{1}, b_{2}$ , and constant $L > 0$ ,

$| η_{t} (a_{1}, b_{1}) - η_{t} (a_{2}, b_{2}) | \leq L | | (a_{1}, b_{1}) - (a_{2}, b_{2}) | | .$

Then,

		$lim m \frac{1}{m} m \sum i = 1 ϕ (r_{i}^{t}, w_{i}) p = E [ϕ (W + \sqrt{λ_{t}^{2} - σ_{w}^{2}} Z_{1}, W)],$		(7)
		$lim n \frac{1}{n} n \sum i = 1 ψ (x_{i}^{t} + [A^{T} r^{t}]_{i}, x_{i}, {˜ x}_{i}) p = E [ψ (X + λ_{t} Z_{2}, X, ˜ X)],$		(7)

where $Z_{1}, Z_{2}$ are standard Gaussians, independent of $W \sim f (W)$ and $(X, ˜ X) \sim f (X, ˜ X)$ . In the above, $x^{t}$ and $r^{t}$ are defined in the AMP-SI recursion (3)-(4), and $λ_{t}$ in the SE (5).

Section III contains the proof of Theorem II.1. The proof follows from Berthier et al. [11, Theorem 14]

and the strong law of large numbers. The main technical details involve showing that our assumptions

(A1) - (A4)

are enough to satisfy the assumptions needed for [11, Theorem 14]. The details are given in Section III.

As a concrete example of how Theorem II.1 provides performance guarantees for AMP-SI, let us consider a few interesting pseudo-Lipschitz loss functions.

Corollary II.1.1.

Under assumptions $(A1) - (A4)$ , letting $ψ^{1} : R^{3} \to R$ be the $ℓ_{2}$ loss, $ψ^{1} (x, y, z) = (x - y)^{2}$ , then by Theorem II.1,

lim n \to \infty \frac{1}{n} | | x^{t} + A^{T} r^{t} - x | |^{2} p = λ_{t}^{2},

where $λ_{t}^{2}$ is defined in (5). Similarly if $ψ^{2} : R^{3} \to R$ is defined as $ψ^{2} (x, y, z) = (η_{t} (x, z) - y)^{2}$ , then by Theorem II.1

lim n \to \infty \frac{1}{n} | | x^{t + 1} - x | |^{2} p = δ (λ_{t}^{2} - σ_{w}^{2}) .

Ii-B Examples

Next, we consider a few signal and SI models to show how one can derive the denoiser in (2), use this to construct the AMP-SI algorithm and the SE, and apply Theorem II.1. Before we get to the examples we state a lemma that allows us know about how functions with bounded derivative are Lipschitz.

Lemma II.2.

A function $ϕ : R^{2} \to R$ having bounded derivatives, $0 < D_{1}, D_{2} < \infty,$

is Lipschitz continuous with Lipschitz constant $\sqrt{D_{1}^{2} + D_{2}^{2}}$ .

Proof.

The result follows using the Triangle Inequality and Cauchy-Schwarz,

\begin{matrix} | ϕ (x_{1}, y_{1}) - ϕ (x_{2}, y_{2}) | = | ϕ (x_{1}, y_{1}) - ϕ (x_{1}, y_{2}) + ϕ (x_{1}, y_{2}) - ϕ (x_{2}, y_{2}) | \leq | ϕ (x_{1}, y_{1}) - ϕ (x_{1}, y_{2}) | + | ϕ (x_{1}, y_{2}) - ϕ (x_{2}, y_{2}) | \leq D_{2} | y_{1} - y_{2} | + D_{1} | x_{1} - x_{2} | \leq \sqrt{D_{2}^{2} + D_{1}^{2}} \sqrt{(y_{1} - y_{2})^{2} + (x_{1} - x_{2})^{2}} = \sqrt{D_{2}^{2} + D_{1}^{2}} | | (x_{1}, y_{1}) - (x_{2}, y_{2}) | | . \end{matrix}

(8)

∎

Ii-B1 Gaussian-Gaussian Signal and SI

In this model, referred to as the GG model henceforth, the signal has i.i.d. Gaussian entries with zero mean and finite variance and we have access to SI in the form of the signal with additive white Gaussian noise (AWGN). The signal, $X$ , and SI, $˜ X$ , are related by

˜ X = X + N (0, σ^{2} I) .

(9)

In this case, the AMP-SI denoiser (2) equals [7]

	$η_{t} (a, b)$	$= E [X ∣ ∣ X + λ_{t} Z = a, ˜ X = b]$		(10)
		$= \frac{σ_{x}^{2} σ^{2} a + σ_{x}^{2} λ_{t}^{2} b}{σ_{x}^{2} (σ^{2} + λ_{t}^{2}) + σ^{2} λ_{t - 1}^{2}} .$		(10)

Then the SE (5) can be computed as

(11)

We note that the denoiser in (10) is Lipschitz continuous as a result of Lemma II.2 because

∣ ∣ \frac{\partial}{\partial a} η_{t} (a, b) ∣ ∣ = ∣ ∣ \frac{σ_{x}^{2} σ^{2}}{σ_{x}^{2} (σ^{2} + λ_{t}^{2}) + σ^{2} λ_{t}^{2}} ∣ ∣ \leq 1,

and

∣ ∣ \frac{\partial}{\partial b} η_{t} (a, b) ∣ ∣ = ∣ ∣ \frac{σ_{x}^{2} λ_{t}^{2}}{σ_{x}^{2} (σ^{2} + λ_{t}^{2}) + σ^{2} λ_{t}^{2}} ∣ ∣ \leq 1,

and therefore the assumptions $(A1) - (A5)$ are satisfied in the GG case and we can apply Thoerem II.1.

Ii-B2 Bernoulli-Gaussian Signal and SI

The Bernoulli-Gaussian (BG) model reflects scenario in which one wishes to recover a sparse signal and has access to SI in the form of the signal with AWGN as in (9). In this model, each entry of the signal is independently generated according to $x_{i} \sim ϵ N (0, 1) + (1 - ϵ) δ_{0}$ , where $δ_{0}$ is the Dirac delta function at $0$ . In words, the entries of the signal independently take the value $0$ with probability $1 - ϵ$ and are $N (0, 1)$ with probability $ϵ$ . In this case, the AMP-SI denoiser (2) equals [7]

$η_{t} (a, b)$	$= E [X ∣ ∣ X + λ_{t} Z = a, ˜ X = b]$	(12)
	$= Pr (X \neq 0 \| a, b) E [X \| a, b, X \neq 0]$
	$= Pr (X \neq 0 \| a, b) \frac{σ^{2} a + λ_{t}^{2} b}{σ^{2} + λ_{t}^{2} + σ^{2} λ_{t}^{2}},$

where, letting $ρ_{τ^{2}} (x)$ be the zero-mean Gaussian density with variance $τ^{2}$ evaluated at $x$ , and defining $ν_{t} := σ^{2} λ_{t}^{2} (σ^{2} + λ_{t}^{2} + σ^{2} λ_{t}^{2})$ ,

(13)

where we denote

\begin{matrix} T_{a, b} & := \frac{(1 - ϵ) ρ_{λ_{t}^{2}} (a) ρ_{σ^{2}} (b)}{ϵ ρ_{1 + σ^{2}} (b) ρ_{\frac{σ^{2}}{1 + σ^{2}} + λ_{t}^{2}} (\frac{b}{1 + σ^{2}} - a)} = (\frac{1 - ϵ}{ϵ}) \sqrt{\frac{σ^{2} + λ_{t}^{2} + σ^{2} λ_{t}^{2}}{λ_{t}^{2} σ^{2}}} exp {\frac{- (σ^{2} a + λ_{t - 1}^{2} b)^{2}}{2 σ^{2} λ_{t - 1}^{2} (σ^{2} + λ_{t}^{2} + σ^{2} λ_{t}^{2})}} = (\frac{1 - ϵ}{ϵ}) \frac{ν_{t} \sqrt{2 π}}{λ_{t}^{2} σ^{2}} ρ_{ν_{t}} (σ^{2} a + λ_{t}^{2} b), \end{matrix}

(14)

Then the SE (5) can be computed as

(15)

We again use Lemma II.2 to show that the denoiser defined in (12) and (13) is Lipschitz continuous so that the assumptions $(A1) - (A5)$ are satisfied in the BG case and we can apply Thoerem II.1. We study the partial derivatives. Denote

f_{a, b} := \frac{σ^{2} a + λ_{t}^{2} b}{σ^{2} + λ_{t}^{2} + σ^{2} λ_{t}^{2}} .

(16)

Combining (13) and (14) and (16),

η_{t} (a, b)

= (1 + T_{a, b})^{- 1} f_{a, b} .

Then,

(17)

Now we show upperbounds for the two terms of (17) separately. For the first term, we see that $\frac{\partial f_{a, b}}{\partial a} \leq 1$ , so

\frac{(1 + 2 T_{a, b})}{(1 + T_{a, b})^{2}} ∣ ∣ \frac{\partial f_{a, b}}{\partial a} ∣ ∣ \leq 1.

Now we consider the second term of

Consider the second term of (17). First we note that

Then from (14) and (16),

\begin{matrix} T_{a, b} f_{a, b} & = (\frac{1 - ϵ}{ϵ}) \sqrt{2 π} (σ^{2} a + λ_{t}^{2} b) ρ_{ν_{t}} (σ^{2} a + λ_{t}^{2} b), \end{matrix}

then using that $\frac{\partial}{\partial x} ρ_{τ^{2}} (x) = - \frac{x}{τ^{2}} ρ_{τ^{2}} (x)$ , we have

(18)

To upper bound the above, we use $exp {- x} \leq \frac{1}{1 + x}$ when $x \geq 0$ , and so

ρ_{τ^{2}} (x) = \frac{1}{\sqrt{2 π τ^{2}}} exp {\frac{- x^{2}}{2 τ^{2}}} \leq \sqrt{\frac{2}{π}} (\frac{τ}{2 τ^{2} + x^{2}}) .

Using this in (18), we find

\begin{matrix} ∣ ∣ \frac{\partial}{\partial a} [T_{a, b} f_{a, b}] ∣ ∣ & \leq \frac{2 σ^{2}}{\sqrt{ν_{t}}} (\frac{1 - ϵ}{ϵ}) \frac{| ν_{t} - (σ^{2} a + λ_{t}^{2} b)^{2} |}{2 ν_{t} + (σ^{2} a + λ_{t}^{2} b)^{2}} \leq \frac{2 σ^{2}}{\sqrt{ν_{t}}} (\frac{1 - ϵ}{ϵ}) \leq \frac{2 (1 - ϵ)}{σ_{w} ϵ}, \end{matrix}

(19)

where in the final inequality we use $λ_{t} \geq σ_{w}$ by (15), and

	$\frac{σ^{2}}{\sqrt{ν_{t}}}$	$= \frac{σ}{λ_{t} \sqrt{σ^{2} + λ_{t}^{2} + σ^{2} λ_{t}^{2}}}$		(20)
		$= \frac{1}{λ_{t} \sqrt{1 + \frac{λ_{t}^{2}}{σ^{2}} + λ_{t}^{2}}} \leq \frac{1}{λ_{t}} .$		(20)

Using the above in (17), we have

∣ ∣ \frac{\partial}{\partial a} η_{t} (a, b) ∣ ∣ \leq 1 + \frac{2 (1 - ϵ)}{σ_{w} ϵ} .

As in (17) we can show

\begin{matrix} ∣ ∣ \frac{\partial}{\partial b} η_{t} (a, b) ∣ ∣ & \leq \frac{(1 + 2 T_{a, b})}{(1 + T_{a, b})^{2}} ∣ ∣ [\frac{\partial}{\partial b} f_{a, b}] ∣ ∣ + \frac{1}{(1 + T_{a, b})^{2}} ∣ ∣ T_{a, b} [\frac{\partial}{\partial b} f_{a, b}] + [\frac{\partial}{\partial b} T_{a, b}] f_{a, b} ∣ ∣, \end{matrix}

Then,

\begin{matrix} \frac{(1 + 2 T_{a, b})}{(1 + T_{a, b})^{2}} ∣ ∣ \frac{\partial}{\partial b} f_{a, b} ∣ ∣ & \leq 1, \end{matrix}

and a bound as in (18) - (19) gives

\begin{matrix} \frac{1}{(1 + T_{a, b})^{2}} & ∣ ∣ T_{a, b} [\frac{\partial}{\partial b} f_{a, b}] + [\frac{\partial}{\partial b} T_{a, b}] f_{a, b} ∣ ∣ \leq ∣ ∣ \frac{\partial}{\partial b} [T_{a, b} f_{a, b}] ∣ ∣ \leq \frac{2 λ_{t}^{2}}{\sqrt{ν_{t}}} (\frac{1 - ϵ}{ϵ}) \frac{| ν_{t} - (σ^{2} a + λ_{t}^{2} b)^{2} |}{2 ν_{t} + (σ^{2} a + λ_{t}^{2} b)^{2}} \leq \frac{2 λ_{t}^{2}}{\sqrt{ν_{t}}} (\frac{1 - ϵ}{ϵ}) \leq \frac{2 (1 - ϵ)}{σ ϵ} . \end{matrix}

Ii-C Numerical Examples

Finally, we provide numerical results to compare the empirical mean square error (MSE) performance of AMP-SI and the performance predicted by SE. Fig. 1 shows the MSE achieved by AMP-SI in the GG scenario and the SE prediction of its performance. In this example, the signal variance $σ_{x}^{2} = 1$ , the measurement noise variance $σ_{w}^{2} = 0.01$ , the variance of AWGN in SI $σ^{2} = 0.04$ . We averaged over 10 trials of a GG recovery problem for empirical results of AMP-SI. The comparison in Fig. 1(a), Fig. 1(b) and Fig. 1(c) given by three different signal length. For smaller $n$ there is some gap between the empirical MSE and the SE prediction, as shown in Fig. 1 for $n = 100$ , but the gap shrinks as $n$ is increased. The results show the empirical MSE tracks the SE prediction nicely.

Fig. 2 shows the MSE achieved by AMP-SI in the BG scenario, and the SE prediction of its performance. We again averaged over 10 trials of a BG recovery problem for empirical results of AMP-SI. The signal length $n = 10000$ , $m = 3000$ , the measurement noise variance $σ_{w}^{2} = 0.01$ , and $ϵ = 0.2$ , where $20 %$ of the entries in the signal are nonzero. We vary the variance of AWGN in SI from $σ^{2} = 0.04$ , $σ^{2} = 0.25$ , and $σ^{2} = 1$ . The results show that SE can predict the MSE achieved by AMP-SI at every iteration.

Fig. 2: Empirical MSE performance of AMP-SI and SE prediction. (BG model, $n = 10000$ , $m = 3000$ , $ϵ = 0.2$ , $σ_{w} = 0.1$ .)

Iii Proof of Theorem ii.1

Proof of The proof Theorem II.1 contains two steps. In the first step we use Berthier et al. [11, Theorem 14] and in the second step we make an appeal to the strong law of large numbers (SLLN). We remind the reader of the strong law:

Definition III.1.

Strong Law of Large Numbers [15]: Let $X_{1}, X_{2}, . . .$

be a sequence of i.i.d. random variables with finite mean

μ

. Then

Pr = (lim n \to \infty \frac{1}{n} (X_{1} + X_{2} + . . . + X_{n}) = μ) = 1,

(21)

In words, the partial averages $\frac{1}{n} (X_{1} + X_{2} + . . . + X_{n})$ converge almost surely to $μ < \infty$ .

Iii-a Step 1

We will make use of Berthier et al. [11, Theorem 14], restated here for convenience. Before proceeding we include a definition of uniformly pseudo-Lipschitz functions, that generalizes the ideal of pseudo-Lipschitz functions given in Definition II.1.

Definition III.2.

Uniformly pseudo-Lipschitz functions [11]: A sequence (in $n$ ) of pseudo-Lipschitz functions ${ϕ_{n}}_{n \in N_{> 0}}$ is called uniformly pseudo-Lipschitz of order $k$ if, denoting by $L_{n}$ is the pseudo-Lipschitz constant of order $k$ of $ϕ_{n}$ , we have $L_{n} < \infty$ for each $n$ and $lim {sup}_{n \to \infty} L_{n} < \infty$ .

Berthier et al. [11, Theorem 14] requires the following assumptions:

The measurement matrix $A$ has Gaussian entries with i.i.d. mean $0$ and variance $1 / m$ .
Define a sequence of denoisers ${˜ η}_{n}^{t} : R^{n} \to R^{n}$ to be those that apply the denoiser $η_{t}$ defined in (2) element-wise on vector input: ${˜ η}_{n}^{t} (X) := η_{t} (X, ˜ X) .$ For each $t$ , ${˜ η}_{n}^{t} (\cdot)$ are uniformly Lipschitz. A function is uniformly Lipschitz in $n$ if the Lipschitz constant does not depend on $n$ .
$| | x | |_{2}^{2} / n$ converges to a constant as $n \to \infty$ .
The limit $σ_{w} = {lim}_{m \to \infty} | | w | |_{2} / \sqrt{m}$ is finite.
For any iterations $s, t \in N$ and for any $2 \times 2$ covariance matrix $Σ$ , the following limits exist.

$lim n \to \infty \frac{1}{n} n \sum i = 1 E_{Z} [x_{i} η_{t} (x_{i} + Z_{i}, {˜ x}_{i})] < \infty,$

$lim n \to \infty \frac{1}{n} n \sum i = 1 E_{Z, Z^{'}} [η_{t} (x_{i} + Z_{i}, {˜ x}_{i}) η_{s} (x_{i} + Z_{i}^{'}, {˜ x}_{i})] < \infty,$

where $(Z, Z^{'}) \sim N (0, Σ \otimes I_{n})$ , with $\otimes$
denoting the tensor product and
$I_{n}$
the identity matrix.

Theorem III.1.

Under the assumptions $(C1) - (C5)$ , for any sequences of uniformly pseudo-Lipschitz functions $ϕ_{n} : R^{m} \times R^{m} \to R$ and $ψ_{n} : R^{n} \times R^{n} \to R$ ,

		$lim m (ϕ_{m} (r^{t}, w) - E_{Z_{1}} [ϕ_{m} (w + \sqrt{λ_{t}^{2} - σ_{w}^{2}} Z_{1}, w)]) p = 0,$
		$lim n (ψ_{n} (x^{t} + A^{T} r^{t}, x) - E_{Z_{2}} [ψ_{n} (x + λ_{t} Z_{2}, x)]) p = 0,$

where $Z_{1} \sim N (0, I_{m})$ , $Z_{2} \sim N (0, I_{n})$ , $x^{t}$ and $r^{t}$ are defined in the AMP-SI recursion (3)-(4), and $λ_{t}$ in the SE (5).

Now we demonstrate that our assumptions $(A1) - (A4)$ stated in Section II are enough to satisfy the assumptions $(C1) - (C6)$ needed to apply Theorem III.1.

Assumptions (A1) and (C1) are identical. We will show that (C2) follows from (A4), (C4) follows from (A2), and (C3) follows from (A3). Finally we show (C5) and (C6) follow from (A3) and (A4).

First consider assumption (C2). The non-separable denoiser ${˜ η}_{n}^{t} (X) = η_{t} (X, ˜ X)$ applies the AMP-SI denoiser defined in (2) entrywise to its vector inputs. From (A4), ${η_{t} (\cdot, \cdot)}_{t \geq 0}$ are Lipschitz continuous. Thus, for length- $n$ vectors $x_{1}, x_{2}$ , and fixed SI $˜ x$ ,

\begin{matrix} | | {˜ η}_{n}^{t} (x_{1}) - {˜ η}_{n}^{t} (x_{2}) | |^{2} = n \sum i = 1 (η_{t} ([x_{1}]_{i}, {˜ x}_{i}) - η_{t} ([x_{2}]_{i}, {˜ x}_{i}))^{2} \leq n \sum i = 1 L^{2} ([x_{1}]_{i} - [x_{2}]_{i})^{2} = L^{2} | | x_{1} - x_{2} | |^{2}, \end{matrix}

and so

| | {˜ η}_{n}^{t} (x_{1}) - {˜ η}_{n}^{t} (x_{2}) | | \leq L | | x_{1} - x_{2} | | .

The Lipschitz constant does not depend on $n$ , so ${˜ η}_{n}^{t} (\cdot)$ is uniformly Lipschitz.

Now consider assumption (C4). From (A2), the measurement noise $w$ in (1) has i.i.d. entries with zero-mean and finite $E [| W |^{k}]$ . Then applying Definition III.1,

lim m \to \infty \frac{| | w | |_{2}^{2}}{m} = lim m \to \infty \frac{1}{m} m \sum i = 1 w_{i}^{2} = σ_{w}^{2} < \infty,

where we have used that $σ_{w}^{2} < \infty$ follows from $E [| W |^{k}] < \infty$ for $k \geq 2$ . The proof of (C3) similarly follows using the SLLN and the finiteness of $E [| X |^{k}]$ given in assumption (A3).

We now show that (C5) is met. Recall $Z \sim N (0, σ_{z}^{2} I_{n})$ . Define $y_{i} := x_{i} E_{Z} [η_{t} (x_{i} + Z_{i}, {˜ x}_{i})]$ for $i = 1, 2, \dots, n$ . By assumption (A3), the signal and side information $(X, ˜ X)$ are sampled i.i.d. from the joint density $f (X, ˜ X)$ . It follows that $y_{1}, y_{2}, \dots, y_{n}$ are also i.i.d., so by Definition III.1 if $E [X η_{t} (X + Z, ˜ X)] < \infty$ where $Z \sim N (0, σ_{z}^{2})$ independent of $(X, ˜ X) \sim f (X, ˜ X)$ , then

lim n \to \infty \frac{1}{n} n \sum i = 1 x_{i} E_{Z} [η_{t} (x_{i} + Z_{i}, {˜ x}_{i})] = E [X η_{t} (X + Z, ˜ X)],

We will now show that $E [X η_{t} (X + Z, ˜ X)] < \infty$ .

First note that (A4) assumes $η_{t} (\cdot, \cdot)$ is Lipschitz, meaning for scalars $a_{1}, a_{2}, b_{1}, b_{2}$ and some constant $L > 0$ ,

	$\| η_{t} (a_{1}, b_{1}) - η_{t} (a_{2}, b_{2}) \| \leq L \| \| (a_{1}, b_{1}) - (a_{2}, b_{2}) \| \|$
	$\leq L \| a_{1} - a_{2} \| + L \| b_{1} - b_{2} \| .$

Therefore letting $a_{2} = b_{2} = 0$ we have

| η_{t} (a_{1}, b_{1}) | - | η_{t} (0, 0) | \leq | η_{t} (a_{1}, b_{1}) - η_{t} (0, 0) | \leq L | a_{1} | + L | b_{1} |,

giving the follows upper bound for constant $L^{'} > 0$ ,

| η_{t} (a_{1}, b_{1}) | \leq L^{'} (1 + | a_{1} | + | b_{1} |) .

(22)

Now using (22) and the triangle inequality,

		$E [X η_{t} (X + Z, ˜ X)] \leq L^{'} E [\| X \| (1 + \| X + Z \| + \| ˜ X \|)]$		(23)
		$\leq L^{'} (E \| X \| + E [X^{2}] + E \| X \| E \| Z \| + E \| X ˜ X \|) .$		(23)

Finally, by assumption (A3) we have that $E [| X |^{k}], E [| ˜ X |^{k}]$ and $E | X ˜ X |$ are all finite. Then noting that for any random variable, $Y$ , we have $| Y |^{r} \leq 1 + | Y |^{k}$ for $1 \leq r \leq k$ , meaning $E [| Y |]^{r} < 1 + E [| Y |^{k}]$ the boundednes of $E [X η_{t} (X + Z, ˜ X)]$ follows from (23) with assumption (A3).

The proof of (C6) follows similarly to the proof of (C5). Recall $(Z, Z^{'}) \sim N (0, Σ \otimes I_{n})$ . Define $y_{i} := E_{Z, Z^{'}} [η_{t} (x_{i} + Z_{i}, {˜ x}_{i}) η_{s} (x_{i} + Z_{i}^{'}, {˜ x}_{i})]$ for $i = 1, 2, \dots, n$ . By assumption (A3), the signal and side information $(X, ˜ X)$ are sampled i.i.d. from the joint density $f (X, ˜ X)$ . It follows that $y_{1}, y_{2}, \dots, y_{n}$ are also i.i.d., so by Definition III.1 if $E [η_{t} (X + Z, ˜ X) η_{s} (X + Z^{'}, ˜ X)] < \infty$ where $Z \sim N (0, σ_{z}^{2})$ and $Z^{'} \sim N (0, σ_{z^{'}}^{2})$ , independent of $(X, ˜ X) \sim f (X, ˜ X)$ , then

	$lim n \to \infty \frac{1}{n} n \sum i = 1 E_{Z, Z^{'}} [η_{t} (x_{i} + Z_{i}, {˜ x}_{i}) η_{s} (x_{i} + Z_{i}^{'}, {˜ x}_{i})]$
	$= E [η_{t} (X + Z, ˜ X) η_{s} (X + Z^{'}, ˜ X)] .$

We will now show that $E [η_{t} (X + Z, ˜ X) η_{s} (X + Z^{'}, ˜ X)] < \infty$ . Using the bound (22),

		$E [η_{t} (X + Z, ˜ X) η_{s} (X + Z^{'}, ˜ X)]$
		$\leq E [\| η_{t} (X + Z, ˜ X) \| \| η_{s} (X + Z^{'}, ˜ X) \|]$
		$\leq {L^{'}}^{2} E [(1 + \| X + Z \| + \| ˜ X \|) (1 + \| X + Z^{'} \| + \| ˜ X \|)] .$

Then using the triangle inequality,

		$E [(1 + \| X + Z \| + \| ˜ X \|) (1 + \| X + Z^{'} \| + \| ˜ X \|)]$		(24)
		$\leq E [(1 + \| X \| + \| Z \| + \| ˜ X \|) (1 + \| X \| + \| Z^{'} \| + \| ˜ X \|)]$
		$= 1 + 2 E [\| X \|] + 2 E [\| ˜ X \|] + 2 E [\| X \| \| ˜ X \|] + E [X^{2}] + E [{˜ X}^{2}]$
		$+ E [\| X \| \| Z^{'} \|] + E [\| X \| \| Z \|] + E [\| ˜ X \| \| Z^{'} \|] + E [\| ˜ X \| \| Z \|]$
		$+ E \| Z \| + E [\| Z^{'}$

		$E [(1 + \| X + Z \| + \| ˜ X \|) (1 + \| X + Z^{'} \| + \| ˜ X \|)]$		(24)
		$\leq E [(1 + \| X \| + \| Z \| + \| ˜ X \|) (1 + \| X \| + \| Z^{'} \| + \| ˜ X \|)]$
		$= 1 + 2 E [\| X \|] + 2 E [\| ˜ X \|] + 2 E [\| X \| \| ˜ X \|] + E [X^{2}] + E [{˜ X}^{2}]$
		$+ E [\| X \| \| Z^{'} \|] + E [\| X \| \| Z \|] + E [\| ˜ X \| \| Z^{'} \|] + E [\| ˜ X \| \| Z \|]$
		$+ E \| Z \| + E [\| Z^{'}$

An Analysis of State Evolution for Approximate Message Passing with Side Information

Rigorous State Evolution Analysis for Approximate Message Passing with Side Information

Analysis of Approximate Message Passing with Non-Separable Denoisers and Markov Random Field Priors

Generating Functional Analysis of Iterative Sparse Signal Recovery Algorithms with Divergence-Free Estimators

Finite Sample Analysis of Approximate Message Passing Algorithms

A Simple Derivation of AMP and its State Evolution via First-Order Cancellation

Learning and Free Energies for Vector Approximate Message Passing

An Approximate Message Passing Framework for Side Information

I Introduction