Phase Transition of Convex Programs for Linear Inverse Problems with Multiple Prior Constraints

I Introduction

The linear inverse problem refers to the problem of recovering an unknown signal from its linear measurements. It is frequently encountered in many applications, such as image processing [1], network data analysis [2] and so on. In practice, we often have less measurements than the dimension of the true signal. As a result, the problem is generally ill-posed. Therefore, to make recovery possible, we may assume that the true signal has low complexity under some structures. Commonly considered structures include sparsity and low rank, and the corresponding recovery problems are known as compressed sensing and matrix completion.

Given the structures of the signal, a popular approach for recovery is to solve a convex program that enforces the known prior information about the structures. For example, we pursue a sparse recovery through $ℓ_{1}$ norm minimization in the compressed sensing problem, and a low-rank recovery through nuclear norm minimization in the matrix completion problem. This approach is shown to be simple and efficient in many practical applications.

Meanwhile, a sharp phase transition is numerically observed, when we use convex programs to recover structured signals. The phase transition refers to the phenomenon that for a certain convex program, when the measurement number is greater than some threshold, it succeeds with high probability; while when the measurement number is smaller than another threshold, it fails with high probability. When we say a sharp phase transition, we mean that the transition region is very narrow. This phenomenon has attracted many researchers, and much work has been done to explain it in theory in the past several years. Some exciting results have been obtained since then.

In [3, 4, 5, 6], Donoho and Tanner analyzed the phase transition of the compressed sensing problem in the asymptotic regime. They first demonstrated that the $ℓ_{1}$ minimization approach succeeds if and only if the random projection preserves the structure of faces of cross-polytope, and then used the theory of polytope angles to deal with this problem. In [7, 8, 9], the authors established a connection between the phase transition and the statistical decision theory, and revealed that the phase transition curve coincides with the minimax risk curve of denoising in many linear inverse problems. In [10], Amelunxen et al. presented a comprehensive analysis of the phase transition of convex programs in the linear inverse problem. They first formulated the phase transition problem to a geometry problem, then used tools from the theory of conic integral geometry to study this geometry problem. The results show that the phase transition of convex programs occurs at the statistical dimension of the descent cone of the structure inducing function at the true signal. In [11], Rudelson and Vershynin studied the performance of the $ℓ_{1}$ minimization approach using the “escape from the mesh” theorem [12] in Gaussian process theory. Later, their ideas were extended in the papers [13, 14, 10], and the phase transition were identified by incorporating the arguments of Rudelson and Vershynin with a polarity argument. The obtained results are stated in terms of Gaussian width, and consistent with the results in [10]. In [15], Bayati et al. made use of a state evolution framework, inspired by ideas from statistical physics, and demonstrated that the phase transition of $ℓ_{1}$ minimization is universal over a class of sensing matrices. Recently, in [16], Oymak and Tropp demonstrated the universality laws for the phase transition of convex programs for linear inverse problems, over a class of sensing matrices.

However, most of the above work focuses on the case when we have no additional prior constraints. But in many practical problems, we do have some additional prior information. For example, in image processing problems, in addition to the structures about texture etc, the fact that the pixel values are non-negative may help to recover the true image. In these cases, we would solve convex problems with (multiple) prior constraints to recover the true signal. While these problems exhibit a sharp phase transition as well, theoretical understanding of the phase transition is far from satisfactory. We mention that in [3, 4], Donoho and Tanner studied the $ℓ_{1}$ minimization problem with an additional non-negativity constraint, and “weak threshold” and “strong threshold” were obtained in the asymptotic regime, which marks the phase transition. Nevertheless, a comprehensive analysis about the phase transition of this problem does not exist. Furthermore, when the signal has structures other than sparsity, or when we have prior constraints other than non-negativity, it remains an open problem to prove the existence and identify the location of the phase transition.

In this paper, we study the phase transition of convex programs with multiple prior constraints under Gaussian random measurements. In our analysis, we first introduce a new set and a new cone, called the prior restricted set and prior restricted cone, respectively. Next, we give a sufficient and necessary condition for the success of convex programs, which involves the prior restricted cone. It states that convex programs succeed if and only if the intersection of the null space of the sensing matrix and the prior restricted cone contains only the origin. This condition has been well studied by Amelunxen et al. in [10] using the theory of conic integral geometry. Utilizing their results, we obtain that the phase transition of convex programs with multiple prior constraints occurs at the statistical dimension of the prior restricted cone. Thus, intuitively, the “dimension” of the prior restricted cone (i.e., the statistical dimension of this cone) can be seen as a measure of how much we know about the true signal from the prior information, if convex programs are used to recover signals. Moreover, to apply our theoretical results in practice, we present two recipes to accurately estimate the statistical dimension of the prior restricted cone. The two recipes work under different conditions, and we give a detailed analysis for them. To further illustrate our results, we apply our theoretical results and the estimation recipes to study the phase transition of two specific problems: One is the linear inverse problem with $ℓ_{2}$ norm constraints, and the other is the linear inverse problem with non-negativity constraints. We obtain computable formulas for the statistical dimension and related error bounds in either problem. The following simulations demonstrate that our results match the empirical successful probability perfectly.

The rest of the paper is organized as follows: In section II, we give a precise statement of the problems studied in this paper. In section III, some preliminaries and notations are introduced. In section IV, we state our main results. In section V, we apply our main results to study the phase transition of two specific problems. In section VI, simulations are provided to demonstrate our theoretical results. In section VII, we conclude the paper.

Ii Problem Formulation

In this section, we provide a precise statement of the problems studied in this paper. In section II-A, we introduce the linear inverse problem. In section II-B, we introduce the convex optimization procedure to recover signals from compressed, linear measurements.

Ii-a Linear Inverse Problem

In the linear inverse problem, we observe a signal via its linear measurements:

y = A x^{⋆},

(1)

where $y \in R^{m}$

is the measurement vector,

A \in R^{m \times n}

is the sensing matrix, and

x^{⋆} \in R^{n}

is the unknown signal. Our goal is to recover

x^{⋆}

given the knowledge of

y

and

A

Ii-B Convex Optimization Procedure

In many applications, we often have compressed measurements, i.e., $m < n$ . As a result, to recovery $x^{⋆}$ from $y$ and $A$ is an ill-posed problem. Hence, to make recovery possible, it is commonly assumed that the signal $x^{⋆}$ is well structured. In this case, a simple yet efficient approach for recovery is to solve a convex program, which forces the solution to have the corresponding structures. Moreover, apart from the assumed structures, we may have some additional prior information about $x^{⋆}$ . For example, we may know the $ℓ_{2}$ norm of $x^{⋆}$ beforehand, or the signal $x^{⋆}$ is non-negative. The additional prior information often acts as constraints.

Suppose that $f_{0} : R^{n} \to ¯ ¯¯ ¯ R$ is a proper convex function and promotes the structures of $x^{⋆}$ , and $f_{i} : R^{n} \to ¯ ¯¯ ¯ R, 1 \leq i \leq k,$ are some proper convex functions and promote the additional prior information of $x^{⋆}$ . Then in practice the following convex program is often used to recover the true signal $x^{⋆}$ :

min f_{0} (x), s . t . y = A x, f_{i} (x) \leq f_{i} (x^{⋆}), i = 1, \dots, k .

(2)

We say that the convex problem (2) succeeds if the unique solution $^x$ satisfies $^x = x^{⋆}$ ; otherwise, we say it fails.

In this paper, we study the phase transition of problem (2). The analysis relies on some knowledge from convex analysis and convex geometry. Hence, in the next section, we give a brief introduction about the needed knowledge.

Iii Preliminaries

In this section, we present some preliminaries that will be used in our analysis.

Iii-a Subgradient

Suppose $h : R^{n} \to ¯ ¯¯ ¯ R$ is a proper convex function. Then the subdifferential of $h$ at $z \in R^{n}$ is the set

\partial h (z) = {u \in R^{n} : h (z + t) \geq h (z) + ⟨ u, t ⟩ for all t \in R^{n}} .

Iii-B Descent Cones and Normal Cones of Convex Functions

The descent cone of a proper convex function $h : R^{n} \to ¯ ¯¯ ¯ R$ at $z \in R^{n}$ is the set of all non-ascent directions of $h$ at $z$ :

D (h, z) = {d \in R^{n} : \exists a > 0, h (z + a \cdot d) \leq h (z)} .

The normal cone of a proper convex function $h : R^{n} \to ¯ ¯¯ ¯ R$ at $z \in R^{n}$ is the polar of the descent cone of $h$ at $z$ :

N (h, z) = D (h, z)^{\circ} = {u \in R^{n} : ⟨ u, d ⟩ \leq 0 for all d \in D (h, z)} .

Suppose $\partial h (z)$ is non-empty, compact, and does not contain the origin, then the normal cone is the cone generated by the subdifferential [17, Corollary 23.7.1]:

Iii-C Normal Cone to Convex Sets

Let $C \subseteq R^{n}$ be a convex set with $¯ x \in C$ . The normal cone to $C$ at $¯ x$ is

N(¯x;C)\coloneqq{v∈Rn: ⟨v,x−¯x⟩≤0, ∀x∈C}.

Iii-D Statistical Dimension of Convex Cones

For a convex cone $K$ , the statistical dimension of $K$ is defined as:

δ (K) = E (sup t \in K \cap B^{n} ⟨ g, t ⟩)^{2}, where g \sim N (0, I_{n}) .

The statistical dimension of a convex cone has a number of important properties, see [10, Proposition 3.1]. Moreover, the statistical dimension satisfies the following additivity property:

Fact 1 (Additivity of statistical dimension).

Let $K_{1}$ and $K_{2}$ be two convex cones in $R^{n}$ . The following holds:

If for any $a \in K_{1}$ and $b \in K_{2}$ , we have $⟨ a, b ⟩ = 0$ . Then

$δ (K_{1} + K_{2}) = δ (K_{1}) + δ (K_{2}) .$
If for any $a \in K_{1}$ and $b \in K_{2}$ , we have $⟨ a, b ⟩ \leq 0$ . Then

$δ (K_{1} + K_{2}) \geq δ (K_{1}) + δ (K_{2}) .$
If for any $a \in K_{1}$ and $b \in K_{2}$ , we have $⟨ a, b ⟩ \geq 0$ . Then

$δ (K_{1} + K_{2}) \leq δ (K_{1}) + δ (K_{2}) .$

Proof.

See Appendix F. ∎

Fact 1 generalizes the fact that for two linear subspaces $L_{1}$ and $L_{2}$ , suppose $L_{1} ⊥ L_{2}$ , then $dim (L_{1} + L_{2}) = dim (L_{1}) + dim (L_{2})$ , since the statistical dimension extends the dimension of a linear subspace to the class of convex cones [10].

Iii-E Indicator Function of a Convex Set

Let $C \subseteq R^{n}$ be a convex set. Then the indicator function of the set $C$ is defined as

IC(x)={0,when x∈C,∞,when x∉C.\

For any $¯ x \in C$ , the subdifferential of $I_{C}$ is [18, Example 2.32]:

(3)

Iii-F Prior Restricted Set and Prior Restricted Cone

We first define the prior restricted set of convex problem (2):

Definition 1 (Prior Restricted Set).

For the convex problem (2), suppose $x^{⋆} \in R^{n}$ is the true signal, then we define its prior restricted set as the following set:

S = {d \in R^{n} : f_{i} (x^{⋆} + d) \leq f_{i} (x^{⋆}), i = 0, 1, \dots, k} .

Using this set, we can define the prior restricted cone of problem (2):

Definition 2 (Prior Restricted Cone).

For the convex problem (2), suppose $x^{⋆} \in R^{n}$ is the true signal, then we define its prior restricted cone as the following set:

C = cone (S) = {u \in R^{n} : \exists t > 0, f_{i} (x^{⋆} + t \cdot u) \leq f_{i} (x^{⋆}), i = 0, 1, \dots, k} .

Iii-G Notations

Throughout, we denote $R_{+}^{n}$ the non-negative orthant in $R^{n}$ : $Rn+\coloneqq{x∈Rn:xi≥0 for 1≤i≤n}$ , and $R_{+ +}^{n}$ the positive part: $Rn++\coloneqq{x∈Rn:xi>0 for 1≤i≤n}$ .

For a set $C \in R^{n}$ , we use $int (C)$ to denote its interior:

int(C)\coloneqq{x∈C:B(x,r)⊆C for some r>0}.

Denote $aff (C)$ the affine hull of $C$ :

aff (C) = {θ_{1} x_{1} + \dots + θ_{k} x_{k} : x_{1}, \dots, x_{k} \in C, θ_{1} + \dots + θ_{k} = 1},

and $ri (C)$ the relative interior of the set $C$ :

ri (C) = {x \in C : B (x, r) \cap a f f (C) \subseteq C for some r > 0} .

The closure of $C$ is denoted by either $¯ ¯¯ ¯ C$ or $cl (C)$ .

Given a point $u \in R^{n}$ and a subset $C \subseteq R^{n}$ , the distance of $u$ to the set $C$ is denoted by $dist (u, C)$ :

dist(u,C)\coloneqqinfx∈C∥u−x∥2.

We denote $Π_{C} (u)$ the projection of $u$ onto the set $C$ :

Π_{C} (u) := {x \in C : ∥ u - x ∥_{2} = dist (u, C)} .

If $C$ is non-empty, convex, and closed, the projection $Π_{C} (u)$ is a singleton. In this case, $Π_{C} (u)$ may denote the unique point in it, depending on the context.

Iv Main Results

In this section, we state our main results in this paper. We first give results about the phase transition of problem (2) in subsection IV-A, and then present two recipes to estimate the statistical dimension of the prior restricted cone in subsections IV-B and IV-C.

Iv-a Phase Transition of Convex Programs with Multiple Prior Constraints

In this subsection, we state our main results about the phase transition of problem (2). We begin by a geometry condition which determines the success of problem (2):

Lemma 1 (Optimality condition).

Consider problem (2) to recover the true signal $x^{⋆}$ . If $f_{i}$ is a proper convex function for any $0 \leq i \leq k$ , problem (2) succeeds if and only if

C \cap null (A) = {0},

where $C$ denotes the prior restricted cone of problem (2).

Proof.

See Appendix A. ∎

Fig. 1 gives a geometric interpretation of Lemma 1. Note that when there is no additional prior constraint, i.e., when we consider problem

min f_{0} (x), s . t . y = A x

(4)

to recover $x^{⋆}$ , the prior restricted cone is exactly $D (f_{0}, x^{⋆})$ , the descent cone of $f_{0}$ at $x^{⋆}$ . In this case, our optimality condition, Lemma 1, will degenerate to the optimality condition given by Chandrasekaran et al. in [13, Fact 2.8].

[scale=0.49]success_noise_free.eps

[scale=0.49]failure_noise_free.eps

Fig. 1: A geometric interpretation of the optimality condition for success of problem (2), i.e., Lemma 1. In both figures, the dark red line denotes the null space of

A

, the light blue region denotes the prior restricted cone of problem (2) (i.e.,

C

), and the dark blue region denotes the prior restricted set of problem (2) (i.e.,

S

). In figure (a), the intersection of

null (A)

and

C

contains only the origin. In this case, problem (2) succeeds. In figure (b), the intersection of

null (A)

and

C

contains a ray. In this case, problem (2) fails.

Using Lemma 1, we can study the phase transition of problem (2). For this purpose, we assume that we have random sensing matrix. In particular, we assume that $A$

is drawn at random from the standard normal distribution on

R^{m \times n}

. According to Lemma 1, to study the phase transition of problem (2), it is sufficient to answer the following questions:

[leftmargin=1em]
Under what conditions the kernel of $A$ intersects the cone $C$ trivially with high probability?
Under what conditions the kernel of $A$ intersects the cone $C$ nontrivially with high probability?

This questions have been well studied in recent years. We borrow the answer from [10]:

Proposition 1 ([10], Theorem I).

Fix a tolerance $ζ$ . Suppose the matrix $A \in R^{m \times n}$ has independent standard normal entries, and $K$ denotes a convex cone. Then when

m \leq δ (K) - a_{ζ} \sqrt{n},

we have $null (A) \cap K = {0}$ with probability less than $ζ$ . On the contrary, when

m \geq δ (K) + a_{ζ} \sqrt{n},

we have $null (A) \cap K = {0}$ with probability at least $1 - ζ$ . The quantity $aζ\coloneqq√8log(4/ζ)$ .

Proposition 1 is a direct consequence of [10, Theorem I]. The proof involves the theory of conic integral geometry. See reference [10] for details. Now combining Lemma 1 and Proposition 1, we obtain our main results about the phase transition:

Theorem 1 (Phase transition of convex programs with multiple prior constraints).

Consider convex problem (2) to solve the linear inverse problem. If the sensing matrix $A$ has independent standard normal entries, the phase transition of problem (2) occurs at the statistical dimension of its prior restricted cone. More precisely, for any $ζ > 0$ , when the measurement number $m$ satisfies

m \leq δ (C) - a_{ζ} \sqrt{n},

problem (2) fails with probability at least $1 - ζ$ . On the contrary, when

m \geq δ (C) + a_{ζ} \sqrt{n},

problem (2) succeeds with probability at least $1 - ζ$ . The quantity $aζ\coloneqq√8log(4/ζ)$ .

Since the phase transition occurs at the statistical dimension of the prior restricted cone, intuitively, we can see it as a measure of how much we know about the true signal from the prior information.

Remark 1.

We can apply our Theorem 1 to analyze the phase transition of problem (4). The prior restricted cone of problem (4) is exactly $D (f_{0}, x^{⋆})$ , the descent cone of $f_{0}$ at $x^{⋆}$ . Thus, in this case, our Theorem 1 can be read as: The phase transition of problem (4) occurs at the statistical dimension of $D (f_{0}, x^{⋆})$ . This coincides with the results in [10, Theorem II].

Iv-B Statistical Dimension of Prior Restricted Cones: Part I

In theory, Theorem 1 have revealed that the phase transition of problem (2) occurs at the statistical dimension of its prior restricted cone. However, if we want to apply these results in practice, we must find ways to compute the statistical dimension efficiently. For this purpose, in this subsection, we present a recipe that provides a reliable estimate for the statistical dimension of the prior restricted cone, when all the subdifferentials are compact and do not contain the origin. The idea is inspired by the recipe proposed by Amelunxen et al. [10, pp. 244-248] for the computation of the statistical dimension of a descent cone.

The basic idea for the recipe is that the statistical dimension of the prior restricted cone can be expressed in terms of its polar, which has a close relation with the normal cones, and furthermore, the subdifferentials, of the functions $f_{i}$ ’s, $0 \leq i \leq k$ . Let us first express the statistical dimension in terms of normal cones.

Lemma 2.

Consider problem (2) to recover the true signal $x^{⋆}$ . Let $D (f_{i}, x^{⋆}), N (f_{i}, x^{⋆})$ denote the descent cone and normal cone of $f_{i}$ at $x^{⋆}$ for $0 \leq i \leq k$ , respectively. Suppose that

ri (D (f_{0}, x^{⋆})) \cap ri (D (f_{1}, x^{⋆})) \cap \dots \cap ri (D (f_{k}, x^{⋆})) \neq \emptyset .

Then the polar of the prior restricted cone can be expressed as follows:

C^{\circ} = k \sum i = 0 N (f_{i}, x^{⋆}),

and the statistical dimension of the prior restricted cone can be expressed as follows:

δ (C) = E {dist}^{2} (g, k \sum i = 0 N (f_{i}, x^{⋆})) .

Proof.

See Appendix B-A. ∎

Lemma 2 establishes connections between the prior restricted cone and the individual normal cones, but it does not allow us to compute the statistical dimension of the prior restricted cone efficiently. This can be done by incorporating the subdifferential expression for normal cones.

Theorem 2 (The statistical dimension of the prior restricted cone).

Let $C$ be the prior restricted cone of problem (2), and let $x^{⋆} \in R^{n}$ be the true signal. Assume that for any $0 \leq i \leq k$ , the subdifferential $\partial f_{i} (x^{⋆})$ is non-empty, compact, and does not contain the origin. Assume that the descent cones satisfy

ri (D (f_{0}, x^{⋆})) \cap ri (D (f_{1}, x^{⋆})) \cap \dots \cap ri (D (f_{k}, x^{⋆})) \neq \emptyset .

Define the function $J : R_{+}^{k + 1} \to R$ to be

J(τ)\coloneqqE[dist2(g,k∑i=0τi⋅∂fi(x⋆))],

where $g \sim N (0, I_{n})$ . Then the statistical dimension of the prior restricted cone has the following upper bound:

δ (C) \leq inf τ \in R_{+}^{k + 1} J (τ) .

The function $J (τ)$ is convex, continuous, and continuously differentiable in $R_{+}^{k + 1}$ . It attains its minimum in a compact subset of $R_{+}^{k + 1}$ . Moreover, suppose that

the two sets k \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}) and k \sum i = 0 {~ τ}_{i} \cdot \partial f_{i} (x^{⋆}) are not identical, for any τ \neq ~ τ \in R_{+}^{k + 1} .

Then the function $J (τ)$ is strictly convex, and attains its minimum at a unique point. For the differential of $J$ at the boundary of $R_{+}^{k + 1}$ , we interpret the partial derivative $\frac{\partial J}{\partial τ_{i}}$ similarly as the right derivative, if $τ_{i} = 0$ .

Proof.

Since the subdifferential is non-empty, compact, and does not contain the origin, the normal cones is the cone generated by the subdifferential [17, Corollary 23.7.1]. Thus, by Lemma 2,

	$δ (C)$	$= E [{dist}^{2} (g, k \sum i = 0 N (f_{i}, x^{⋆}))] = E [{dist}^{2} (g, k \sum i = 0 (⋃ τ_{i} \geq 0 τ_{i} \cdot \partial f_{i} (x^{⋆})))]$
		$= E [{dist}^{2} (g, ⋃ τ \in R_{+}^{k + 1} (k \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆})))] = E [inf τ \in R_{+}^{k + 1} {dist}^{2} (g, k \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}))]$
		$\leq inf τ \in R_{+}^{k + 1} E [{dist}^{2} (g, k \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}))] .$

The inequality results from Jensen’s inequality. The proof of properties of $J$ appears in Appendix B-B and Appendix B-C. ∎

Assume that for $0 \leq i \leq k$ , the function $f_{i} : R^{n} \to ¯ ¯¯ ¯ R$ is a proper convex function and $x^{⋆} \in R^{n}$ .
Assume that the intersection of interiors of descent cones are non-empty, i.e., $ri (D (f_{0}, x^{⋆})) \cap ri (D (f_{1}, x^{⋆})) \cap \dots \cap ri (D (f_{k}, x^{⋆})) \neq \emptyset$ .
Assume that for $0 \leq i \leq k$ , the subdifferential $\partial f_{i} (x^{⋆})$ is non-empty, compact, and does not contain the origin.
Assume that for any $τ \neq ~ τ \in R_{+}^{k + 1}$ , the two sets $\sum_{i = 0}^{k} τ_{i} \cdot \partial f_{i} (x^{⋆})$ and $\sum_{i = 0}^{k} {~ τ}_{i} \cdot \partial f_{i} (x^{⋆})$ are not identical.

1:Identify the subdifferential

S_{i} = \partial f_{i} (x^{⋆})

, for

0 \leq i \leq k

2:For any

τ \in R_{+}^{q + 1}

, find the following set, which is the Minkowski sum of the subdifferentials:

S (τ) = \sum_{i = 0}^{k} τ_{i} \cdot S_{i}

3:Compute the function

J(τ)\coloneqqEdist2(g,S(τ))

, where

g \sim N (0, I_{n})

4:Compute the differential,

\nabla J (τ)

, of function

J (τ)

5:Find the minimizer

τ^{⋆}

J (τ)

over

R_{+}^{k + 1}

, using its differential

\nabla J (τ)

6:The statistical dimension of the prior restricted cone has the upper bound

δ (C) \leq J (τ^{⋆})

Recipe 1 The statistical dimension of the prior feasible descent cone

Theorem 2 provides an effective way to estimate the statistical dimension of the prior restricted cone, when all the subdifferentials are non-empty, compact, and does not contain the origin. We summarize it in Recipe 1. In subsection V-A, we apply Recipe 1 to study the phase transition of linear inverse problems with $ℓ_{2}$ norm constraints.

Remark 2.

In [10], Amelunxen et al. studied the phase transition of problem (4). They proved that the phase transition occurs at the statistical dimension of the descent cone $D (f_{0}, x^{⋆})$ , and provided a recipe to compute it. Our Recipe 1 can be seen as a generalization of this recipe from one function to multiple functions, and our proof idea for Theorem 2 is inspired by the proof for [10, Proposition 4.1].

Iv-C Statistical Dimension of Prior Restricted Cones: Part II

Recipe 1 gives a reliable estimate of the statistical dimension of the prior restricted cone, when all the subdifferentials are compact and do not contain the origin. However, in many practical applications, we may encounter the case that some of the subdifferentials are unbounded or contain the origin. For example, consider the linear inverse problem with non-negativity constraints, i.e.,

min f_{0} (x), s . t . y = A x, x \geq 0 .

(5)

Note that $x \geq 0$ is equivalent to $I_{R_{+}^{n}} (x) \leq I_{R_{+}^{n}} (x^{⋆})$ . The subdifferential of indicator functions has specific formula (3). It is easy to verify that the subdifferential of $I_{R_{+}^{n}}$ at $x^{⋆}$ contains the origin, and if $x^{⋆}$ contains zero entries, it is unbounded. Therefore, Recipe 1 cannot be used directly, and we have to find other ways to compute the statistical dimension. Actually, an effective way in this case is to express the normal cones via the subdifferentials, only for those functions whose subdifferentials are compact and do not contain the origin.

Theorem 3 (The statistical dimension of the prior restricted cone).

Let $C$ be the prior restricted cone of problem (2), and let $x^{⋆} \in R^{n}$ be the true signal. Assume that for $0 \leq i \leq q$ , the subdifferentials $\partial f_{i} (x^{⋆})$ ’s are non-empty, compact, and do not contain the origin, and for $q < i \leq k$ , the subdifferentials $\partial f_{i} (x^{⋆})$ ’s are non-empty, where $0 \leq q < k$ is a natural number. Assume that the descent cones satisfy

ri (D (f_{0}, x^{⋆})) \cap ri (D (f_{1}, x^{⋆})) \cap \dots \cap ri (D (f_{k}, x^{⋆})) \neq \emptyset .

Assume that for any $τ \in S^{q} \cap R_{+}^{q + 1}$ , we have

0 \notin cl (k \sum i = q + 1 N (f_{i}, x^{⋆})) + q \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}),

Define the function $J : R_{+}^{q + 1} \to R$ by

J(τ)\coloneqqE[dist2(g,q∑i=0τi⋅∂fi(x⋆)+k∑i=q+1N(fi,x⋆))],

where $g \sim N (0, I_{n})$ . Then the statistical dimension of the prior restricted cone has the following upper bound:

δ (C) \leq inf τ \in R_{+}^{q + 1} J (τ) .

The function $J (τ)$ is convex, continuous, and continuously differential in $R_{+}^{q + 1}$ . It attains its minimum in a compact subset of $R_{+}^{q + 1}$ . Moreover, suppose that for any $τ \neq ~ τ \in R_{+}^{q + 1}$ ,

the two sets cl (k \sum i = q + 1 N (f_{i}, x^{⋆})) + q \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}) and cl (k \sum i = q + 1 N (f_{i}, x^{⋆})) + q \sum i = 0 {~ τ}_{i} \cdot \partial f_{i} (x^{⋆}) are not identical .

Then the function $J (τ)$ is strictly convex, and attains its minimum at a unique point. For the differential of $J$ at the boundary of $R_{+}^{q + 1}$ , we interpret the partial derivative $\frac{\partial J}{\partial τ_{i}}$ similarly as the right derivative, if $τ_{i} = 0$ .

Proof.

We proceed similarly as in Theorem 2,

	$δ (C)$	$= E [{dist}^{2} (g, k \sum i = 0 N (f_{i}, x^{⋆}))] = E [{dist}^{2} (g, q \sum i = 0 (⋃ τ_{i} \geq 0 τ_{i} \cdot \partial f_{i} (x^{⋆})) + k \sum i = q + 1 N (f_{i}, x^{⋆}))]$
		$= E [{dist}^{2} (g, ⋃ τ \in R_{+}^{q + 1} (q \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}) + k \sum i = q + 1 N (f_{i}, x^{⋆})))]$
		$= E [inf τ \in R_{+}^{q + 1} {dist}^{2} (g, q \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}) + k \sum i = q + 1 N (f_{i}, x^{⋆}))]$
		$\leq inf τ \in R_{+}^{q + 1} E [{dist}^{2} (g, q \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}) + k \sum i = q + 1 N (f_{i}, x^{⋆}))] .$

The inequality results from Jensen’s inequality. The proof of properties of $J$ appears in Appendix C-A and Appendix C-B. ∎

Theorem 3 further generalizes our Theorem 2 and [10, Proposition 4.1] to the case when some of the subdifferentials are unbounded or contain the origin, and the proofs share similar ideas. We summarize it in Recipe 2. In subsection V-B, we apply Recipe 2 to study the phase transition of linear inverse problems with non-negativity constraints.

Assume that for $0 \leq i \leq k$ , the function $f_{i} : R^{n} \to ¯ ¯¯ ¯ R$ is a proper convex function and $x^{⋆} \in R^{n}$ .
Assume that the intersection of interiors of descent cones are non-empty, i.e., $ri (D (f_{0}, x^{⋆})) \cap ri (D (f_{1}, x^{⋆})) \cap \dots \cap ri (D (f_{k}, x^{⋆})) \neq \emptyset$ .
Assume that for $0 \leq i \leq q$ , the subdifferential $\partial f_{i} (x^{⋆})$ is non-empty, compact, and does not contain the origin.
Assume that for $q + 1 \leq i \leq k$ , the subdifferential $\partial f_{i} (x^{⋆})$ is non-empty.
Assume that for any $τ \in S^{q} \cap R_{+}^{q + 1}$ , the set $cl (\sum_{i = q + 1}^{k} N (f_{i}, x^{⋆})) + \sum_{i = 0}^{q} τ_{i} \cdot \partial f_{i} (x^{⋆})$ does not contain the origin.
Assume that for any $τ \neq ~ τ \in R_{+}^{q + 1}$ , the two sets

cl (k \sum i = q + 1 N (f_{i}, x^{⋆})) + q \sum i = 0 τ_{i} \cdot \partial f_{i} (x^{⋆}) and cl (k \sum i = q + 1 N (f_{i}, x^{⋆})) + q \sum i = 0 {~ τ}_{i} \cdot \partial f_{i} (x^{⋆})

are not identical.

1:Identify the subdifferential

S_{i} = \partial f_{i} (x^{⋆})

, for

0 \leq i \leq q

, and the normal cone

N_{i} = N (f_{i}, x^{⋆})

, for

q + 1 \leq i \leq k

2:For any

τ \in R_{+}^{q + 1}

, find the following set, which is the Minkowski sum of the subdifferentials and the normal cones:

S (τ) = \sum_{i = 0}^{q} τ_{i} S_{i} + \sum_{i = q + 1}^{k} N_{i}

3:Compute the function

J(τ)\coloneqqEdist2(g,S(τ))

, where

g \sim N (0, I_{n})

4:Compute the differential,

\nabla J (τ)

, of function

J (τ)

5:Find the minimizer

τ^{⋆}

J (τ)

over

R_{+}^{k + 1}

, using its differential

\nabla J (τ)

6:The statistical dimension of the prior restricted cone has the upper bound

δ (C) \leq J (τ^{⋆})

Recipe 2 The statistical dimension of the prior feasible descent cone

V Examples and Applications

In this section, we make use of our theoretical results and the computation recipes to study the phase transition of several specific problems. In subsection V-A, we study the phase transition of linear inverse problems with $ℓ_{2}$ norm constraints, and in subsection V-B, we study the phase transition of linear inverse problems with non-negativity constraints.

V-a Phase Transition of Linear Inverse Problem with $ℓ_{2}$ Norm Constraints

In this subsection, we make use of Recipe 1 to study the phase transition of linear inverse problems with $ℓ_{2}$ norm constraints. In other words, we study the phase transition of the following convex problem:

min f_{0} (x), s . t . y = A x, ∥ x ∥_{2} \leq ∥ x^{⋆} ∥_{2} .

(6)

Note that for $x^{⋆} \neq 0$ , the subdifferential of $ℓ_{2}$ norm is $\partial ∥ x^{⋆} ∥_{2} = {\frac{x^{⋆}}{∥ x^{⋆} ∥_{2}}}$ . Therefore, applying Recipe 1 directly, we obtain the following results:

Corollary 1.

Let $C_{1}$ be the prior restricted cone of problem (6), and let $x^{⋆} \in R^{n}$ be the true signal. Assume that the subdifferential $\partial f_{0} (x^{⋆})$ is non-empty, compact, do not contain the origin. Assume that the descent cones satisfy

ri (D (f_{0}, x^{⋆})) \cap ri (D (∥ \cdot ∥_{2}, x^{⋆})) \neq \emptyset .

Define the function $J_{1} : R_{+}^{2} \to R$ to be

J_{1} (τ) = E {dist}^{2} [(g, τ_{0} \cdot \partial f_{0} (x^{⋆}) + τ_{1} \cdot \frac{x^{⋆}}{∥ x^{⋆} ∥_{2}})],

where $g \sim N (0, I_{n})$ . Then the statistical dimension of the prior restricted cone of problem (6) has the following upper bound:

δ (C_{1}) \leq inf τ \in R_{+}^{2} J_{1} (τ) .

The function $J_{1} (τ)$ is convex, continuous, and continuously differentiable in $R_{+}^{2}$ . It attains its minimum in a compact subset of $R_{+}^{2}$ . Moreover, suppose that

the two sets τ_{0} \cdot \partial f_{0} (x^{⋆}) + τ_{1} \cdot \frac{x^{⋆}}{∥ x^{⋆} ∥_{2}} and {~ τ}_{0} \cdot \partial f_{0} (x^{⋆}) + {~ τ}_{1} \cdot \frac{x^{⋆}}{∥ x^{⋆} ∥_{2}} are not % identical for any τ \neq ~ τ \in R_{+}^{2} .

Then the function $J_{1} (τ)$ is strictly convex, and attains its minimum at a unique point. For the differential of $J_{1}$ at the boundary of $R_{+}^{2}$ , we interpret the partial derivative $\frac{\partial J_{1}}{\partial τ_{i}}$ similarly as the right derivative, if $τ_{i} = 0$ .

Proof.

Applying Theorem 2 to problem (6) directly, we obtain Corollary 1. ∎

Corollary 1 implies that to study the phase transition of problem (6), we need to find the infimum of $J_{1}$ . When $f_{0}$ is a general proper convex function, the infimum may be attained anywhere in $R_{+}^{2}$ . However, when $f_{0}$ is a norm, an important result is that the infimum of $J_{1}$ must be attained in $R_{+} \times {0}$ .

Proposition 2.

Consider problem (6). Assume that $f_{0}$ is a norm, and the conditions in Corollary 1 hold. Define the function $J_{2} : R_{+} \to R$ as

J_{2} (τ) = E {dist}^{2} (g, τ \cdot \partial f_{0} (x^{⋆})) for τ \geq 0.

The function $J_{2} (τ)$ is strictly convex, continuously differentiable in $R_{+}$ , and attains its minimum at a unique point. Moreover, the unique minimizer $τ^{⋆} = (τ_{0}^{⋆}, τ_{1}^{⋆})$ of $J_{1}$ satisfies $τ_{1}^{⋆} = 0$ and $τ_{0}^{⋆}$ is the unique minimizer of $J_{2}$ , and the minimum of $J_{1}$ over $R_{+}^{2}$ and that of $J_{2}$ over $R_{+}$ are equal:

inf τ \in R_{+}^{2} J_{1} (τ) = J_{1} (τ^{⋆}) = J_{2} (τ_{0}^{⋆}) = inf τ \in R_{+} J_{2} (τ) .

Proof.

The first part of this proposition, i.e., the properties of $J_{2} (τ)$ , has been proved in [10]. For the proof of the results about $τ^{⋆}$ , please see Appendix D-A. ∎

Remark 3.

Let $C_{2}$ denote the prior restricted cone of problem (4), i.e., $C_{2} = D (f_{0}, x^{⋆})$ . In [10], Amelunxen et al. have proved that ${inf}_{τ \in R_{+}} J_{2} (τ)$ is a reliable estimate of $δ (C_{2})$ . Therefore, Proposition 2 implies that when we use Recipe 1 to compute the statistical dimension of the prior restricted cone of problem (6), the obtained phase transition point is exactly the same as that of problem (4).

At the first sight, the above results may be surprising, since we have more prior information, but the obtained phase transition point is the same. From another point of view, this implies that if our Recipe 1 can provide an accurate estimation of the statistical dimension, $δ (C_{1})$ and $δ (C_{2})$ must be nearly equal. Actually, we can verify that this is the case.

Proposition 3.

Let $C_{1}$ and $C_{2}$ denote the prior restricted cones of problem (6) and problem (4), respectively. Assume that $f_{0}$ is a norm. Then

δ (C_{1}) \leq δ (C_{2}) \leq δ (C_{1}) + \frac{1}{2} .

(7)

Proof.

See Appendix D-B. ∎

Remark 4.

Proposition 3 implies that in the case when $f_{0}$ is a norm, the additional $ℓ_{2}$ norm constraint $∥ x ∥_{2} \leq ∥ x^{⋆} ∥_{2}$ has little effect on the phase transition of linear inverse problem. Moreover, since ${inf}_{τ \in R_{+}} J_{2} (τ)$ is an accurate estimate of $δ (C_{2})$ , it follows that ${inf}_{τ \in R_{+}^{2}} J_{1} (τ)$ is an accurate estimate of $δ (C_{1})$ .

Using the above results, we can obtain an error bound for Recipe 1 when applied to problem (6).

Proposition 4.

Consider problem (6) to recover $x^{⋆}$ . Assume that $f_{0}$ is a norm and denote $C_{1}$ the prior restricted cone of problem (6). Then under the conditions of Corollary 1, we have

0 \leq inf τ \in R_{+}^{2} J_{1} (τ) - δ (C_{1}) \leq \frac{2 sup {∥ s ∥_{2} : s \in \partial f_{0} (x^{⋆})}}{} f_{0} (x^{⋆} / ∥ x^{⋆} ∥_{2}) + \frac{1}{2} .

(8)

Proof.

This is a direct consequence of [10, Theorem 4.3], Proposition 2, and Proposition 3. We omit the proof. ∎

Proposition 4 implies that our Recipe 1 can provide an accurate estimate of the statistical dimension of the prior restricted cone, when applied to problem (6). Next, as a more concrete example, let us study the phase transition of the compressed sensing problem with $ℓ_{2}$ norm constraints:

min ∥ x ∥_{1}, s . t . y = A x, ∥ x ∥_{2} \leq ∥ x^{⋆} ∥_{2} .

(9)

We have the following results:

Corollary 2.

Let $C_{1}$ be the prior restricted cone of problem (9), and $x^{⋆} \in R^{n}$ be the true signal. Assume that $x^{⋆}$ has exactly $s$ non-zero entries. Then the statistical dimension of the prior restricted cone of problem (9) satisfies

ψ_{1} (s / n) - \frac{2}{\sqrt{s n}} - \frac{1}{2 n} \leq \frac{δ (C_{1})}{n} \leq ψ_{1} (s / n),

where the function $ψ_{1} : [0, 1] \to [0, 1]$ is

ψ_{1} (ρ) = inf τ \in R_{+} {ρ (1 + τ^{2}) + (1 - ρ) \int_{τ}^{\infty} (u - τ)^{2} \cdot φ (u) d u},

(10)

where the function $φ (u) = \sqrt{\frac{2}{π}} e^{- u^{2} / 2}$ . The infimum is attained at the unique $τ$ , which solves the stationary equation

\frac{ρ}{1 - ρ} = \int_{τ}^{\infty} (\frac{u}{τ} - 1) \cdot φ (u) d u .

Proof.

This is a direct consequence of Proposition 2, Proposition 3, Proposition 4 and [10, Proposition 4.1]. We omit the proof. ∎

V-B Phase Transition of Linear Inverse Problem with Non-negativity Constraints

In this subsection, we make use of Recipe 2 to study the phase transition of linear inverse problems with non-negativity constraints, i.e., problem (5). We have confirmed that the subdifferential of $I_{R_{+}^{n}}$ is unbounded and contains the origin. Therefore, to apply Recipe 2, we have to find the normal cone $N (I_{R_{+}^{n}}, x^{⋆})$ directly. Indeed, notice that the descent cone of $I_{R_{+}^{n}}$ at $x^{⋆}$ is

D (I_{R_{+}^{n}}, x^{⋆}) = ⋃ υ \geq 0 {υ (x -

Phase Transition of Convex Programs for Linear Inverse Problems with Multiple Prior Constraints

Authors

Related Research

A Precise Analysis of PhaseMax in Phase Retrieval

Corrupted Sensing: Novel Guarantees for Separating Structured Signals

Regularization by architecture: A deep prior approach for inverse problems

Phase Retrieval Meets Statistical Learning Theory: A Flexible Convex Relaxation

Phase Transitions in Recovery of Structured Signals from Corrupted Measurements

Convex Reconstruction of Structured Matrix Signals from Linear Measurements (I): Theoretical Results

Phase Transition Behavior of Cardinality and XOR Constraints

This week in AI

I Introduction

Ii Problem Formulation

Ii-a Linear Inverse Problem

Ii-B Convex Optimization Procedure

Iii Preliminaries

Iii-a Subgradient

Iii-B Descent Cones and Normal Cones of Convex Functions

Iii-C Normal Cone to Convex Sets

Iii-D Statistical Dimension of Convex Cones

Fact 1 (Additivity of statistical dimension).

Proof.

Iii-E Indicator Function of a Convex Set

Iii-F Prior Restricted Set and Prior Restricted Cone

Definition 1 (Prior Restricted Set).

Definition 2 (Prior Restricted Cone).

Iii-G Notations

Iv Main Results

Iv-a Phase Transition of Convex Programs with Multiple Prior Constraints

Lemma 1 (Optimality condition).

Proof.

Proposition 1 ([10], Theorem I).

Theorem 1 (Phase transition of convex programs with multiple prior constraints).

Remark 1.

Iv-B Statistical Dimension of Prior Restricted Cones: Part I

Lemma 2.

Proof.

Theorem 2 (The statistical dimension of the prior restricted cone).

Proof.

Remark 2.

Iv-C Statistical Dimension of Prior Restricted Cones: Part II

Theorem 3 (The statistical dimension of the prior restricted cone).

Proof.

V Examples and Applications

V-a Phase Transition of Linear Inverse Problem with ℓ2 Norm Constraints

Corollary 1.

Proof.

Proposition 2.

Proof.

Remark 3.

Proposition 3.

Proof.

Remark 4.

Proposition 4.

Proof.

Corollary 2.

Proof.

V-B Phase Transition of Linear Inverse Problem with Non-negativity Constraints

V-a Phase Transition of Linear Inverse Problem with $ℓ_{2}$ Norm Constraints