Bundle Method Sketching for Low Rank Semidefinite Programming

1 Introduction

We consider solving semidefinite problems of the following form:

$% maximize X \in S^{n} \subset R^{n \times n}$	$⟨ - C, X ⟩$	(P)
subject to	$A X = b$
$X ⪰ 0,$

where $C \in S^{n} \subset R^{n \times n}$ , $A : S^{n} \to R^{m}$ being linear, and $b \in R^{n}$ . Denote the solution set as $X_{⋆}$ . To accomplish the task of solving (P), we consider the dual problem:

		$minimize y \in R^{d}$		$⟨ - b, y ⟩$		(D)
		subject to		$A^{*} y ⪯ C,$		(D)

whose solution set is denoted as $Y_{⋆}$ . Then for all sufficiently large $α$ , e.g., larger than the trace of any solution $X_{⋆} \in X_{⋆}$ (ding2019optimal, , Lemma 6.1), we can reformulate this as

minimize y \in R^{d}

F (y) = ⟨ - b, y ⟩ - α min {λ_{min} (C - A^{*} y), 0} .

(1)

We propose applying the bundle method to solve this problem, which generates a sequence of dual solutions $y_{t}$ . While the bundle method runs on the dual problem, a primal solution $X_{t}$ can be constructed through a series of rank one updates corresponding to subgradients of $F$ . However maintaining such a primal solution greatly increases memory costs. Fortunately, the primal problem enjoys having a low rank solution in many applications, e.g., matrix completion srebro2005rank and phase retrieval candes2013phaselift . Also, without specifying the detailed structure the problem, there always exists a solution $X_{⋆}$ to (P) with rank $r_{⋆}$ satisfying $\frac{r_{⋆} (r_{⋆} + 1)}{2} \leq m$ pataki1998rank

To utilize the existence of such a low rank solution, we employ the matrix sketching methods introduced in tropp2017practical . The main idea is the following: the skecthing method forms a linear sketch of the column and row spaces of the primal decision variable $X$ , and then uses the sketched column and row spaces to recover the primal decision variable. The recovered decision variable approximates the original $X$ well if $X$ is (approximately) low rank. Notably, we need not store the entire decision variable $X$ at each iteration, but only the sketch. Hence the memory requirements of the algorithm are substantially reduced.

Our Contributions.

Our proposed sketching bundle method produces a sequence of dual solutions $y_{k}$ and a sequence of primal solutions $X_{t}$ (which are sketched by low rank matrices ${^X}_{t}$ ). This is done without ever needing to write down the solutions $X_{t}$ , which can be a substantial boon to computational efficiency.

In particular, we consider problems satisfying the following pair of standard assumptions: (i) Strong duality holds, meaning that there is a solution pair $(X_{⋆}, y_{⋆}) \in X_{⋆} \times Y_{⋆}$ satisfying

p_{⋆} := ⟨ - C, X ⟩ = ⟨ - b, y ⟩ =: d_{⋆}

and (ii) Strict Complementarity holds, meaning there is a solution pair $(X_{⋆}, y_{⋆})$ satisfying

rank (X_{⋆}) + rank (C - A^{*} (y_{⋆})) = n .

Under these condtions, we show $X_{t}$ and $y_{t}$ converge in terms of primal and dual feasibility and optimality. In particular, all three of these quantities converge at a rate of $O (1 / t)$ .

Theorem 1.1 (Primal-Dual Convergence).

Suppose the sets $X_{⋆}, Y_{⋆}$ are both compact, and that strong duality and a strict complementary condition holds. For any $ϵ > 0$ , Algorithm 1 with properly chosen parameters produces a solution pair $X_{t}$ and $y_{t}$ with

that satisfies

	approximate primal feasibility:	$∥ b - A X_{t} ∥ \leq ϵ, X_{t} ⪰ 0,$
	approximate dual feasibility:	$λ_{min} (C - A^{*} y_{t}) \geq - ϵ,$
	approximate primal-dual optimality:	$\| ⟨ b, y_{t} ⟩ - ⟨ C, X_{t} ⟩ \| \leq ϵ$

Moreover, we show that assuming all of the minimizers in $X_{⋆}$ are low rank, the sketched primal solutions ${^X}_{t}$ converge to the set of minimizers at the following rate.

Theorem 1.2 (Sketched Solution Convergence).

Suppose the sets $X_{⋆}, Y_{⋆}$ are both compact, strong duality and a strict complementary condition holds, and all solutions $X_{⋆} \in X_{⋆}$ have rank at most $r$ . For any $ϵ > 0$ , Algorithm 1 with properly chosen parameters produces a sketched primal solution ${^X}_{t}$ with

t \leq ˜ O (1 / ϵ^{2})

that satisfies $Edist\tiny{F}(^Xt,X⋆)≤ϵ.$

2 Defining The Sketching Bundle Method

Our proposed proximal bundle method relies on an approximation of $F$ given by ${˜ F}^{t} (y) = max {F (z_{t}) + ⟨ g_{t}, y - z_{t} ⟩, {¯ F}^{t} (y)}$ where ${¯ F}^{t} (y)$ is a convex combination of the lower bounds from previous subgradients. Notice that a subgradient $g_{k}$ of $F$ at $z_{t}$ can be computed as

	$v_{t}$	$= {\begin{matrix} M i n E i g e n v e c t o r (C - A^{} z_{t}) & if λ_{min} (C - A^{} z_{t}) \leq 0 0 & otherwise \end{matrix}$		(2)
	$g_{k}$	$= - b + α A v_{t} v_{t}^{*} \in \partial F (z_{t}) .$		(3)

Each iteration then computes a proximal step on this piecewise linear function given by

z_{t + 1} \in arg min y \in R^{d} {˜ F}^{t} (y) + \frac{ρ}{2} ∥ y - y_{t} ∥^{2}

(4)

for some $ρ > 0$ . The optimality condition of this subproblem ensures that for some $θ_{t} \in [0, 1]$ Our aggregate lower bound ${¯ F}^{t} (y)$ is updated to match this certifying subgradient ${¯ F}^{t + 1} (y) = {¯ F}^{t} (z_{t + 1}) + ⟨ θ_{t} g_{t} + (1 - θ_{t}) \nabla {¯ F}^{t} (z_{t}), y - z_{t} ⟩ .$ Then the following is an exact solution for the subproblem (4):

	$θ_{t}$			(5)
	$z_{t + 1}$			(6)

If the decrease in value of $F$ from $y_{t}$ to $z_{t + 1}$ is at least $β$ fraction of the decrease in value of ${˜ F}^{t}$ from $y_{t}$ to $z_{t + 1}$ , then the bundle method sets $y_{t + 1} = z_{t + 1}$ (called a descent step). Otherwise the method sets $y_{t + 1} = y_{t}$ (called a null step).

Input:

A : R^{n \times n} \to R^{d}

b \in R^{d}

C \in R^{n \times n}

y_{1} \in R^{d}

ρ > 0, β \in (0, 1), T \geq 0

α > 0

Define

{¯ F}^{1} (\cdot) = - \infty

z_{1} = y_{1}

X_{1} = 0

for

t = 1, 2, \dots T

Compute

v_{t}

and

g_{t}

as in (2) and (3) Compute One Subgradient

Compute

θ_{t}

and

z_{t + 1}

as in (5) and (6) Compute One Step

if then

y_{t + 1} = z_{t + 1}

Take Descent Step

else

y_{t + 1} = y_{t}

end if Take Null Step

Set

{¯ F}^{t + 1} (y) = {¯ F}^{t} (z_{t + 1}) + ⟨ θ_{t} g_{t} + (1 - θ_{t}) \nabla {¯ F}^{t} (z_{t}), y - z_{t} ⟩

Update Model of

F

end for

Algorithm 1 Bundle Method with Cut Aggregation

Extracting Primal Solutions Directly.

A solution to the primal problem (P) can be extracted from the sequence of subgradients $g_{k}$ (as these describe the dual of the dual problem). Set our initial primal solution to be $X_{0} = 0$ . When each iteration updates the model ${¯ F}^{t}$ , we could update our primal variable similarly:

X_{t + 1}

= θ_{t} α v_{t} v_{t}^{*} + (1 - θ_{t}) X_{t} .

(7)

As stated in Theorem 1.1, these primal solutions $X_{t}$ converge to optimality and feasibility at a rate of $O (1 / t)$ . Alas this approach requires us to compute and store the full $n \times n$ matrix $X_{t}$ at each iteration. Assuming every solution is low rank, an ideal method would only require $O (n r)$ memory. The following section shows matrix sketching can accomplish this.

Extracting Primal Solutions Via Matrix Sketching.

Here we describe how the matrix sketching method of tropp2017practical can be used to store an approximation of our primal solution $X_{t} \in S_{+}^{n}$ . First, we draw two matrices with independent standard normal entries

	$Ψ \in R^{n \times k} with k = 2 r + 1;$
	$Φ \in R^{l \times n} with l = 4 r + 3;$

Here $r$

is chosen by the user. It either represents the estimate of the true rank of the primal solution or the user’s computational budget in dealing with larges matrices.

We use $Y_{t}^{C}$ and $Y_{t}^{R}$ to capture the column space and the row space of $X_{t}$ :

Y_{t}^{C} = X_{t} Ψ \in R^{n \times k}, Y_{t}^{R} = Φ X_{t} \in R^{l \times n} .

(8)

Hence we initially have $Y_{0}^{C} = 0$ and $Y_{0}^{R} = 0$ . Notice Algorithm 1 does not observe the matrix $X_{t}$ directly. Rather, it observes a stream of rank one updates

X_{t + 1} = θ_{t} α v_{t} v_{t}^{*} + (1 - θ_{t}) X_{t} .

In this setting, $Y_{t + 1}^{C}$ and $Y_{t + 1}^{R}$ can be directly computed as

			(9)
			(10)

This observation allows us to form the sketch $Y_{t}^{C}$ and $Y_{t}^{R}$ from the stream of updates.

We then reconstruct $X_{t}$ and get the reconstructed matrix ${^X}_{t}$ by

Y_{t}^{C} = Q_{t} R_{t}, B_{t} = (Φ Q_{t})^{†} Y_{t}^{R}, {^X}_{t} = Q_{t} [B_{t}]_{r},

(11)

where $Q_{t} R_{t}$ is the $Q R$ factorization of $Y_{t}^{C}$ and $[\cdot]_{r}$ returns the best rank $r$ approximation in Frobenius norm. Specifically, the best rank $r$ approximation of a matrix $Z$ is $U Σ V^{*}$ , where $U$ and $V$

are right and left singular vectors corresponding to the

r

largest singular values of

Z

and

Σ

is a diagonal matrix with

r

largest singular values of

Z

. In actual implementation, we may only produce the factors

(Q U, Σ, V)

defining

{^X}_{T}

in the end instead reconstructing

{^X}_{t}

in every iteration.

Hence the expensive primal update (7) can be replaced by much more efficient operations (9) and (10). Then a low rank approximation of $X_{t}$ can be computed by (11).

We remark that the reconstructed matrix ${^X}_{t}$ is not necessarily positive definite. However, this suffices for the purpose of finding a matrices close to $X_{t}$ . More sophisticated procedure is available for producing a positive semidefinite approximation of $X_{t}$ (tropp2017practical, , Section 7.3).

3 Numerical Experiments

In this section, we demonstrate Algorithm 1

equipped with the sketching procedure does solve problem instances in four important problem classes: (1) Generalized eigenvalue

(boumal2018deterministic, , Section 5.1), (2)

Z_{2}

synchronization bandeira2018random , (3) Max-Cut goemans1995improved , and (4) matrix completionsrebro2005rank . For all experiments, we set

β = \frac{1}{4}

ρ = 1

, and

α = 2 ⟨ I, X_{⋆} ⟩

where

X_{⋆}

is computed via MOSEK mosek2010mosek or prior knowledge of the problem. We present the results of convergences in Figure 1 for the following problem instances (results for other problems instances are similar for each problem class): Let

D

be the distribution of symmetric matrices in

S^{800}

with upper triangular part (including the diagonal) being independent standard Gaussians.

Generalized eigenvalue (GE): $C \sim D$ , $A (X) = ⟨ B, X ⟩$ for any $X$ , where $B = I + \frac{1}{n} W W^{*}$ and $W \sim D$ and is independent of $C$ , and $b = 1 \in R$ .
$Z_{2}$ synchronization (Z2): $C = J + \frac{1}{2 \sqrt{5}} W$ where $J \in S^{800}$ is the all one matrix and $W \sim D$ , $A = d i a g$ , and $b \in R^{800}$ is the all one vector.
Max-Cut(MCut): $C = - L$ where $L$ is the Laplacian matrix of the G1 graph Gset with $800$ vertices, $A$ and $b$ are the same as $Z_{2}$ synchronization
Matrix Completion(MComp): A random rank $3$ matrix $X^{♮} \in S^{400}$ is generated. The index set $Ω \subset {1, \dots, 400} \times {1, \dots, 400} =: [400]^{2}$ is generated in a way that each $(i, j) \in [400]^{2}$ is in $Ω$
with probability
$0.2$ independently from anything else. Set $C = I \in S^{800}$ . The linear constraint is $X_{n + i, n + j} = X_{i, j}^{♮}$ for each $(i, j) \in Ω$ . So $[A (X)]_{i, j} = X_{i + n, j + n}$ and $b_{i, j} = X_{i, j}^{♮}$ for each $(i, j) \in Ω$ .

As can be seen from the experiments, with $2000$ iterations or less, the dual and primal objective converges fairly fast except the matrix completion problem. The infeasibility measured by $∥ A X - b ∥$ and distance to solution is about $10^{- 2}$ for most of the problems except GE. In general, we note the convergence is quicker when the problems have rank $1$ optimal solutions (GE and Z2) comparing to problems with higher rank optimal solutions (MCut and MComp).

Figure 1: The upper, middle, and lower plots give the different convergence measures (as indicated on the title) for the dual iterates $z_{t}$ and $y_{t}$ , the primal iterates $X_{k}$ , and the sketching iterate ${^X}_{t}$ respectively. The color corresponds to different problem instances as indicated in the legend of the upper plots. Due to numerical error in updating $X_{t}$ and non-asymmetry of ${^X}_{t}$ , we compute the minimum eigenvalue of their symmetrized version. $λ_{min} (\frac{1}{2} (X_{t} + X_{t}^{'}))$ appears to be negative but very small due to numerical error.

References

[1] The university of florida sparse matrix collection: Gset group.
[2] Afonso S Bandeira. Random laplacian matrices and convex relaxations. Foundations of Computational Mathematics, 18(2):345–379, 2018.
[3] Nicolas Boumal, Vladislav Voroninski, and Afonso S Bandeira. Deterministic guarantees for burer-monteiro factorizations of smooth semidefinite programs. Communications on Pure and Applied Mathematics, 2018.
[4] Emmanuel J Candes, Thomas Strohmer, and Vladislav Voroninski. Phaselift: Exact and stable signal recovery from magnitude measurements via convex programming. Communications on Pure and Applied Mathematics, 66(8):1241–1274, 2013.
[5] Lijun Ding, Alp Yurtsever, Volkan Cevher, Joel A Tropp, and Madeleine Udell. An optimal-storage approach to semidefinite programming using approximate complementarity. arXiv preprint arXiv:1902.03373, 2019.
[6] Yu Du and Andrzej Ruszczyński. Rate of convergence of the bundle method. J. Optim. Theory Appl., 173(3):908–922, June 2017.
[7] Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42(6):1115–1145, 1995.
[8] APS Mosek. The mosek optimization software. Online at http://www. mosek. com, 54(2-1):5, 2010.
[9] Gábor Pataki. On the rank of extreme matrices in semidefinite programs and the multiplicity of optimal eigenvalues. Mathematics of operations research, 23(2):339–358, 1998.
[10] Andrzej P Ruszczyński and Andrzej Ruszczynski. Nonlinear optimization, volume 13. Princeton university press, 2006.
[11] Nathan Srebro and Adi Shraibman. Rank, trace-norm and max-norm. In
International Conference on Computational Learning Theory
, pages 545–560. Springer, 2005.
[12] Jos F Sturm. Error bounds for linear matrix inequalities. SIAM Journal on Optimization, 10(4):1228–1248, 2000.
[13] Joel A Tropp, Alp Yurtsever, Madeleine Udell, and Volkan Cevher. Practical sketching algorithms for low-rank matrix approximation. SIAM Journal on Matrix Analysis and Applications, 38(4):1454–1485, 2017.
[14] Joel A Tropp, Alp Yurtsever, Madeleine Udell, and Volkan Cevher. Randomized single-view algorithms for low-rank matrix approximation. 2017.

Appendix A Proofs of Convergence Guarantees

a.1 Auxiliary lemmas

Lemma A.1.

(Sketching Guarantee)[14, Theorem 5.1] Fix a target rank $r$ . Let $X$ be a matrix, and let $(Y, W)$ be a sketch of $X$ of the form (8). The procedure (11) yields a rank- $r$ matrix $^X$ with

Here is the Frobenius norm. Similar operator $2$ -norm bounds hold with high probability.

Lemma A.2 (Compact sublevel set).

If a convex lower semicontinuous function $f (x) : R^{n} \to R \cup {\infty}$ has a compact nonempty solution set, then all of its sublevel set is compact.

Proof.

Suppose for some $L \in R$ , then the closed sublevel set $S_{L} = {x ∣ f (x) \leq L}$ is unbounded. Then there is a unit direction vector $γ \in R$ such that for all $x \in S_{L}, α \geq 0$ , $x + α γ \in S_{L}$ . This in particular violates the fact the solution set is bounded and the proof is completed. ∎

Lemma A.3 (Quadratic Growth).

[12, Section 4] If the solution sets $X_{⋆}$ and $Y_{⋆}$ are compact and strict complementarity holds, then for any fixed $ϵ > 0$ , there are some $γ_{1}, γ_{2}$ such that for all $y$ with $F (y) \leq ⟨ - b, y_{⋆} ⟩ + ϵ$ with $α = 2 {sup}_{X_{⋆} \in X_{⋆}} trace (X_{⋆})$ , and all $X ⪰ 0$ with $| ⟨ C, X ⟩ - ⟨ C, X_{⋆} ⟩ | \leq ϵ$ and :

Proof.

Using the proof of Lemma A.5, we find that $λ_{min} (C - A^{*} (y)) \geq - \frac{ϵ}{{sup}_{X_{⋆} \in X_{⋆}} trace X_{⋆}}$ . Thus $⟨ - b, y ⟩ \leq ⟨ - b, y_{⋆} ⟩ + 3 ϵ$ . The result in [12, Section 4] requires the set $S_{1} = {y ∣ F (y) \leq ⟨ - b, y_{⋆} ⟩ + ϵ}$ , and the set being compact. Using [10, Theorem 7.21], the optimization problem has the same solution set as the primal SDP (P) for some large $γ > 0$ . Thus the compactness of the set $S_{1}$ and $S_{2}$ is ensured by Lemma A.2, and the proof is completed. ∎

a.2 Proof of Theorem 1.1

Let $D_{X_{⋆}} \geq {sup}_{X_{⋆} \in X_{⋆}} t r a c e (X_{⋆})$ and $D_{y} \geq {sup}_{F (y) \leq F (y_{0})} ∥ y ∥_{2} .$ Then we set the bundle method’s parameters to be $α = 2 D_{X_{⋆}}$ , $β = 1 / 2$ , and $ρ = ϵ / D_{y}^{2}$ . We recall the inner product for matrices is the trace inner product and is the dot product for the vectors.

In the following three lemmas, we prove bounds on primal feasibility, dual feasibility, and optimality in terms of $F (y_{t}) - F (y^{*})$ . From this, we can conclude these quantities converge at the claimed rate since the Du and Ruszcynskii [6] recently showed the bundle method has $F (y_{t}) - F (y^{*})$ converge at a $˜ O (1 / ϵ)$ rate.

Lemma A.4 (Primal Feasibility).

At every descent step $t$ , we have approximate primal feasibility

X_{t + 1} ⪰ 0,

∥ b - A X_{t + 1} ∥ \leq \sqrt{\frac{2 ϵ (F (y_{t}) - F (y^{*}))}{β D_{y}^{2}}} .

Proof.

Noting that $X_{t + 1}$ is built out of a convex combination of the rank matrices $v_{t} v_{t}^{T}$ , its immediate that it is always a positive semidefinite matrix.

The definition of $X_{t}$ immediately gives the following alternative characterization of ${¯ F}^{t + 1}$ ,

{¯ F}^{t + 1} (y) = - ⟨ C, X_{t + 1} ⟩ - ⟨ b - A X_{t + 1}, y ⟩ .

Since we constructed ${¯ F}^{t + 1}$ to correspond to the first-order optimality condition of the subproblem (4), we have

0 = \nabla {¯ F}^{t + 1} (y_{t + 1}) + ρ (z_{t + 1} - y_{t}) .

Hence $∥ b - A X_{t + 1} ∥^{2} = ρ^{2} ∥ y_{t + 1} - y_{t} ∥^{2}$ . The distance traveled during any descent step can be bounded by the objective value gap as

\frac{ρ}{2} ∥ y_{t + 1} - y_{t} ∥^{2} \leq F (y_{t}) - {˜ F}^{t} (y_{t + 1}) \leq \frac{F (y_{t}) - F (y_{t + 1})}{β} \leq \frac{F (y_{t}) - F (y_{⋆})}{β}

where the first inequality uses the fact that $z_{t + 1}$ minimizes ${˜ F}^{t} (\cdot) + \frac{ρ}{2} ∥ \cdot - y_{t} ∥$ and the second inequality uses the definition of a descent step. Combining this with our feasibility bound shows

∥ b - A X_{t + 1} ∥^{2} \leq \frac{2 ρ (F (y_{t}) - F (y^{*}))}{β} .

Then our choice of $ρ$ completes the proof. ∎

Lemma A.5 (Dual Feasibility).

At every descent step $t$ , we have approximate dual feasibility

λ_{min} (C - A^{*} y_{t + 1}) \geq \frac{- (F (y_{t}) - F (y_{⋆}))}{D_{X_{⋆}}} .

Proof.

Standard strong duality and exact penalization arguments show for any $X_{⋆} \in X_{⋆}$ ,

	$⟨ b, y_{t + 1} - y_{⋆} ⟩$	$= ⟨ A X_{⋆}, y_{t + 1} ⟩ - ⟨ C, X_{⋆} ⟩$
		$\leq ⟨ X_{⋆}, A^{*} (y_{t + 1} - C) ⟩$
		$\leq - trace (X_{⋆}) min {λ_{min} (C - A^{*} y_{t + 1}), 0} .$

Recalling our assumption that $α \geq 2 D \geq 2 trace (X_{⋆})$ yields the claimed feasibility bound

	$F (y_{t}) - F (y_{⋆}) \geq F (y_{t + 1}) - F (y_{⋆})$	$= ⟨ - b, y_{t + 1} - y_{⋆} ⟩ - α min {λ_{min} (C - A^{*} y_{t + 1}), 0}$
		$≥−trace(X⋆)min{λmin(C−A∗yt+1),0}.\qed$

Lemma A.6 (Primal-Dual Optimality).

At every descent step $t$ , we have approximate primal-dual optimality bounded above by

and below by

⟨ b, y_{t + 1} ⟩ - ⟨ C, X_{t + 1} ⟩ \geq - \frac{1 - β}{β} (F (y_{t}) - F (y_{⋆})) - \sqrt{\frac{2 ϵ (F (y_{t}) - F (y_{⋆}))}{β}} .

Proof.

The standard duality analysis shows the primal-dual objective gap equals

	$⟨ b, y_{t + 1} ⟩ - ⟨ C, X_{t + 1} ⟩$	$= ⟨ A X_{t + 1}, y_{t + 1} ⟩ - ⟨ C, X_{t + 1} ⟩ + ⟨ b - A X_{t + 1}, y_{t + 1} ⟩$
		$= ⟨ X_{t + 1}, A^{*} y_{t + 1} - C ⟩ + ⟨ b - A X_{t + 1}, y_{t + 1} ⟩ .$

Notice that the second term here is bounded above and below as

| ⟨ b - A X_{t + 1}, y_{t + 1} ⟩ | \leq \sqrt{\frac{2 ϵ (F (y_{t}) - F (y_{⋆}))}{β D_{y}^{2}}} ∥ y_{t + 1} ∥_{2} \leq \sqrt{\frac{2 ϵ (F (y_{t}) - F (y_{⋆}))}{β}}

by Lemma A.4. Hence we only need to show that the first term also approaches zero (that is, we approach holding complementary slackness).

An upper bound on this inner product follows from Lemma A.5 as

⟨ X_{t + 1}, A^{*} y_{t + 1} - C ⟩ \leq - trace (X_{t + 1}) min {λ_{min} (C - A^{*} y_{t + 1}), 0} \leq \frac{t r a c e (X_{t + 1}) (F (y_{t}) - F (y_{⋆}))}{D_{X_{⋆}}} .

Hence

A lower bound on this inner product follows as

	$\frac{1 - β}{β} (F (y_{t}) - F (y_{t + 1}))$	$\geq F (y_{t + 1}) - {˜ F}^{t} (y_{t + 1})$
		$= F (y_{t + 1}) - {¯ F}^{t + 1} (y_{t + 1})$
		$= - α min {λ_{min} (C - A^{*} y_{t + 1}), 0} + ⟨ C, X_{t + 1} ⟩ - ⟨ A X_{t + 1}, y_{t + 1} ⟩$
		$\geq ⟨ X_{t + 1}, C - A^{*} y_{t + 1} ⟩,$

where the first inequality follows from the definition of a descent step. Hence

⟨ b, y_{t + 1} ⟩ - ⟨ C, X_{t + 1} ⟩

\geq - \frac{1 - β}{β} (F (y_{t}) - F (y_{⋆})) - \sqrt{\frac{2 ϵ (F (y_{t}) - F (y_{⋆}))}{β}} .

∎

a.3 Proof of Theorem 1.2

Using triangle inequality, we see that

(12)

The first term of (12) is bounded by

where step $(a)$ is due to Lemma A.1, and step $(b)$ is because $X_{⋆} \in X_{⋆}$ all has rank less than or equal to $r$ and $[X_{t}]_{r}$ is the best rank $r$ approximation of $X_{t}$ in terms of Frobenius norm.

Combining the above inequalities, we see that

Edist\tiny{F}(^Xt,X⋆)≤(1+3√2)dist\tiny{F}(Xt,X⋆).

(13)

Our task now is to have an estimate the rates convergence of $dist\tiny{F}(Xt,X⋆)$ . Denote the shorthand . Using Lemma A.3, there are some $γ$ such that for all $X$ with $g (X) \leq g (X_{⋆}) + 1$ satisfy

dist\tiny{F}2(X,X⋆)≤γ(g(X)−g(X⋆)).

(14)

Then the theorem follows from Theorem 1.1 as $g (X_{t}) - g (X_{⋆})$ converges at a rate of $˜ O (1 / ϵ)$ .

Bundle Method Sketching for Low Rank Semidefinite Programming

Authors

A relaxed interior point method for low-rank semidefinite programming problems

On the regularity and conditioning of low rank semidefinite programs

An Optimal-Storage Approach to Semidefinite Programming using Approximate Complementarity

Low-rank semidefinite programming for the MAX2SAT problem

Primal-Dual Block Frank-Wolfe

Robust Learning from Noisy Side-information by Semidefinite Programming

The Mixing method: coordinate descent for low-rank semidefinite programming

1 Introduction

Our Contributions.

Theorem 1.1 (Primal-Dual Convergence).

Theorem 1.2 (Sketched Solution Convergence).

2 Defining The Sketching Bundle Method

Extracting Primal Solutions Directly.

Extracting Primal Solutions Via Matrix Sketching.

3 Numerical Experiments

References

Appendix A Proofs of Convergence Guarantees

a.1 Auxiliary lemmas

Lemma A.1.

Lemma A.2 (Compact sublevel set).

Proof.

Lemma A.3 (Quadratic Growth).

Proof.

a.2 Proof of Theorem 1.1

Lemma A.4 (Primal Feasibility).

Proof.

Lemma A.5 (Dual Feasibility).

Proof.

Lemma A.6 (Primal-Dual Optimality).

Proof.

a.3 Proof of Theorem 1.2

Bundle Method Sketching for Low Rank Semidefinite Programming

Authors

A relaxed interior point method for low-rank semidefinite programming problems

On the regularity and conditioning of low rank semidefinite programs

An Optimal-Storage Approach to Semidefinite Programming using Approximate Complementarity

Low-rank semidefinite programming for the MAX2SAT problem

Primal-Dual Block Frank-Wolfe

Robust Learning from Noisy Side-information by Semidefinite Programming

The Mixing method: coordinate descent for low-rank semidefinite programming

This week in AI

1 Introduction

Our Contributions.

Theorem 1.1 (Primal-Dual Convergence).

Theorem 1.2 (Sketched Solution Convergence).

2 Defining The Sketching Bundle Method

Extracting Primal Solutions Directly.

Extracting Primal Solutions Via Matrix Sketching.

3 Numerical Experiments

References

Appendix A Proofs of Convergence Guarantees

a.1 Auxiliary lemmas

Lemma A.1.

Lemma A.2 (Compact sublevel set).

Proof.

Lemma A.3 (Quadratic Growth).

Proof.

a.2 Proof of Theorem 1.1

Lemma A.4 (Primal Feasibility).

Proof.

Lemma A.5 (Dual Feasibility).

Proof.

Lemma A.6 (Primal-Dual Optimality).

Proof.

a.3 Proof of Theorem 1.2