Quasi-Newton Methods for Saddle Point Problems

1 Introduction

In this paper, we focus on the following smoothed saddle point problem

min x \in R^{n_{x}} max y \in R^{n_{y}} f (x, y),

(1)

where $f$ is strongly-convex in $x$ and strongly-concave in $y$ . We target to find the saddle point $(x^{*}, y^{*})$ which holds that

f (x^{*}, y) \leq f (x^{*}, y^{*}) \leq f (x, y^{*})

for all $x \in R^{n_{x}}$ and $y \in R^{n_{y}}$

. This formulation contains a lot of scenarios including game theory

[2, 36], AUC maximization [14, 40], robust optimization [3, 13, 33], empirical risk minimization [42]

and reinforcement learning

[11].

There are a great number of first-order optimization algorithms for solving problem (1), including extragradient method [18, 35], optimistic gradient descent ascent [8], proximal point method [28] and dual extrapolation [23]. These algorithms iterate with first-order oracle and achieve linear convergence. Lin et al. [21], Wang and Li [37] used Catalyst acceleration to reduce the complexity for unbalanced saddle point problem, nearly matching the lower bound of first-order algorithms [25, 41] in specific assumptions. Compared with first-order methods, second-order methods usually enjoy superior convergence in numerical optimization. Huang et al. [16] extended cubic regularized Newton (CRN) method [23, 22] to solve saddle point problem (1), which has quadratic local convergence. However, each iteration of CRN requires accessing the exact Hessian matrix and solving the corresponding linear systems. These steps arise $O ((n_{x} + n_{y})^{3})$ time complexity, which is too expensive for high dimensional problems.

Quasi-Newton methods [6, 5, 34, 4, 9] are popular ways to avoid accessing exact second-order information applied in standard Newton methods. They approximate the Hessian matrix based on the Broyden family updating formulas [4], which significantly reduces the computational cost. These algorithms are well studied for convex optimization. The famous quasi-Newton methods including Davidon-Fletcher-Powell (DFP) method [9, 12], Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [6, 5, 34] and symmetric rank 1 (SR1) method [4, 9] enjoy local superlinear convergence [27, 7, 10] when the objective function is strongly-convex. Recently, Rodomanov and Nesterov [29, 30, 31] proposed greedy variant of quasi-Newton methods, which first achieves non-asymptotic superlinear convergence. Later, Lin et al. [20] established a better convergence rate which is condition-number-free. Jin and Mokhtari [17], Ye et al. [39] showed the non-asymptotic superlinear convergence rate also holds for classical DFP, BFGS and SR1 methods.

In this paper, we study quasi-Newton methods for saddle point problem (1). Noticing the Hessian matrix of our objective function is indefinite, the existing Broyden family update formulas and their convergence analysis cannot be applied directly. To overcome this issue, we propose a variant framework of greedy quasi-Newton methods for saddle point problems, which approximates the square of the Hessian matrix during the iteration. Our theoretical analysis characterizes the convergence rate by the Euclidean distance to the saddle point, rather than the weighted norm of gradient used in convex optimization [29, 30, 31, 20, 39, 17]. We summarize the theoretical results for proposed algorithms in Table 1. The local convergence behaviors for all of the algorithms have two periods. The first period has $k_{0}$ iterations with a linear convergence rate $O ({(1 - \frac{1}{κ^{2}})}^{k_{0}})$ . The second one enjoys superlinear convergence:

For general Broyden family methods, we have an explicit rate $O ({(1 - \frac{1}{n κ^{2}})}^{k (k - 1) / 2})$ .
For BFGS method and SR1 method, we have the faster explicit rate $O ({(1 - \frac{1}{n})}^{k (k - 1) / 2})$ , which is condition-number-free.

Additionally, our ideas also can be used for solving general non-linear equations.

Paper Organization

In Section 2, we start with notation and preliminaries throughout this paper. In Section 3, we first propose greedy quasi-Newton methods for quadratic saddle point problem which enjoys local superlinear convergence. Then we extend it to solve general strongly-convex-strongly-concave saddle point problems. In Section 4, we show our theory also can be applied to solve more general non-linear equations and give the corresponding convergence analysis. We conclude our work in Section 5. All proofs are deferred to appendix.

Algorithms	Upper Bound of $λ_{k + k_{0}}$	$k_{0}$
Broyden family (Algorithm 5)	${(1 - \frac{1}{n κ^{2}})}^{k (k - 1) / 2} {(\frac{1}{2})}^{k} {(1 - \frac{1}{4 κ^{2}})}^{k_{0}}$	$O (n κ^{2} ln (n κ))$
BFGS/SR1 (Algorithm 6/7)	${(1 - \frac{1}{n})}^{k (k - 1) / 2} {(\frac{1}{2})}^{k} {(1 - \frac{}{1} 4 κ^{2})}^{k_{0}}$	$O (max {n, κ^{2}} ln (n κ))$

Table 1: We summarize the convergence behaviors of three proposed algorithms for solving saddle point problem in the view of Euclidean distance

λ_{k + k_{0}} def = {∥ ∥ x_{k + k_{0}} - x^{*} ∥ ∥}_{k + k_{0}}^{2} + {∥ ∥ y_{k + k_{0}} - y^{*} ∥ ∥}_{k + k_{0}}^{2}

after

(k + k_{0})

iterations, where

n def = n_{x} + n_{y}

is dimensions of the problem and

κ

is the condition number. The results come from Corollary 3.17.1.

2 Notation and Preliminaries

We use $∥ \cdot ∥$

to present spectral norm and Euclidean norm of matrix and vector respectively. We denote the standard basis for

R^{n}

{e_{1}, \dots, e_{n}}

and let

I

be the identity matrix. The trace of a square matrix is denoted by

tr (\cdot)

. Given two positive definite matrices

G

and

H

, we define their inner product as

⟨ G, H ⟩ def = tr (G H) .

We introduce the following notation to measure how well does matrix $G$ approximate matrix $H$ :

σ_{H} (G) def = ⟨ H^{- 1}, G - H ⟩ = ⟨ H^{- 1}, G ⟩ - n .

(2)

If we further suppose $G ⪰ H$ , it holds that

G - H ⪯ ⟨ H^{- 1}, G - H ⟩ H = σ_{H} (G) H

(3)

by Rodomanov and Nesterov [29].

Using the notation of problem (1), we let $z = [x; y] \in R^{n}$ where $n = n_{x} + n_{y}$ and denote the gradient and Hessian matrix of $f$ at $(x, y)$ as

g (z) def = [\begin{matrix} \nabla_{x} f (x, y) \nabla_{y} f (x, y) \end{matrix}] \in R^{n} and^H (z) def = \nabla^{2} f (x, y) = [\begin{matrix} \nabla_{x x}^{2} f (x, y) & \nabla_{x y}^{2} f (x, y) \nabla_{y x}^{2} f (x, y) & \nabla_{y y}^{2} f (x, y) \end{matrix}] \in R^{n \times n} .

We suppose the saddle point problem (1) satisfies the following assumptions.

Assumption 2.1.

The objective function $f (x, y)$ is twice differentiable and has $L$ -Lipschitz continuous gradient and $L_{2}$ -Lipschitz continuous Hessian , i.e., there exists constants $L > 0$ and $L_{2} > 0$ such that

∥ ∥ g (z) - g (z^{'}) ∥ ∥ \leq L ∥ ∥ z - z^{'} ∥ ∥

(4)

and

∥ ∥^H (z) -^H (z^{'}) ∥ ∥ \leq L_{2} ∥ ∥ z - z^{'} ∥ ∥ .

(5)

for any $z = [x; y], z^{'} = [x^{'}; y^{'}] \in R^{n}$ .

Assumption 2.2.

The objective function $f (x, y)$ is twice differentiable, $μ$ -strongly-convex in $x$ and $μ$ -strongly-concave in $y$ , i.e., there exists $μ > 0$ such that $\nabla_{x x}^{2} f (x, y) ⪰ μ I$ and $\nabla_{y y}^{2} f (x, y) ⪯ - μ I$ for any $(x, y), (x^{'}, y^{'}) \in R^{n_{x}} \times R^{n_{y}}$ .

Note that inequality (4) means the spectral norm of Hessian matrix $^H (z)$ can be upper bounded, that is

∥^H (z) ∥ \leq L .

(6)

for all $z \in R^{n}$ . Additionally, the condition number of the objective function is defined as

κ def = \frac{L}{μ} .

3 Quasi-Newton Methods for Saddle Point Problems

The update rule of standard Newton’s method for solving problem (1) can be written as

z_{+} = z - (^H (z))^{- 1} g (z) .

(7)

This iteration scheme has quadratic local convergence, but solving linear system (7) takes $O (n^{3})$ time complexity. For convex minimization, quasi-Newton methods including BFGS/SR1 [6, 5, 34, 4, 9] and their variants [20, 39, 29, 30] focus on approximating the Hessian and reduce the computational cost to $O (n^{2})$ for each round. However, all of these algorithms and related convergence analysis are based on the assumption that the Hessian matrix is positive definite, which is not suitable for our saddle point problems since $^H (z)$ is indefinite.

We introduce the auxiliary matrix $H (z)$ be the square of Hessian

H (z) def = {(^H (z))}^{2} .

The following lemma means $H (z)$ is always positive definite.

Lemma 3.1.

Under Assumption 2.1 and 2.2, we have

μ^{2} I ⪯ H (z) ⪯ L^{2} I

(8)

for all $z \in R^{n}$ , where $H (z) = (^H (z))^{2}$ .

Hence, we can reformulate the update of Newton’s method (7) by

(9)

Then it is natural to characterize the second-order information by estimating the auxiliary matrix $H (z)$ , rather than the indefinite Hessian $^H (z)$ . If we have obtained a symmetric positive definite matrix $G \in R^{n \times n}$ as an estimator for $H (z)$ , the update rule of (9) can be approximated by

z_{+} = z - G^{- 1}^H (z) g (z) .

(10)

The remainder of this section introduce several strategies to construct $G$ , resulting the quasi-Newton methods for saddle point problem with local superlinear convergence. We should point out the implementation of iteration (10) is unnecessary to construct Hessian matrix $^H (z)$ explicitly, since we are only interested in the Hessian-vector product $^H (z) g (z)$ , which can be computed efficiently [26, 32].

3.1 The Broyden Family Updates

We first introduce the Broyden family [24, Section 6.3] of quasi-Newton updates for approximating an positive definite matrix $H \in R^{n \times n}$ by using the information of current estimator $G \in R^{n \times n}$ .

Definition 3.2.

Suppose two positive definite matrices $H, G \in R^{n \times n}$ satisfy $H ⪯ G$ . For any $u \in R^{n}$ , if $G u = H u$ , we define ${B r o y d}_{τ} (G, H, u) d e f = H$ . Otherwise, we define

	${B r o y d}_{τ} (G, H, u) d e f =$	$τ [G - \frac{H u u^{⊤} G + G u u^{⊤} H}{u^{⊤} H u} + (\frac{u^{⊤} G u}{u^{⊤} H u} + 1) \frac{H u u^{⊤} H}{u^{⊤} H u}]$		(11)
		$+ (1 - τ) [G - \frac{(G - H) u u^{⊤} (G - H)}{u^{⊤} (G - H) u}] .$

The different choices of parameter $τ$ for formula (11) contain several popular quasi-Newton updates:

For $τ = \frac{u^{⊤} A u}{u^{⊤} G u} \in [0, 1]$ , it corresponds to BFGS update

$BFGS (G, H, u) def = G - \frac{G u u^{⊤} G}{u^{⊤} G u} + \frac{H u u^{⊤} H}{u^{⊤} H u} .$ (12)
For $τ = 0$ , it corresponds to SR1 update

$S R 1 (G, H, u) d e f = G - \frac{(G - H) u u^{⊤} (G - H)}{u^{⊤} (G - H) u} .$ (13)

The general Broyden family update as Definition 3.2 has the following properties.

Lemma 3.3 ([29, Lemma 2.1 and Lemma 2.2]).

Suppose two positive definite matrices $H, G \in R^{n \times n}$ satisfy $H ⪯ G ⪯ η H$ for some $η \geq 1$ , then for any $u \in R^{n}$ and $τ_{1} < τ_{2}$ , we have

{B r o y d}_{τ_{1}} (G, H, u) ⪯ {B r o y d}_{τ_{2}} (G, H, u) .

Additionally, for any $τ \in [0, 1]$ , we have

H ⪯ {B r o y d}_{τ} (G, H, u) ⪯ η H .

(14)

We first introduce the greedy update method [29] by choosing $u$ as follows

\begin{matrix} u_{H} (G) & def = a r g m a x u \in {e_{1}, \dots, e_{n}} \frac{u^{⊤} (G - H) u}{u^{⊤} H u} = a r g m a x u \in {e_{1}, \dots, e_{n}} \frac{u^{⊤} G u}{u^{⊤} H u} . \end{matrix}

(15)

The following lemma shows applying update rule (11) with formula (15) leads to a new estimator with tighter error bound in the measure of $σ_{H} (\cdot)$ .

Lemma 3.4 ([29, Theorem 2.5]).

Suppose two positive definite matrices $H, G \in R^{n \times n}$ satisfy $H ⪯ G ⪯ κ^{2} H$ . Let $G_{+} = {B r o y d}_{τ} (G, H, u_{H} (G))$ , then for any $τ \in [0, 1]$ , we have

σ_{H} (G_{+}) \leq (1 - \frac{1}{n κ^{2}}) σ_{H} (G)

(16)

For specific Broyden family updates BFGS and SR1 shown in (12) and (13), we can replace (15) by scaling greedy direction [20], which leads to a better convergence result. Concretely, for BFGS method, we first find $L$ such that $G^{- 1} = L^{⊤} L$ , where $L$ is an upper triangular matrix. This step can be implemented with $O (n^{2})$ complexity [20, Proposition 15]. We present the subroutine for factorizing $G^{- 1}$ in Algorithm 1 and give its detailed implementation in Appendix B.

Then we use direction $L^{⊤} {~ u}_{H} (L)$ for BFGS update, where

{~ u}_{H} (L) d e f = a r g m a x u \in {e_{1}, \dots, e_{n}} u^{⊤} L^{- ⊤} H^{- 1} L^{- 1} u .

(17)

For SR1 method, we choose the direction by

{¯ u}_{H} (G) d e f = a r g m a x u \in {e_{1}, \dots, e_{n}} \frac{u^{⊤} (G - H)^{2} u}{u^{⊤} (G - H) u} .

(18)

Applying the BFGS update rule (12) with formula (17), we obtain a condition-number-free result as follows.

Lemma 3.5 ([20, Theorem 2.6]).

Suppose two positive definite matrices $H, G \in R^{n \times n}$ satisfy $H ⪯ G$ . Let $G_{+} = B F G S (G, H, L^{⊤} {~ u}_{H} (L))$ , where $L$ is an upper triangular matrix such that $G^{- 1} = L^{⊤} L$ . Then we have

σ_{H} (G_{+}) \leq (1 - \frac{1}{n}) σ_{H} (G) .

(19)

The effect of SR1 update can be characterized by the following measure

τ_{H} (G) d e f = t r (G - H) .

Applying the SR1 update rule (13) with formula (18), we also hold a condition-number-free result.

Lemma 3.6 ([20, Theorem 2.3]).

Suppose two positive definite matrices $H, G \in R^{n \times n}$ satisfy $H ⪯ G$ . Let $G_{+} = S R 1 (G, H, {¯ u}_{H} (G))$ , then we have

τ_{H} (G_{+}) \leq (1 - \frac{1}{n}) τ_{H} (G) .

(20)

1: Input: positive definite matrix

H \in R^{n \times n}

, upper triangular matrix

L \in R^{n}

, greedy direction

u \in R^{n}

[Q, R] = Q R (L (I - \frac{H u u^{⊤}}{u^{⊤} H u}))

v = \frac{u}{\sqrt{u^{⊤} H u}}

[Q^{'}, R^{'}] = Q R ([\begin{matrix} v^{⊤} R \end{matrix}])

5: Output:

^L = R^{'}

Algorithm 1 Fast-Cholesky

(H, L, u)

1: Input:

G_{0} = L^{2} I

and

τ_{k} \in [0, 1]

2: for

k = 0, 1, \dots

z_{k + 1} = z_{k} - G_{k}^{- 1}^H g (z_{k})

u_{k} = u_{H} (G_{k})

G_{k + 1} = {B r o y d}_{τ_{k}} (G_{k}, H, u_{k})

6: end for

Algorithm 2 Greedy Broyden Class Method for Quadratic Problems

1: Input:

G_{0} = L^{2} I

and

L_{0} = L^{- 1} I

2: for

k = 0, 1 \dots

z_{k + 1} = z_{k} - G_{k}^{- 1}^H g (z_{k})

u_{k} = L_{k}^{⊤} {~ u}_{H} (L_{k})

G_{k + 1} = B F G S (G_{k}, H, u_{k})

L_{k + 1} =

Fast-Cholesky

(H, L_{k}, u_{k})

7: end for

Algorithm 3 Greedy BFGS Method for Quadratic Problem

1: Input:

G_{0} = L^{2} I

and

K \leq n

2: for

k = 0, 1 \dots, K

z_{k + 1} = z_{k} - G_{k}^{- 1}^H g (z_{k})

u_{k} = {¯ u}_{H} (G_{k})

G_{k + 1} = S R 1 (G_{k}, H, u_{k})

6: end for

Algorithm 4 Greedy SR1 method for Quadratic Problem

3.2 Algorithms for Quadratic Saddle Point Problems

Then we consider solving quadratic saddle point problem of the form

min x \in R^{n_{x}} max y \in R^{n_{y}} f (x, y) def = \frac{1}{2} [\begin{matrix} x^{⊤} y^{⊤} \end{matrix}] A [\begin{matrix} x y \end{matrix}] - b^{⊤} [\begin{matrix} x y \end{matrix}],

(21)

where $b \in R^{n}$ , $A \in R^{n \times n}$ is symmetric and $n = n_{x} + n_{y}$ . We suppose $A$ could be partitioned as

A = [\begin{matrix} A_{x x} & A_{x y} A_{y x} & A_{y y} \end{matrix}] \in R^{n \times n}

where the sub-matrices $A_{x x} \in R^{n_{x} \times n_{x}}$ , $A_{x y} \in R^{n_{x} \times n_{y}}$ , $A_{y x} \in R^{n_{y} \times n_{x}}$ and $A_{y y} \in R^{n_{y} \times n_{y}}$ satisfy $A_{x x} ⪰ μ I$ , $A_{y y} ⪯ - μ I$ and $∥ A ∥ \leq L$ . Recall the condition number is defined as $κ def = L / μ$ . Using notations introduced in Section 2, we have

z = [x; y], g (z) = A z - b,^H def =^H (z) = A and H def = H (z) = A^{2} .

We present the detailed procedure of greedy quasi-Newton methods for quadratic saddle point problem by using Broyden family update, BFGS update and SR1 update in Algorithm 2, 3 and 4 respectively. We define $λ_{k}$ as the Euclidean distance from $z_{k}$ to the saddle point $z^{*}$ for our convergence analysis, that is

λ_{k} def = ∥ z_{k} - z^{*} ∥ .

The definition of $λ_{k}$ in this paper is different from the measure used in convex optimization [29, 20]¹¹1In later section, we will see the measure $λ_{k} def = ∥ z_{k} - z^{*} ∥$ is suitable to convergence analysis of quasi-Newton methods for saddle point problems., but it also holds the similar property as follows.

Lemma 3.7.

Assume we have $η_{k} \geq 1$ and $G_{k} \in R^{n \times n}$ such that $H ⪯ G_{k} ⪯ η_{k} H$ for Algorithm 2, 3 and 4, then we have

λ_{k + 1} \leq (1 - \frac{1}{η_{k}}) λ_{k} .

The next theorem states the assumptions of Lemma 3.7 always holds with $η_{k} = κ^{2} \geq 1$ , which means $λ_{k}$ converges to 0 linearly.

Theorem 3.8.

For all $k \geq 0$ , Algorithm 2, 3 and 4 holds that

H ⪯ G_{k} ⪯ κ^{2} H,

(22)

and

λ_{k} \leq {(1 - \frac{1}{κ^{2}})}^{k} λ_{0} .

(23)

Lemma 3.7 also implies superlinear convergence can be obtained if there exists $η_{k}$ which converges to 1. Applying Lemma 3.4, 3.5 and 3.6, we can show it holds for proposed algorithms.

Theorem 3.9.

Solving quadratic saddle point problem (21) by proposed greedy quasi-Newton algorithms, we have the following results:

For greedy Broyden family method (Algorithm 2), we have

$H ⪯ G_{k} ⪯ (1 + {(1 - \frac{1}{n κ^{2}})}^{k} n κ^{2}) H and λ_{k + 1} \leq {(1 - \frac{1}{n κ^{2}})}^{k} n κ^{2} λ_{k} for % all k \geq 0.$
For greedy BFGS method (Algorithm 3), we have
For greedy SR1 method (Algorithm 4), we have

Combing the results of Theorem 3.8 and 3.9, we achieve the two-stages convergence behavior, that is, the algorithm has global linear convergence and local superlinear convergence. The formal description is summarized as follows.

Corollary 3.9.1.

Solving quadratic saddle point problem (21) by proposed greedy quasi-Newton algorithms, we have the following results:

Using greedy Broyden family method (Algorithm 2), we have

$λ_{k_{0} + k} \leq {(1 - \frac{1}{n κ^{2}})}^{\frac{k (k - 1)}{2}} {(\frac{1}{2})}^{k} {(1 - \frac{1}{κ^{2}})}^{k_{0}} λ_{0} for all k > 0 and k_{0} = O (n κ^{2} ln (n κ^{2})) .$ (24)
Using greedy BFGS method (Algorithm 3), we have

$λ_{k_{0} + k} \leq {(1 - \frac{1}{n})}^{\frac{k (k - 1)}{2}} {(\frac{1}{2})}^{k} {(1 - \frac{1}{κ^{2}})}^{k_{0}} λ_{0} for all k > 0 and k_{0} = O (n ln (n κ^{2})) .$ (25)
Using greedy SR1 method (Algorithm 4), we have

$λ_{k} \leq ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} {(1 - \frac{1}{κ^{2}})}^{k} λ_{0} & 0 \leq k \leq k_{0}, \frac{(n - 1)!}{(n - k + k_{0} - 1)!} {(\frac{1}{2 n})}^{k - k_{0}} {(1 - \frac{1}{κ^{2}})}^{k_{0}} & k_{0} < k < n, 0 & k = n, \end{matrix}$ (26)

where $k_{0} = ⌈ (1 - \frac{1}{2 n κ^{4}}) n ⌉$ .

3.3 Algorithms for General Saddle Point Problems

In this section, we consider the general saddle point problem

min x \in R^{n_{x}} max y \in R^{n_{y}} f (x, y) .

where $f (x, y)$ satisfies Assumption 2.1 and 2.2. We target to propose quasi-Newton methods for solving above problem with local superlinear convergence and $O (n^{2})$ time complexity for each iteration.

3.3.1 Algorithms

The key idea of designing quasi-Newton methods for saddle point problems is characterizing the second-order information by approximating the auxiliary matrix $H (z) def = (^H (z))^{2}$ . Note that we have supposed the Hessian of $f$ is Lipschitz continuous and bounded in Assumption 2.1 and 2.2, which means the auxiliary matrix operator $H (z)$ is also Lipschitz continuous.

Lemma 3.10.

Under Assumption 2.1 and 2.2, we have $H (z)$ is $2 L L_{2}$ -Lipschitz continuous, that is

∥ H (z) - H (z^{'}) ∥ \leq 2 L_{2} L ∥ z - z^{'} ∥ .

(27)

Combining Lemma 3.1 and 3.10, we achieve the following properties of $H (z)$ , which analogizes the strongly self-concordance in convex optimization [29].

Lemma 3.11.

Under Assumption 2.1 and 2.2, the auxiliary matrix operator $H (\cdot)$ satisfies

H (z) - H (z^{'}) ⪯ M ∥ z - z^{'} ∥ H (w),

(28)

for all $z, z^{'}, w \in R^{n}$ , where $M = 2 κ^{2} L_{2} / L$ .

Corollary 3.11.1.

Let $z, z^{'} \in R^{n}$ and $r = ∥ z^{'} - z ∥$ . Suppose the objective function $f$ satisfies Assumption 2.1 and 2.2, then the auxiliary matrix operator $H (\cdot)$ holds that

\frac{H (z)}{1 + M r} ⪯ H (z^{'}) ⪯ (1 + M r) H (z),

(29)

for all $z, z^{'} \in R^{n}$ , where $M = 2 κ^{2} L_{2} / L$ .

Different with the quadratic case, auxiliary matrix $H (z)$ is not fixed for general saddle point problem. Based on the smoothness of $H (z)$ , we apply Lemma 3.11.1 to generalize Corollary 3.3 as follows.

Lemma 3.12.

Let $z \in R^{n}$ and $G \in R^{n \times n}$ be a positive definite matrix such that

H (z) ⪯ G ⪯ η H (z),

(30)

for some $η \geq 1$ . We additionally define $z_{+} \in R^{n}$ and $r = ∥ z_{+} - z ∥$ , then we have

~ G d e f = (1 + M r) G ⪰ H (z)

(31)

and

H (z_{+}) ⪯ {B r o y d}_{τ} (~ G, H (z_{+}), u) ⪯ (1 + M r)^{2} η H (z_{+})

(32)

for all $u \in R^{n}$ , $τ \in [0, 1]$ and $M = 2 κ^{2} L_{2} / L$ .

The relationship (32) implies it is reasonable to establish the algorithms by update rule

G_{k + 1} = {B r o y d}_{τ_{k}} ({~ G}_{k}, H_{k + 1}, u_{k})

(33)

with ${~ G}_{k} = (1 + M r_{k}) G_{k}$ and $r_{k} = ∥ z_{k + 1} - z_{k} ∥$ . Similarly, we can also achieve $G_{k + 1}$ by such ${~ G}_{k}$ for specific BFGS and SR1 update. Combining iteration (10) with (33), we propose the quasi-Newton methods for general strongly-convex-strongly-concave saddle point problems. The details is shown in Algorithm 5, 6 and 7 for greedy Broyden family, BFGS and SR1 updates respectively.

1: Input:

G_{0} ⪰ H

τ_{k} \in [0, 1]

and

M \geq 0

2: for

k \geq 0

z_{k + 1} = z_{k} - G_{k}^{- 1} {^H}_{k} g_{k}

r_{k} = ∥ z_{k + 1} - z_{k} ∥

{~ G}_{k} = (1 + M r_{k}) G_{k}

u_{k} = u_{H_{k + 1}} ({~ G}_{k})

G_{k + 1} = {B r o y d}_{τ_{k}} ({~ G}_{k}, H_{k + 1}, u_{k})

8: end for

Algorithm 5 Greedy Broyden Method for General Case

1: Input:

G_{0} ⪰ H

and

M \geq 0

L_{0} = G_{0}^{- 1 / 2}

3: for

k = 0, 1 \dots

z_{k + 1} = z_{k} - G_{k}^{- 1} {^H}_{k} g_{k}

r_{k} = ∥ z_{k + 1} - z_{k} ∥

{~ G}_{k} = (1 + M r_{k}) G_{k}

{~ L}_{k} = L_{k} / \sqrt{1 + M r_{k}}

u_{k} = {~ L}_{k}^{⊤} {~ H}_{k + 1} ({~ L}_{k})

G_{k + 1} = B F G S ({~ G}_{k}, H_{k + 1}, u_{k})

10:

L_{k + 1} =

Fast-Cholesky

(H_{k + 1}, L_{k}, u_{k})

11: end for

Algorithm 6 Greedy BFGS Method for General Case

1: Input:

G_{0} \geq H

and

M \geq 0

2: for

k = 0, 1 \dots

z_{k + 1} = z_{k} - G_{k}^{- 1} {^H}_{k} g_{k}

r_{k} = ∥ z_{k + 1} - z_{k} ∥

{~ G}_{k} = (1 + M r_{k}) G_{k}

u_{k} = {~ u}_{H_{k + 1}} ({~ G}_{k})

G_{k + 1} = S R 1 ({~ G}_{k}, H_{k + 1}, u_{k})

8: end for

Algorithm 7 Greedy SR1 Method for General Case

3.3.2 Convergence Analysis

We turn to consider building the convergence guarantee for algorithms proposed in Section 3.3.1. We start from the following lemma which is useful to further analysis.

Lemma 3.13.

Suppose objective function $f$ satisfies Assumption 2.1 and 2.2 and $z^{*} = [x^{*}; y^{*}]$ is its saddle point, then we have

∥ ∥ g (z) -^H (z) (z - z^{*}) ∥ ∥ \leq \frac{L_{2}}{2} ∥ z - z^{*} ∥^{2},

(34)

for all $z, z^{'} \in R^{n}$ .

We introduce the following notations to simplify the presentation. We let ${z_{k}}$ be the sequence generated from Algorithm 5, 6 or 7 and denote

{^H}_{k} def = {^H}_{k} (z_{k}), H_{k} def = ({^H}_{k} (z_{k}))^{2}, g_{k} def = g (z_{k}) and r_{k} def = ∥ z_{k + 1} - z_{k} ∥ .

We still use Euclidean distance $λ_{k} d e f = ∥ z_{k} - z^{*} ∥$ for analysis and establish the relationship between $λ_{k}$ and $λ_{k + 1}$ , which is shown in Lemma 3.14.

Lemma 3.14.

Using Algorithm 5, 6 and 7, suppose we have

H_{k} ⪯ G_{k} ⪯ η_{k} H_{k},

(35)

for some $η_{k} \geq 1$ . Then we have

	$λ_{k + 1}$	$\leq$	$(1 - \frac{1}{η_{k}}) λ_{k} + β λ_{k}^{2},$		(36)
	$r_{k}$	$\leq$	$λ_{k} + λ_{k + 1},$		(37)

where $β = \frac{κ^{2} L_{2}}{2 L}$ .

Rodomanov and Nesterov [29, Lemma 4.3] derive a result similar to Lemma 3.14 for minimizing the strongly-convex function $^f (\cdot)$ , but depends on the different measure $λ_{^f} (\cdot) def = ⟨ \nabla^f (\cdot), (\nabla^{2}^f (\cdot))^{- 1} \nabla^f (\cdot) ⟩$ .²²2The original notations of Rodomanov and Nesterov [29] is minimizing the strongly-convex function $f (\cdot)$ and establishing the convergence result by $λ_{f} (\cdot) def = ⟨ \nabla f (\cdot), (\nabla^{2} f (\cdot))^{- 1} \nabla f (\cdot) ⟩$ . To avoid ambiguity, we use notations $^f (\cdot)$ and $λ_{^f}$ to describe their work in this paper. Note that our algorithms are based on the iteration rule

z_{k + 1} = z_{k} - G_{k}^{- 1} {^H}_{k} g_{k} .

Compared with quasi-Newton methods for convex optimization, there exists an additional term ${^H}_{k}$ between $G_{k}^{- 1}$ and $g_{k}$ , which leads to the convergence analysis based on $λ_{k} d e f = ∥ z_{k} - z^{*} ∥$ is so difficult. Fortunately, we find directly using Euclidean distance $λ_{k}$ makes sure everything goes smoothly.

For further analysis, we also denote

ρ_{k} d e f = 2 M λ_{k},

(38)

and

σ_{k} d e f = σ_{H_{k}} (G_{k}) = ⟨ H_{k}^{- 1}, G_{k} ⟩ - n .

(39)

Then we establish linear convergence for the first period of iterations, which can be viewed as the generalization of Theorem 3.8. Note that the following result holds for all Algorithm 5, 6 and 7.

Theorem 3.15.

Using Algorithm 5, 6 and 7 with $M = 2 κ^{2} L_{2} / L$ and $G_{0} = L^{2} I$ , we suppose the initial point $z_{0}$ is sufficiently close to the saddle point such that

M λ_{0} \leq \frac{ln b}{8 b κ^{2}}

(40)

where $1 < b < 5$ . Let $ρ_{k} = 2 M λ_{k}$ , then for all $k \geq 0$ , we have

λ_{k + 1} \leq λ_{k},

(41)

H_{k} ⪯ G_{k} ⪯ exp (2 k - 1 \sum i = 0 ρ_{i}) κ^{2} H_{k} ⪯ b κ^{2} H_{k}

(42)

and

λ_{k} \leq {(1 - \frac{1}{2 b κ^{2}})}^{k} λ_{0} .

(43)

Then we analyze how does $σ_{k}$ change after one iteration to show the local superlinear convergence for greedy Broyden family method (Algorithm 5) and greedy BFGS method (Algorithm 6). Recall that $σ_{k}$ is defined in (39) to measure how well does the matrix $G_{k}$ approximate the auxiliary matrix $H_{k}$ .

Lemma 3.16.

Solving general strongly-convex strongly-concave saddle point problem (1) under Assumption 2.1 and 2.2 by proposed greedy quasi-Newton algorithms, we have the following results for $σ_{k}$ :

For greedy Broyden family method (Algorithm 5) with $M = 2 κ^{2} L_{2} / L$ and $G_{0} = L^{2} I$ we have

$σ_{k + 1} \leq (1 - \frac{1}{n κ^{2}}) (1 + M r_{k})^{2} (σ_{k} + \frac{2 n M r_{k}}{1 + M r_{k}}) for all k \geq 0$ (44)
For greedy BFGS method (Algorithm 6) with $M = 2 κ^{2} L_{2} / L$ , $G_{0} = L^{2} I$ and $L_{0} = \frac{1}{L} I$ , we have

$σ_{k + 1} \leq (1 - \frac{1}{n}) (1 + M r_{k})^{2} (σ_{k} + \frac{2 n M r_{k}}{1 + M r_{k}}) for all k \geq 0.$ (45)

The analysis for greedy SR1 method is based on constructing $η_{k}$ such that

t r (G_{k} - H_{k}) = η_{k} t r (H

Quasi-Newton Methods for Saddle Point Problems

Authors

Non-asymptotic Superlinear Convergence of Standard Quasi-Newton Methods

Proximal Quasi-Newton Methods for Regularized Convex Optimization with Linear and Accelerated Sublinear Convergence Rates

Identification and Adaptation with Binary-Valued Observations under Non-Persistent Excitation Condition

Newton-MR: Newton's Method Without Smoothness or Convexity

Convergence Rate Improvement of Richardson and Newton-Schulz Iterations

Stochastic Orthant-Wise Limited-Memory Quasi-Newton Methods

Fixed Point Algorithm Based on Quasi-Newton Method for Convex Minimization Problem with Application to Image Deblurring

1 Introduction

Paper Organization

2 Notation and Preliminaries

Assumption 2.1.

Assumption 2.2.

3 Quasi-Newton Methods for Saddle Point Problems

Lemma 3.1.

3.1 The Broyden Family Updates

Definition 3.2.

Lemma 3.3 ([29, Lemma 2.1 and Lemma 2.2]).

Lemma 3.4 ([29, Theorem 2.5]).

Lemma 3.5 ([20, Theorem 2.6]).

Lemma 3.6 ([20, Theorem 2.3]).

3.2 Algorithms for Quadratic Saddle Point Problems

Lemma 3.7.

Theorem 3.8.

Theorem 3.9.

Corollary 3.9.1.

3.3 Algorithms for General Saddle Point Problems

3.3.1 Algorithms

Lemma 3.10.

Lemma 3.11.

Corollary 3.11.1.

Lemma 3.12.

3.3.2 Convergence Analysis

Lemma 3.13.

Lemma 3.14.

Theorem 3.15.

Lemma 3.16.

Quasi-Newton Methods for Saddle Point Problems

Authors

Related Research

Non-asymptotic Superlinear Convergence of Standard Quasi-Newton Methods

Proximal Quasi-Newton Methods for Regularized Convex Optimization with Linear and Accelerated Sublinear Convergence Rates

Identification and Adaptation with Binary-Valued Observations under Non-Persistent Excitation Condition

Newton-MR: Newton's Method Without Smoothness or Convexity

Convergence Rate Improvement of Richardson and Newton-Schulz Iterations

Stochastic Orthant-Wise Limited-Memory Quasi-Newton Methods

Fixed Point Algorithm Based on Quasi-Newton Method for Convex Minimization Problem with Application to Image Deblurring

This week in AI

1 Introduction

Paper Organization

2 Notation and Preliminaries

Assumption 2.1.

Assumption 2.2.

3 Quasi-Newton Methods for Saddle Point Problems

Lemma 3.1.

3.1 The Broyden Family Updates

Definition 3.2.

Lemma 3.3 ([29, Lemma 2.1 and Lemma 2.2]).

Lemma 3.4 ([29, Theorem 2.5]).

Lemma 3.5 ([20, Theorem 2.6]).

Lemma 3.6 ([20, Theorem 2.3]).

3.2 Algorithms for Quadratic Saddle Point Problems

Lemma 3.7.

Theorem 3.8.

Theorem 3.9.

Corollary 3.9.1.

3.3 Algorithms for General Saddle Point Problems

3.3.1 Algorithms

Lemma 3.10.

Lemma 3.11.

Corollary 3.11.1.

Lemma 3.12.

3.3.2 Convergence Analysis

Lemma 3.13.

Lemma 3.14.

Theorem 3.15.

Lemma 3.16.