We consider first order gradient methods for effectively optimizing a composite objective in the form of a sum of smooth and, potentially, non-smooth functions. We present accelerated and adaptive gradient methods, called FLAG and FLARE, which can offer the best of both worlds. They can achieve the optimal convergence rate by attaining the optimal first-order oracle complexity for smooth convex optimization. Additionally, they can adaptively and non-uniformly re-scale the gradient direction to adapt to the limited curvature available and conform to the geometry of the domain. We show theoretically and empirically that, through the compounding effects of acceleration and adaptivity, FLAG and FLARE can be highly effective for many data fitting and machine learning applications.

## Authors

• 11 publications
• 12 publications
• 1 publication
• 42 publications
• 80 publications
• ### Tight Complexity Bounds for Optimizing Composite Objectives

We provide tight upper and lower bounds on the complexity of minimizing ...
05/25/2016 ∙ by Blake Woodworth, et al. ∙ 0

• ### NEON+: Accelerated Gradient Methods for Extracting Negative Curvature for Non-Convex Optimization

Accelerated gradient (AG) methods are breakthroughs in convex optimizati...
12/04/2017 ∙ by Yi Xu, et al. ∙ 0

• ### Higher-Order Accelerated Methods for Faster Non-Smooth Optimization

We provide improved convergence rates for various non-smooth optimizatio...
06/04/2019 ∙ by Brian Bullins, et al. ∙ 0

• ### Theoretical Limits of Pipeline Parallel Optimization and Application to Distributed Deep Learning

We investigate the theoretical limits of pipeline parallel learning of d...
10/11/2019 ∙ by Igor Colin, et al. ∙ 0

• ### Online Adaptive Methods, Universality and Acceleration

We present a novel method for convex unconstrained optimization that, wi...
09/08/2018 ∙ by Kfir Y. Levy, et al. ∙ 0

• ### On Curvature-aided Incremental Aggregated Gradient Methods

This paper studies an acceleration technique for incremental aggregated ...
05/31/2018 ∙ by Hoi-To Wai, et al. ∙ 0

• ### Hybrid Deterministic-Stochastic Methods for Data Fitting

Many structured data-fitting applications require the solution of an opt...
04/13/2011 ∙ by Michael P. Friedlander, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

We consider the problem of minimizing a convex function over a convex set , and we focus on first order methods, which exploit only value and gradient information about . These methods have become very important for many of the large-scale optimization problems that arise in machine learning applications. Two techniques have emerged as powerful tools for large-scale optimization: First, Nesterov’s accelerated algorithms [11] and its proximal variants, Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) [2, 3], can exploit smoothness to improve the convergence rate of a simple gradient/ISTA method from accuracy to after iterations. Second, adaptive regularization approaches such as AdaGrad [7]

can optimize a gradient method’s step-size in different directions, in some sense making the optimization problem better-conditioned. These two techniques have also become popular in non-convex optimization problems, such as parameter estimation in deep neural networks (see, for example,the work of

In this paper, we introduce a new algorithm, called FLAG, that combines the benefits of these two techniques. Like Nesterov’s accelerated algorithms, our method has a convergence rate of . Like AdaGrad, our method adaptively chooses a regularizer, in a way that performs almost as well as the best choice of regularizer in hindsight. In addition, our improvement over FISTA is roughly a square of AdaGrad’s improvement over mirror descent (see section 2.1 for details).

There have been a number of papers in recent years dealing with ways to interpret acceleration. Our algorithm is heavily influenced by [1]. This insightful paper introduced a view of accelerated gradient descent as a linear combination of gradient and mirror descent steps. We exploit this viewpoint in introducing adaptive regularization to our accelerated method. Authors in [1] use a fixed schedule to combine gradient and mirror descent steps. However, [4] views acceleration as an ellipsoid-like algorithm, and instead, determines an appropriate ratio using a line search. This latter idea is also crucial for our algorithm, as we do not know the right ratio in advance.

The optimal stepsize for an accelerated method generally depends on both the smoothness parameter of the objective function as well as the properties of the regularizer. In FLAG, the regularizer is chosen adaptively, and we show that it is also possible to adapt the stepsize in a way that is competitive with the best choice in hindsight. In fact, our method for picking the adaptive stepsizes is closely related to the approach in [8]. There, the authors considered the problem of picking the right stepsize to adapt to an unknown strong-convexity parameter. We use a related approach, but with different proof techniques, to choose the stepsize to adapt to a changing smoothness parameter.

Finally, it should also be noted that there are some interesting papers that study the continuous-time limit of acceleration algorithms, e.g., [12, 10, 15]. Indeed, studying adaptive regularization in the continuous time setting is an interesting future direction for research.

### 1.1 Notation and Definitions

In what follows, vectors are considered as column vectors and are denoted by bold lower case letters, e.g.,

and matrices are denoted by regular capital letters, e.g., . We overload the “diag” operator as follows: for a given matrix and a vector , and denote the vector made from the diagonal elements of and a diagonal matrix made from the elements of , respectively. Vector norms and denote the standard , and respectively. We adopt the Matlab notation for accessing the elements of vectors and matrices, i.e., components of a vector is indicated by and denotes the entire row of the matrix . Finally, signifies that is the augmentation of the matrix with the column vector .

Consider the optimization problem

 minx∈CF(x)=f(x)+h(x), (1)

where and are closed proper convex convex functions and is a closed convex set. We further assume that is differentiable with -Lipschitz gradient, i.e.,

 ∥∇f(x)−∇f(y)∥2≤L∥x−y∥2,∀x,y∈C, (2)

and we allow to be (possibly) non-smooth with sub-differential at denoted by .

The proximal operator [11] associated with , and is defined as

 {prox}(x):= argminy∈Ch(y)+⟨∇f(x),y−x⟩+L2∥y−x∥22 (3a) = argminy∈Ch(y)+L2∥y−(x−1L∇f(x))∥22. (3b)

For a symmetric positive definite(SPD) matrix , define . Note that is -strongly convex with respect to the norm , i.e., ,

 ψ(x)≥ψ(y)+⟨∇ψ(y),x−y⟩+12∥x−y∥2S.

The Bregman divergence associated with , is define as

 Bψ(x,y):=ψ(x)−ψ(y)−⟨∇ψ(y),x−y⟩=12∥x−y∥2S.

We will use the fact that, for any ,

 ⟨S(x−y),x−z⟩=12∥x−y∥2S+12∥x−z∥2S−12∥y−z∥2S. (4)

It is easy to see that the dual of is given by

 ψ∗(x):=supv∈Rd⟨x,v⟩−ψ(v)=12xTS−1x. (5)

## 2 The Algorithm and Main Result

In this section, we describe our main algorithm, FLAG, and give its convergence properties in Theorem 1. The core of FLAG consists of five essential ingredients:

1. A proximal gradient step (Step 4)

2. Construction of the adaptive regularization (Steps 5-10)

3. Picking the Adaptive stepsize (Step 12)

4. A mirror descent step (Step 13)

5. Linear combination of the proximal gradient step and the mirror descent step (Step 14)

The subroutine BinarySearch is given in Algorithm 2, where is the usual bisection routine for finding the root of a single variable function in the interval and to the accuracy of . More specifically, for a root such that and given , the sub-routine returns an approximation to such that and this is done with only function evaluations.

Theorem 1, gives the convergence properties of Algorithm 1:

###### Theorem 1 (Main Result: Convergence Property of FLAG)

Let . For any , after iterations of Algorithm 1, we get

 F(yT+1)−F(u)≤O(q2TTLDT2),

where . Also each iteration takes time.

Let be the subgradient of at iteration , , , and . Using a similar argument to that used in Lemma 5, we see that . AdaGrad achieves a rate of

 F(¯y)−F(x∗)≤O⎛⎜ ⎜⎝√JA√Dd∑Tk=1∥gTk∥22T⎞⎟ ⎟⎠, (6)

which is in contrast to mirror descent, which achieves

 F(¯y)−F(x∗)≤O⎛⎜ ⎜⎝√D2∑Tk=1∥gTk∥22T⎞⎟ ⎟⎠,

where is the average of all ’s. Thus, one can see that the improvement of AdaGrad compared to mirror descent is .

Now, let be defined as in Algorithm 1. One can verify using Lemma 5 that and ( is analogous to ). Thus, from Theorem 1, we get

 F(yT+1)−F(x∗)≤O(JBLDdT2). (7)

This is in contrast to FISTA, which achieves

 F(yT+1)−F(x∗)≤O(LD2dT2).

As a result, the improvement of FLAG compared to FISTA is . FLAG’s improvement over FISTA can thus be up to a square of AdaGrad’s improvement over mirror descent. Here, we note that and though analogous, are not the same.

Finally, we can directly compare the rates of (6) and (7) to see the speed-up offered by FLAG over AdaGrad. In particular, FLAG enjoys from the optimal rate of , compared to the sub-optimal rate of of AdaGrad. However, we do stress that AdaGrad does not make any smoothness assumptions, and thus works in more general settings than FLAG.

## 3 Proofs

Below we give the details for the proof of our main result. The proofs of technical lemmas in Section 3.1 are given in Appendix A.

The proof sketch of the convergence of FLAG is as follows.

1. FLAG is essentially a combination of mirror descent and proximal gradient descent steps (see Lemmas 1 and 4).

2. in Algorithm 1 plays the role of an ”effective gradient lipschitz constant” in each iteration. The convergence rate of our algorithm ultimately depends on . (see Lemmas 7 and 2)

3. By picking adaptively, as in AdaGrad, we achieve a non-trivial upper bound for . (see Lemma 5)

4. Our algorithm relies on picking an at each iteration that satisfies an inequality involving (see Corollary 1). However, because

is not known at the moment of picking

, we must choose an to roughly satisfy the inequality for all possible values of . We do this by picking using binary search. (see Lemmas 2 and 3 and Corollary 1)

5. Finally, we need to pick the right stepsizes for each iteration. Our scheme is very similar to the one used in [1], but generalized to handle a different each iteration. (see Lemma 6 and 7 and Corollary 2).

Theorem 2 combines items III, and IV. Theorem 1 combines Theorem 2 with items III and V to get our final result.

### 3.1 Technical Lemmas

We have the following key result (see Lemma 2.3 [2]) regarding the vector , as in Step 5 of FLAG, which is called the Gradient Mapping of on .

For any , we have

 F({prox}(x))≤F(y)+⟨L({prox}(x)−x),y−x⟩−L2∥x−{prox}(x)∥22,

where is defined as in (3). In particular, .

The following lemma establishes the Lipschitz continuity of the prox operator.

###### Lemma 2 (Prox Operator Continuity)

is a -Lipschitz continuous, that is, for any , we have

 ∥{prox}(x)−{prox}(y)∥2≤2∥x−y∥2.

Using prox operator continuity Lemma 2, we can conclude that given any , if and , then there must be a for which gives . Algorithm 2 finds an approximation to in iterations.

###### Lemma 3 (Binary Search Lemma)

Let defined as in Algorithm 2. Then one of 3 cases happen:

1. and ,

2. and , or

3. for some and .

Using the above result, we can prove the following:

###### Corollary 1

Let , , and be defined as in Algorithm 1 and . Then for all ,

 ⟨pk,xk−zk⟩≤(ηkLk−1)⟨pk,yk−xk⟩+DLηkLkT3.

Next, we state a result regarding the mirror descent step. Similar results can be found in most texts on online optimization, e.g. [1].

###### Lemma 4 (Mirror Descent Inequality)

Let and be the diameter of measured by infinity norm. Then for any , we have

 T∑k=1⟨ηkpk,zk−u⟩≤T∑k=1η2k2∥pk∥2S∗k+D2∥sT∥1

Finally, we state a similar result to that of [7] that captures the benefits of using in FLAG.

Define , where is as in Step 7 of Algorithm 1. We have

1. , where , and

2. .

### 3.2 Master Theorem

We can now prove the central theorems of this section.

###### Theorem 2

Let . For any , after iterations of Algorithm 1, we get

 T∑k=1{(η2k−1Lk−1−η2kLk+ηk)F(yk)−ηkF(u)}+η2TLTF(yT+1)≤T∑k=1DLη2kLkT3+D2∥sT∥1.

###### Proof (Proof of Theorem 2)

Noting that is the gradient mapping of on , it follows that

 T∑k=1ηk(F(yk+1)−F(u))= T∑k=1ηk(F({prox}(xk))−F(u)) (By the Gradient Mapping Lemma~{}???) ≤ T∑k=1ηk⟨pk,xk−u⟩−ηk2L∥pk∥22 = T∑k=1ηk⟨pk,(zk−u)⟩+T∑k=1ηk⟨pk,xk−zk⟩−ηk2L∥pk∥22 (by Lemma 4) ≤T∑k=1η2k2∥pk∥2S∗k+D2∥sT∥1 +T∑k=1ηk⟨pk,xk−zk⟩−ηk2L∥pk∥22 (By Step 11 of Algorithm 1) =T∑k=1ηk(ηkLk−1)2L∥pk∥22+D2∥sT∥1+T∑k=1ηk⟨pk,xk−zk⟩ (By Corollary 1) ≤T∑k=1ηk(ηkLk−1)2L∥pk∥22+D2∥sT∥1 +T∑k=1(ηk(ηkLk−1)⟨pk,yk−xk⟩+DLη2kLkT3) (By the Gradient Mapping Lemma 1) ≤T∑k=1DLη2kLkT3+D2∥sT∥1 +T∑k=1ηk(ηkLk−1)(F(yk)−F(yk+1)).

Now rearranging terms and re-indexing the summations gives the desired result.

### 3.3 Choosing the Stepsize

In this section, we discuss the final piece of our algorithm: choosing the stepsize for the mirror descent step.

###### Lemma 6

For the choice of in Algorithm 1 and , we have

1. ,

2. , and

3. .

###### Proof

We prove (i) by induction. For , is is easy to verify that , and so and the base case follows trivially. Now suppose . Re-arranging (i) for gives

 0=η2kLk−ηk−k−1∑i=1ηi=η2kLk−ηk−η2k−1Lk−1.

Now, it is easy to verify that the choice of in Algorithm 1 is a solution of the above quadratic equation. The rest of the items follow immediately from part i.

###### Corollary 2

Let . For any , after iterations of Algorithm 1, we get

 F(yT+1)−F(u)≤LDT2+D∥sT∥12∑Tk=1ηk.

###### Proof (Proof of corollary 2)

Combining Theorem 2 and Lemma 6, and noting that gives the desired result.

Finally, it only remains to lower bound , which is done in the following Lemma.

###### Lemma 7

For the choice of in Algorithm 1, we have

 T∑k=1ηk≥T31000∑Tk=1Lk

###### Remark 1

We note here that we made little effort to minimize constants, and that we used rather sloppy bounds such as . As a result, the constant appearing above is very conservative and a mere by product of our proof technique. We have numerically verified that a much smaller constant (e.g., ) indeed satisfies the bound above.

And now, the proof of our main result, Theorem 1, follows rather immediately:

###### Proof (Proof of Theorem 1)

The result follows immediately from Lemma 7 and Corollary 2 and noting that by Lemma 5 and by Step 8 of Algorithm 1 and definition of in Lemma 5. This gives

 F(yT+1)−F(u)≤LDT2+q2TT1000LDT2≤q2TT1001LDT2

The run-time per iteration follows from having to do calls to bisection, each taking time.

## Appendix A Proofs of Technical Lemmas

###### Proof (Proof of Lemma 1)

This result is the same as Lemma 2.3 in [2]. We bring its proof here for completeness.

For any , any sub-gradient, , of at , i.e., , and by optimality of in (3), we have

 0 ≤⟨∇f(x)+v+L({prox}(x)−x),y−{prox}(x)⟩ =⟨∇f(x)+v+L({prox}(x)−x),y−x⟩+⟨∇f(x)+v+L({prox}(x)−x),x−{prox}(x)⟩,

and so

 ⟨∇f(x),{prox}(x)−x⟩ ≤⟨∇f(x)+v+L({prox}(x)−x),y−x⟩ +⟨v,x−{prox}(x)⟩−L∥x−{prox}(x)∥22,

Now from -Lipschitz continuity of as well as convexity of and , we get

 F({prox}(x)) =f({prox}(x))+h({prox}(x)) ≤f(x)+⟨∇f(x),{prox}(x)−x⟩+L2∥{prox}(x)−x∥22+h({% prox}(x)) ≤f(x)+⟨∇f(x)+v+L({prox}(x)−x),y−x⟩+⟨v,x−{prox}(x)⟩ −L2∥x−{prox}(x)∥22+h({prox}(x)) ≤f(y)+⟨v+L({prox}(x)−x),y−x⟩+⟨v,x−{prox}(x)⟩ −L2∥x−{prox}(x)∥22+h({prox}(x)) =f(y)+⟨L({prox}(x)−x),y−x⟩+⟨v,y−{prox}(x)⟩ −L2∥x−{prox}(x)∥22+h({prox}(x)) ≤F(y)+⟨L({prox}(x)−x),y−x⟩−L2∥x−{prox}(x)∥22.

###### Proof (Proof of Lemma 2)

By Definition (3), for any , , and , we have

 ⟨v,z−{prox}(x)⟩≥−⟨∇f(x)+L({prox}(x)−x),z−{prox}(x)⟩, ⟨w,z′−{prox}(y)⟩≥−⟨∇f(y)+L({prox}(y)−y),z′−{prox}(y)⟩.

In particular, for and , we get

 ⟨v,{prox}(y)−{prox}(x)⟩ ≥−⟨∇f(x)+L({prox}(x)−x),{prox}(y)−{prox}(x)⟩, ⟨w,{prox}(y)−{prox}(x)⟩ ≤⟨∇f(y)+L({prox}(y)−y),{prox}(x)−{prox}(y)⟩.

By monotonicity of sub-gradient, we get

 ⟨v,{prox}(y)−{prox}(x)⟩≤⟨w,{prox}(y)−{prox}(x)⟩.

So

 ⟨∇f(x)+L({prox}(x)−x),% {prox}(x)−{prox}(y)⟩≤⟨∇f(y)+L({prox}(y)−y),{prox}(x)−{prox}(y)⟩,

and as a result

 ⟨∇f(x)+L({prox}(x)−x),% {prox}(x)−{prox}(y)⟩ =⟨∇f(x)+L({prox}(x)−{% prox}(y)+{prox}(y)−x),{prox}(x)−{prox}(y)⟩ =L∥{prox}(x)−{prox}(y)∥22+⟨∇f(x)+L({prox}(y)−x),{prox}(x)−{prox}(y)⟩ ≤⟨∇f(y)+L({prox}(y)−y),{prox}(x)−{prox}(y)⟩,

which gives

 L∥{prox}(x)−{prox}(y)∥22 ≤⟨∇f(y)−∇f(x)+L(x−y),{prox}(x)−{prox}(y)⟩ ≤(∥∇f(y)−∇f(x)∥2+L∥x−y∥2)∥{prox}(x)−{prox}(y)∥2 ≤2L∥x−y∥2∥{prox}(x)−{% prox}(y)∥2,

and the result follows.

###### Proof (Proof of Lemma 3)

Items i and ii, are simply Steps 3 and 4, respectively. For item iii, we have

 ∥x−w∥2=∥ty+(1−t)z−t∗y−(1−t∗)z∥2=∥(t−t∗)y−(t−t∗)z∥2=|t−t∗|∥y−z∥2≤ϵ∥y−z∥2.

Now it follows that

 |⟨{prox}(x)−x,y−z⟩| =|⟨{prox}(x)−x,y−z⟩−⟨{prox}(w)−w,y−z⟩| ≤∥⟨{prox}(x)−{prox}(w),y−z⟩∥2+|⟨x−w,y−z⟩| ≤∥{prox}(x)−{prox}(w)∥2∥y−z∥2+∥x−w∥2∥y−z∥2 By Lemma 2 ≤2∥x−w∥2∥y−z∥2+∥x−w∥2∥y−z∥2 =3∥x−w∥2∥y−z∥2 ≤3ϵ∥y−z∥22.

###### Proof (Proof of Corollary 1)

Note that by Step 5 of Algorithm 1), . For , since , the inequality is trivially true. For , we consider the three cases of Corollary 3: (i) if , the right hand side is and the left hand side is , (ii) if , the left hand side is and , so the inequality holds trivially, and (iii) in this last case, for some , we have

 ⟨pk,xk−zk⟩ =⟨−L({prox}(xk)−xk),tyk+(1−t)zk−zk⟩ =−Lt⟨({prox}(xk)−xk),yk−zk⟩,

and

 ⟨pk,yk−xk⟩ =⟨−L(