# Optimal Stochastic Strongly Convex Optimization with a Logarithmic Number of Projections

We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent (SGD) methods. To reduce the computation costs pertaining to the projections, we propose an Epoch-Projection Stochastic Gradient Descent (Epro-SGD) method. The proposed Epro-SGD method consists of a sequence of epochs; it applies SGD to an augmented objective function at each iteration within the epoch, and then performs a projection at the end of each epoch. Given a strongly convex optimization and for a total number of T iterations, Epro-SGD requires only (T) projections, and meanwhile attains an optimal convergence rate of O(1/T), both in expectation and with a high probability. To exploit the structure of the optimization problem, we propose a proximal variant of Epro-SGD, namely Epro-ORDA, based on the optimal regularized dual averaging method. We apply the proposed methods on real-world applications; the empirical results demonstrate the effectiveness of our methods.

## Authors

• 10 publications
• 62 publications
• 26 publications
• 60 publications
• 34 publications
• ### Unified Optimal Analysis of the (Stochastic) Gradient Method

In this note we give a simple proof for the convergence of stochastic gr...
07/09/2019 ∙ by Sebastian U. Stich, et al. ∙ 0

• ### Convergence Analysis of Homotopy-SGD for non-convex optimization

First-order stochastic methods for solving large-scale non-convex optimi...
11/20/2020 ∙ by Matilde Gargiani, et al. ∙ 0

• ### Gradient Descent Averaging and Primal-dual Averaging for Strongly Convex Optimization

Averaging scheme has attracted extensive attention in deep learning as w...
12/29/2020 ∙ by Wei Tao, et al. ∙ 0

• ### Non-Asymptotic Bounds for Zeroth-Order Stochastic Optimization

We consider the problem of optimizing an objective function with and wit...
02/26/2020 ∙ by Nirav Bhavsar, et al. ∙ 0

• ### Better Communication Complexity for Local SGD

We revisit the local Stochastic Gradient Descent (local SGD) method and ...
09/10/2019 ∙ by Ahmed Khaled, et al. ∙ 0

• ### Convex Optimization with an Interpolation-based Projection and its Application to Deep Learning

Convex optimizers have known many applications as differentiable layers ...
11/13/2020 ∙ by Riad Akrour, et al. ∙ 0

• ### Practical Precoding via Asynchronous Stochastic Successive Convex Approximation

We consider stochastic optimization of a smooth non-convex loss function...
10/03/2020 ∙ by Basil M. Idrees, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Recent years have witnessed an increased interest in adopting the stochastic (sub)gradient (SGD) methods [1, 3, 21]

for solving large-scale machine learning problems. In each of the algorithmic iterations, SGD reduces the computation cost by sampling one (or a small number of) example for computing a stochastic (sub)gradient. Thus the computation cost in SGD is independent of the size of the data available for training; this property makes SGD appealing for large-scale optimization. However, when the optimization problems involve a complex domain (for example a positive definite constraint or a polyhedron one), the projection operation in each iteration of SGD, which is used to ensure the feasibility of the intermediate solutions, may become the computational bottleneck.

In this paper we consider to solve the following constrained optimization problem

 minx∈Rdf(x) (1) s.t.c(x)≤0,

where is -strongly convex [23] and is convex. We assume a stochastic access model for , in which the only access to is via a stochastic gradient oracle; in other words, given arbitrary

, this stochastic gradient oracle produces a random vector

, whose expectation is a subgradient of at the point , i.e., , where denotes the subdifferential set of at . On the other hand we have the full access to the (sub)gradient of .

The standard SGD method [5] solves Eq. (1) by iterating the updates in Eq. (2) with an appropriate step size , e.g., ), as below

 xt+1=P{x∈Rd:c(x)≤0}[xt−ηtg(xt)], (2)

and then returning as the final solution for a total number of iterations . Note that is a projection operator defined as

 (3)

If the involved constraint function is complex (e.g., a polyhedral or a positive definite constraint), computing the associated projection may be computationally expensive; for example, a projection onto a positive definite cone over

requires a full singular value decomposition (SVD) operation with time complexity of

.

In this paper, we propose an epoch-based SGD method, called Epro-SGD, which requires only a logarithmic number of projections (onto the feasible set), and meanwhile achieves an optimal convergence rate for stochastic strongly convex optimization. Specifically, the proposed Epro-SGD method consists of a sequence of epochs; within each of the epochs, the standard SGD is applied to optimize a composite objective function augmented by the complex constraint function, hence avoiding the expensive projections steps; at the end of every epoch, a projection operation is performed to ensure the feasibility of the intermediate solution. Our analysis shows that given a strongly convex optimization and for a total number of iterations, Epro-SGD requires only projections, and meanwhile achieves an optimal rate of convergence at , both in expectation and with a high probability.

To exploit the structure (for example the sparisty) of the optimization problem, we propose a proximal variant of the Epro-SGD method, namely Epro-ORDA, which utilizes an existing optimal dual averaging method to solve the involved proximal mapping. Our analysis shows that Epro-ORDA similarly requires only a logarithmic number of projections while enjoys an optimal rate of convergence.

For illustration we apply the proposed Epro-SGD methods on two real-world applications, i.e., the constrained Lasso formulation and the large margin nearest neighbor (LMNN) classification. Our experimental results demonstrate the efficiency of the proposed methods, in comparison to the existing methods.

## 2 Related Work

The present work is inspired from the break-through work in [20], which proposed two novel one-projection-based stochastic gradient descent (OneProj) methods for stochastic convex optimizations. Specifically the first OneProj method was developed for general convex optimization; it introduces a regularized Lagrangian function as

 L(x,λ)=f(x)+λc(x)−γ2λ2,λ≥0,

then applies SGD to the convex-concave problem , and finally performs only one projection at the end of all iterations, where is a bounded ball subsuming as a subset.

The second OneProj method was developed for strongly convex optimization. The proposed method introduced an augmented objective function

 (4)

where is a parameter dependent on the total number of iterations , and is a problem specific parameter [20]. OneProj applies SGD to the augmented objective function, specifically using a stochastic subgradient of and a subgradient of , and then performs a projection step after all iterations. For a total number iterations, the OneProj method achieves a rate of convergence at , which is suboptimal for stochastic strongly convex optimization.

Several recent works [15, 26] propose optimal methods with optimal rates of convergence at for stochastic strongly convex optimization. In particular, the Epoch-SGD method [15] consists of a sequence of epochs, each of which has a geometrically decreasing step size and a geometrically increasing iteration number. This method however needs to project the intermediate solutions onto a feasible set at every algorithmic iteration; when the involved constraint is complex, the involved projection is usually computationally expensive. This limitation restricts the practical applications on large scale data analysis. Therefore we are motivated to develop an optimal stochastic algorithm for strongly convex optimization but with a constant number of projections.

Another closely related work is the logT-SGD [33] for stochastic strongly convex and smooth optimization. LogT-SGD achieves an optimal rate of convergence, while requires to perform projections, where is the ratio of the smoothness parameter to the strong convexity parameter. There are several key differences between our proposed Epro-SGD method and logT-SGD: (i) logT-SGD and its analysis rely on both the smoothness and the strong convexity of the objective function; in contrast, Epro-SGD only assumes that the objective function is strongly convex; (ii) the number of the required projections in logT-SGD is , where the conditional number can be very large in real applications; in contrast, Epro-SGD requires at most projections.

Besides reducing the number of projections in SGD, another line of research is based on the conditional gradient algorithms [7, 14, 17, 18, 32]; this type of algorithms mostly build upon the Frank-Wolfe technique [11], which eschews the projection in favor of a linear optimization step; however in general, they require the smoothness assumption in the objective function. On the other hand, [12, 16] extended Frank-Wolfe techniques to stochastic or online setting for general and strongly convex optimizations. Specifically [16] presents an online/stochastic Frank-Wolfe (OFW) algorithm with a convergence rate for general convex optimization problems, which is slower than the optimal rate [12] presents an algorithm for online strongly convex optimization with an regret bound, implying an convergence rate for stochastic stronlgy convex optimization. This algorithm requires the problem domain to be a polytope, instead of a convex inequality constraint used in this paper; it also hinges on an efficient local linear optimization oracle that amounts to approximately solving a linear optimization problem over an intersection of a ball and and the feasible domain; furthermore the convergence result only holds in expectation and is sub-optimal.

## 3 Epoch-Projection Sgd Algorithm

In this section, we present an epoch-projection SGD method, called Epro-SGD, for solving Eq. (1) and discuss its convergence result. Based on a stochastic dual averaging algorithm, we then present a proximal variant of the proposed Epro-SGD method.

### 3.1 Setup and Background

Denote the optimal solution to Eq. (1) by and its domain set by . Since is -strongly convex [23] and is convex, the optimization problem in Eq. (1) is strongly convex. Note that the strong convexity in implies that for any . Our analysis is based on the following assumptions:

1. The stochastic subgradient is uniformly bounded by , i.e., .

2. The subgradient is uniformly bounded by , i.e., for any .

3. There exists a positive value such that

 [minc(x)=0,v∈∂c(x),v≠0∥v∥2]≥ρ. (5)

Remarks Assumptions A1 and A2 respectively impose an upper bound on the stochastic subgradient of the objective function and the constraint function . Assumption A3 ensures that the projection of a point onto a feasible domain does not deviate too much from this intermediate point. Note that Assumption A1 is previously used in [15]; a condition similar to Assumption A3 is used in [20], which however simply assumes that , without considering possible non-differentiability in .

A key consequence of Assumption A3 is presented in the following lemma.

###### Lemma 1.

For any , let . If Assumption A3 holds, then

 ∥ˆx−˜x∥2≤1ρ[c(ˆx)]+,ρ>0, (6)

where is a hinge operator defined as if , and otherwise.

###### Proof.

If , we have ; the inequality in Eq. (6) trivially holds. If , we can verify that , and there exists and such that  (using duality theory). It follows that  (), and thus is the same direction as . It follows that

 c(ˆxT) = c(ˆxT)−c(˜xT)≥(ˆxT−˜xT)⊤v = ∥v∥2∥ˆxT−˜xT∥2≥ρ∥ˆxT−˜xT∥2,

where the last inequality uses Assumption A3. This completes the proof of this lemma. ∎

The result in Lemma 1 is closely related to the polyhedral error bound condition [13, 31]; this condition shows that the distance of a point to the optimal set of a convex optimization problem whose epigraph is a polyhedron is bounded by the distance of the objective value at this point to the optimal objective value scaled by a constant. For illustration, we consider the optimization problem

 minx∈Rd[c(x)]+

with an optimal set as . If , is the closest point in the optimal set to . Therefore, by the polyhedral error bound condition of a polyhedral convex optimization, if is a polyhedral function, there exists a such that

 ∥ˆx−˜x∥2≤1ρ([c(ˆx)]+−minx[c(x)]+)=1ρ[c(ˆx)]+.

Below we present three examples in which Assumption A3 or Lemma 1 is satisfied. Example : an affine function with . Example : the norm constraint where . Example : the maximum of a finite number of affine functions satisfying Lemma 1 as well as the polyhedral error bound condition [31].

### 3.2 Main Algorithm

To solve Eq. (1) (using Epro-SGD), we introduce an augmented objective function by incorporating the constraint function as

 F(x)=f(x)+λ[c(x)]+. (7)

It is worth noting that the augmented function in Eq. (7) does not have any iteration-dependent parameter, for example the parameter in Eq. (4). is a prescribed parameter satisfying , as illustrated in Lemma 2.

The details of our proposed Epro-SGD algorithm is presented in Algorithm 1. Similar to Epoch-SGD [15], Epro-SGD consists of a sequence of epochs, each of which has a geometrically decreasing step size and a geometrically increasing iteration number (Line in Algorithm 1). The updates in every intra-epoch (Line - ) are standard SGD steps applied to the augmented objective function with . Epro-SGD is different from Epoch-SGD in that the former computes a projection only at the end of each epoch, while the latter computes a projection at each iteration. Consequently, when the projection step is computationally expensive (e.g., projecting onto a positive definite constraint), Epro-SGD may require much less computation time than Epoch-SGD.

In Lemma 2, we present an important convergence analysis for the intra-epoch steps of Algorithm 1, which are key building blocks for deriving the main results in Theorem 1.

###### Lemma 2.

Under Assumptions A1A3, if we apply the update for a number of iterations, the following equality holds

where .

###### Proof.

Let and denote by the expectation conditioned on the randomness until round . It is easy to verify that , and . For any , we have

 (xt−x)⊤∇F(xt) ≤ 12η(∥x−xt∥22−∥x−xt+1∥22)+ η2∥˜∇f(xt,ξt)+λ∇[c(xt)]+∥22+ (x−xt)⊤(g(xt)−∇f(xt)) ≤ 12η(∥x−xt∥22−∥x−xt+1∥22)+

where . Furthermore by the convexity of , we have

 F(xt)−F(x) ≤ 12η(∥x−xt∥22−∥x−xt+1∥22)+ η(G21+λ2G22)+ζt(x).

Noting that , taking expectation over randomness and summation over , we have

 1TE[T∑t=1(F(xt)−F(x))]=E[F(ˆxt)−F(x)] ≤E[∥x1−x∥22]2ηT+η(G21+λ2G22).

Let . Since , we have

 E[F(ˆxt)−F(x∗)]≤B. (8)

It follows that

 (9)

If , we have . Following from and , we can verify that and also holds.

Next we show that holds when . From Lemma 1, we have

 c(ˆxT)≥ρ∥ˆxT−˜xT∥2. (10)

Moreover it follows from and that the following inequality holds

 f(x∗)−f(ˆxT) ≤ f(x∗)−f(˜xT)+f(˜xT)−f(ˆxT) (11) ≤ G1∥ˆxT−˜xT∥2.

Substituting Eqs. (10) and (11) into Eq. (9), we have

 λρE[∥ˆxT−˜xT∥2] ≤ E[f(x∗)−f(ˆxT)]+B ≤ G1E[∥ˆxT−˜xT∥2]+B.

By some rearrangement, we have . Furthermore we have

 E[f(˜xT)]−f(x∗)≤E[f(˜xT)−f(ˆxT)]+E[f(ˆxT)] −f(x∗)≤E[G1∥ˆxT−˜xT∥2]+B≤λρλρ−G1B,

where the second inequality follows from , and for any . This completes the proof of the lemma. ∎

We present a main convergence result of the Epro-SGD algorithm in the following theorem.

###### Theorem 1.

Under Assumptions A1A3 and given that is -strongly convex, if we let , , and set , the total number of epochs in Algorithm 1 is given by

 k†=⌈log2(T8+1)⌉≤log2(T4), (12)

the solution enjoys a convergence rate of

 E[f(xk†+11)]−f(x∗)≤32μ2G2β(T+8), (13)

and .

###### Proof.

From the updating rule , we can easily verify Eq. (12). Since , the inequality trivially holds.

Let . It follows that and . Next we show the inequality

 E[f(xk)]−f(x∗)≤Vk (14)

holds by induction. Note that Eq. (14) implies , due to . Let . It follows from Lemma 5, , and , the inequality in Eq. (14) holds when . Assuming that Eq. (14) holds for , we show that Eq. (14) holds for .

For a random variable

measurable with respect to the randomness up to epoch . Let denote the expectation conditioned on all the randomness up to epoch . Following Lemma 2, we have

 Ek†[Δk†+1]≤μ⎡⎣ηk†G2+E[∥xk†1−x∗∥22]2ηk†Tk†⎤⎦.

Since by the strong convexity in , we have

 E[Δk†+1] ≤ μ[ηk†G2+E[Δk†]ηk†Tk†β] = μηk†G2+Vk†μηk†Tk†β=Vk†4+Vk†4=Vk†+1,

which completes the proof of this theorem. ∎

Remark We compare the obtained main results in Theorem 1 with several existing works. Firstly Eq. (13) implies that Epro-SGD achieves an optimal bound , matching the lower bound for a strongly convex problem [15]. Secondly in contrast to the OneProj method [20] with a convergence rate , Epro-SGD uses no more than projections to obtain an convergence rate. Epro-SGD thus has better control over the solution for not deviating (too much) from the feasibility domain in the intermediate iterations. Thirdly compared to Epoch-SGD with its convergence rate bounded by , the convergence rate bound of Epro-SGD is only worse by a factor of constant . Particularly consider a positive definite constraint with , , and , we have and the bound of Epro-SGD is only worse by a factor of than Epoch-SGD. Finally compared to the logT-SGD algorithm [33] which requires projections ( is the conditional number), the number of projections in Epro-SGD is independent of the conditional number.

The main results in Lemma 2 and Theorem 1 are expected convergence bounds. In Theorem 2 (proof provided in Appendix) we show that Epro-SGD also enjoys a high probability bound under a boundedness assumption, i.e., for all . Note that the existing Epoch-SGD method [15] uses two different methods to derive its high probability bounds. Specifically the first method relies on an efficient function evaluator to select the best solutions among multiple trials of run; while the second one modifies the updating rule by projecting the solution onto the intersection of the domain and a center-shifted bounded ball with decaying radius. These two methods however may lead to additional computation steps, if being adopted for deriving high probability bounds for Epro-SGD.

###### Theorem 2.

Under Assumptions A1A3 and given for all . If we let , , and set , , the total number of epochs in Algorithm 1 is given by

 k†=⌊log2(TT1+1)⌋≤log2(T/4),

and the final solution enjoys a convergence rate of

 f(xk†+11)−f(x∗)≤4T1μ2G2β(T+T1)

with a probability at least , where .

Remark The assumption

can be satisfied practically, if we estimate the value of

such that , and then project the intermediate solutions onto at every iteration. Note that Epoch-SGD [15] requires a total number of projections, and its high probability bound of Epoch-SGD is denoted by with a probability at lest , where .

### 3.3 A Proximal Variant

We propose a proximal extension of Epro-SGD, by exploiting the structure of the objective function. Let the objective function in Eq. (1) be a sum of two components

 ˆf(x)=f(x)+g(x),

where is a relatively simple function, for example a squared -norm or -norm, such that the involved proximal mapping

 minx∈Rdg(x)+12∥x−ˆx∥22

is easy to compute. The optimization problem in Eq. (1) can be rewritten as

 minx∈Rdf(x)+g(x) (15) s.t.c(x)≤0.

Denote by the optimal solution to Eq. (15). We similarly introduce an augmented objective function as

 (16)

The update of the proximal SGD method for solving (15[9, 10, 22] is given by

 xt+1=argminx∈D12∥x−(xt−ηg(xt))∥22+ηg(x). (17)

If is a sparse regularizer, the proximal SGD can guarantee the sparsity in the intermediate solutions and usually yields better convergence than the standard SGD. However, given a complex constraint, solving the proximal mapping may be computational expensive. Therefore, we consider a proximal variant of Epro-SGD which involves only the proximal mapping of without the constraint . An instinctive solution is to use the following update in place of step 6 in Algorithm 1:

 xkt+1 = argminx∈Rd12∥x− (18) [xkt−ηk(g(xkt)+λ∂[c(xkt)]+)]∥22+ηkg(x).

Based on this update and using techniques in Lemma 2, we obtain a similar convergence result (proof provided in Appendix), as presented in the following lemma [8].

###### Lemma 3.

Under Assumptions A1A3 and setting , by applying the update in Eq. (18) a number of iterations, we have

 E[ˆf(˜xkT)]−ˆf(x∗) ≤ μE[ηG2+∥xk1−x∗∥222ηT (19) +g(xk1)−g(xkT+1)T],

where , and denotes the projected solution of the averaged solution .

Different from the main result in Lemma 2, Eq. (19) has an additional term ; it makes the convergence analysis in Epro-SGD difficult. To overcome this difficulty, we adopt the optimal regularized dual averaging (ORDA) algorithm [6] for solving Eq. (16). The details of ORDA are presented in Algorithm 2. The main convergence results of ORDA are summarized in the following lemma (proof provided in Appendix).

###### Lemma 4.

Under Assumptions A1A3 and setting , by running ORDA a number of iterations for solving the augmented objective (16), we have

 E[F(ˆxT)−F(x∗)]≤4∥x1−x∗∥22η√T+2η(3G1+2λG2)2√T,

and

 E[ˆf(˜xT)]−ˆf(x∗) ≤ μE[4∥x1−x∗∥22η√T+ 2η(3G1+2λG2)2√T],

where denotes the projected solution of the final solution .

We present a proximal variant of Epro-SGD, namely Epro-ORDA, in Algorithm 3, and summarize its convergence results in Theorem 3. Note that Algorithm 2 and the convergence analysis in Lemma 4 are independent of the strong convexity in ; the strong convexity is however used for analyzing the convergence of Epro-ORDA in Theorem 3 (proof provided in Appendix).

###### Theorem 3.

Under Assumptions A1A3 and given that is -strongly convex, if we let and , and set , then the total number of epochs in Algorithm 3 is given by

 k†=⌊log2(T17+1)⌋≤log2(T/8),

and the final solution enjoys a convergence rate of

 E[ˆf(xk†+11)]−ˆf(x∗)≤68μ2G2β(T+17),

and .

## 4 An Example of Solving Lmnn via Epro-Sgd

In this section, we discuss an application of applying the proposed Epro-SGD to solve a high dimensional distance metric learning (DML) with a large margin formulation, i.e., the large margin nearest neighbor (LMNN) classification method [30]. LMNN classification is one of the state-of-the-art methods for k-nearest neighbor classification. It learns a positive semi-definite distance metric, based on which the examples from the k-nearest neighbors always belong to the same class, while the examples from different classes are separated by a large margin.

To describe the LMNN method, we first present some notations. Let , be a set of data points, where and denote the feature representation and the class label, respectively. Let be a positive definite matrix that defines a distance metric as . To learn a distance metric that separates the examples from different classes by a large margin, one needs to extract a set of similar examples (from the same class) and dissimilar examples (from a different class), denoted by , where shares the same class label to and a different class from . To this end, for each example one can form by extracting the k nearest neighbors (defined by an Euclidean distance metric) that share the same class label to , and form by extracting a set of examples that have a different class label. Then an appropriate distance metric could be obtained from the following constrained optimization problem

 minA s.t. A⪰ϵI, (20)

where is a hinge loss and is a trade-off parameter. In Eq. (4), is used as the constraint to ensure that Assumption A3 holds. Minimizing the first term is equivalent to maximizing the margin between and . The matrix encodes certain prior knowledge about the distance metric; for example, the original LMNN work [30] defines as , where are all k-nearest neighbor pairs from the same class. Other works [19] have used a weighted summation of distances between all data pairs or intra-class covariance matrix [25]. The last term is used as a regularization term and also makes the objective function strongly convex.

For data sets of very high dimensionality, i.e., , LMNN in Eq. (4) usually produces a sub-optimal solution [25], as this formulation does not capture the sparsity structure of the features. Therefore we add a sparse regularizer and express the formulation below

 minA cNN∑j=1ℓ(A,xj1,xj2,xj3)+(1−c)tr(AL) +μ12∥A∥2F+μ2∥A∥off1 s.t. A⪰ϵI, (21)

where