This project collects the different accepted papers and their link to Arxiv or Gitxiv
One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a fixed step size by hand, which can be time consuming in practice. In this paper, we propose to use the Barzilai-Borwein (BB) method to automatically compute step sizes for SGD and its variant: stochastic variance reduced gradient (SVRG) method, which leads to two algorithms: SGD-BB and SVRG-BB. We prove that SVRG-BB converges linearly for strongly convex objective functions. As a by-product, we prove the linear convergence result of SVRG with Option I proposed in , whose convergence result is missing in the literature. Numerical experiments on standard data sets show that the performance of SGD-BB and SVRG-BB is comparable to and sometimes even better than SGD and SVRG with best-tuned step sizes, and is superior to some advanced SGD variants.READ FULL TEXT VIEW PDF
This project collects the different accepted papers and their link to Arxiv or Gitxiv
Python implementation of SVRG-BB and SGD-BB algorithms
The following optimization problem, which minimizes the sum of cost functions over samples from a finite training set, appears frequently in machine learning:
where is the sample size, and each is the cost function corresponding to the -th sample data. Throughout this paper, we assume that each is convex and differentiable, and the function is strongly convex. Problem (1.1) is challenging when is extremely large so that computing and for given is prohibited. Stochastic gradient descent (SGD) method and its variants have been the main approaches for solving (1.1). In the -th iteration of SGD, a random training sample is chosen from and the iterate is updated by
where denotes the gradient of the -th component function at , and is the step size (a.k.a. learning rate). In (1.2), it is usually assumed that
is an unbiased estimation to, i.e.,
However, it is known that the total number of gradient evaluations of SGD depends on the variance of the stochastic gradients and it is of sublinear convergence rate for the strongly convex and smooth problem (1.1), which is inferior to the full gradient method. As a result, many works along this line have been focusing on designing variants of SGD that can reduce the variance and improve the complexity. Some popular methods along this line are briefly summarized as follows. The stochastic average gradient (SAG) method proposed by Le Roux et al.  updates the iterates by
where at each iteration a random training sample is chosen and is defined as
It is shown in  that SAG converges linearly for strongly convex problems. The SAGA method proposed by Defazio et al.  is an improved version of SAG, and it does not require the strong convexity assumption. It is noted that SAG and SAGA need to store the latest gradients for the component functions . The SDCA method proposed by Shalev-Shwartz and Zhang  also requires to store all the component gradients. The stochastic variance reduced gradient (SVRG) method proposed by Johnson and Zhang  is now widely used in the machine learning community for solving (1.1), because it achieves the variance reduction effect for SGD, and it does not need to store the component gradients.
As pointed out by Le Roux et al. , one important issue regarding to stochastic algorithms (SGD and its variants) that has not been fully addressed in the literature, is how to choose an appropriate step size while running the algorithm. In classical gradient descent method, the step size is usually obtained by employing line search techniques. However, line search is computationally prohibited in stochastic gradient methods because one only has sub-sampled information of function value and gradient. As a result, for SGD and its variants used in practice, people usually use a diminishing step size , or use a best-tuned fixed step size. Neither of these two approaches can be efficient.
Some recent works that discuss the choice of step size in SGD are summarized as follows. AdaGrad  scales the gradient by the square root of the accumulated magnitudes of the gradients in the past iterations, but it still requires a fixed step size .  suggests a line search technique on the component function selected in each iteration, to estimate step size for SAG.  suggests performing line search for an estimated function, which is evaluated by a Gaussian process with samples .  suggests to generate the step sizes by a given function with an unknown parameter, and to use the online SGD to update this unknown parameter.
Our contributions in this paper are in several folds.
We propose to use the Barzilai-Borwein (BB) method to compute the step size for SGD and SVRG. The two new methods are named as SGD-BB and SVRG-BB, respectively. The per-iteration computational cost of SGD-BB and SVRG-BB is almost the same as SGD and SVRG, respectively.
We prove the linear convergence of SVRG-BB for strongly convex functions. As a by-product, we show the linear convergence of SVRG with Option I (SVRG-I) proposed in . Note that in  only convergence of SVRG with Option II (SVRG-II) was given, and the proof for SVRG-I has been missing in the literature. However, SVRG-I is numerically a better choice than SVRG-II, as demonstrated in .
We conduct numerical experiments for SGD-BB and SVRG-BB on solving logistic regression and SVM problems. The numerical results show that SGD-BB and SVRG-BB are comparable to and sometimes even better than SGD and SVRG with best-tuned step sizes. We also compare SGD-BB with some advanced SGD variants, and demonstrate that our method is superior.
The rest of this paper is organized as follows. In Section 2 we briefly introduce the BB method in the deterministic setting. In Section 3 we propose our SVRG-BB method, and prove its linear convergence for strongly convex functions. As a by-product, we also prove the linear convergence of SVRG-I. In Section 4 we propose our SGD-BB method. A smoothing technique is also implemented to improve the performance of SGD-BB. We conduct numerical experiments for SVRG-BB and SGD-BB in Section 5. Finally, we draw some conclusions in Section 6.
The BB method, proposed by Barzilai and Borwein in , has been proved to be very successful in solving nonlinear optimization problems. The key idea behind the BB method is motivated by quasi-Newton methods. Suppose we want to solve the unconstrained minimization problem
where is differentiable. A typical iteration of quasi-Newton methods for solving (2.1) takes the following form:
where is an approximation of the Hessian matrix of at the current iterate . Different choices of give different quasi-Newton methods. The most important feature of is that it must satisfy the so-called secant equation:
where and for . It is noted that in (2.2) one needs to solve a linear system, which may be time consuming when is large and dense. One way to alleviate this burden is to use the BB method, which replaces by a scalar matrix . However, one cannot choose a scalar such that the secant equation (2.3) holds with . Instead, one can find such that the residual of the secant equation is minimized, i.e.,
which leads to the following choice of :
Therefore, a typical iteration of the BB method for solving (2.1) is
where is computed by (2.4).
For convergence analysis, generalizations and variants of the BB method, we refer the interested readers to [19, 20, 9, 5, 6, 4] and references therein. Recently, BB method has been successfully applied for solving problems arising from emerging applications, such as compressed sensing , sparse reconstruction  and image processing .
We see from (2.5) and (2.4) that the BB method does not need any parameter and the step size is computed while running the algorithm. This has been the main motivation for us to work out a black-box stochastic gradient descent method that can compute the step size automatically without requiring any parameters. In this section, we propose to incorporate the BB step size to SVRG which leads to the SVRG-BB method.
The following assumption is made throughout this section.
We assume that (1.3) holds for any . We assume that the objective function is -strongly convex, i.e.,
We also assume that the gradient of each component function is -Lipschitz continuous, i.e.,
Under this assumption, it is easy to see that is also -Lipschitz continuous:
There are two loops in SVRG (Algorithm 1). In the outer loop (each outer iteration is called an epoch), a full gradient is computed, which is used in the inner loop for generating stochastic gradients with lower variance. is then chosen, based on the outputs of inner loop, for the next outer loop. Note that two options for choosing are suggested in SVRG. Intuitively, Option I in SVRG (denoted as SVRG-I) is a better choice than Option II (denoted as SVRG-II), because the former used the latest information from the inner loop. This has been confirmed numerically in  where SVRG-I was applied to solve real applications. However, the convergence analysis is only available for SVRG-II (see, e.g., ,  and ), and the convergence for SVRG-I has been missing in the literature. We now cite the convergence analysis of SVRG-II given in  as follows.
There has been a series of follow-up works on SVRG and its variants. Xiao and Zhang  developed a proximal SVRG method for minimizing the finite sum function plus a nonsmooth regularizer.  applied Nesterov’s acceleration technique to SVRG to improve the convergence rate that depends on the condition number .  proved if the full gradient computation was replaced by a growing-batch estimation, the linear convergence rate can be preserved.  and  showed that SVRG with minor modifications can converge to a stationary point for nonconvex optimization problems.
It is noted that in SVRG, the step size needs to be provided by the user. According to (3.1), the choice of is dependent on , which may be difficult to estimate in practice. In this section, we propose the SVRG-BB method that computes the step size using the BB method. Our SVRG-BB algorithm is described in Algorithm 2. Note that the only difference between SVRG and SVRG-BB is that in the latter we use BB method to compute the step size , instead of using a prefixed as in SVRG.
A few remarks are in demand for the SVRG-BB algorithm.
One may notice that is equal to the step size computed by the BB formula (2.4) divided by . This is because in the inner loop for updating , unbiased gradient estimators are added to to get .
If we always set in SVRG-BB instead of using (3.2), then it reduces to SVRG-I.
For the first outer loop of SVRG-BB, a step size needs to be specified, because we are not able to compute the BB step size for the first outer loop. However, we observed from our numerical experiments that the performance of SVRG-BB is not sensitive to the choice of .
The BB step size can also be naturally incorporated to other SVRG variants, such as SVRG with batching .
In this section, we analyze the linear convergence of SVRG-BB (Algorithm 2) for solving (1.1) with strongly convex objective , and as a by-product, our analysis also proves the linear convergence of SVRG-I.
The following lemma, which is from , is useful in our analysis.
If is convex and its gradient is -Lipschitz continuous, then
In the following, we first prove the following lemma, which reveals the relationship between the distances of two consecutive iterates to the optimal point.
For both SVRG-I and SVRG-BB, we have the following inequality for the -th epoch:
where is the optimal solution to (1.1).
Let for the -th epoch of SVRG-I or SVRG-BB. Then,
where in the first inequality we used the inequality twice, in the second inequality we applied Lemma 3.4 to and used the Lipschitz continuity of and , and in the last equality we used the facts that and .
In the next, we bound the distance of to conditioned on and .
where in the third equality we used the fact that , and in the second inequality we used the strong convexity of .
By recursively applying the above inequality over , and noting that and , we can obtain
The linear convergence of SVRG-I follows immediately.
In SVRG-I, if and are chosen such that
then SVRG-I (Algorithm 1 with Option I) converges linearly in expectation:
We now give some remarks on this convergence result.
To the best of our knowledge, this is the first time that the linear convergence of SVRG-I is established.
The condition required in (3.3) is different from the condition required in (3.1) for SVRG-II. As , the first term in (3.1) converges to 0 sublinearly, while the first term in (3.3) converges to 0 linearly. On the other hand, the second term in (3.1) reveals that depends on the condition number linearly, while the second term in (3.3) suggests that depends on condition number quadratically. As a result, if the problem is ill-conditioned, then the convergence rate given in Corollary 3.6 might be slow.
The following theorem establishes the linear convergence of SVRG-BB (Algorithm 2).
Denote . It is easy to see that . In SVRG-BB, if is chosen such that
then SVRG-BB (Algorithm 2) converges linearly in expectation:
Using the strong convexity of function , it is easy to obtain the following upper bound for the BB step size computed in Algorithm 2.
Similarly, by the -Lipschitz continuity of , it is easy to obtain that is uniformly lower bounded by . Therefore, in (3.3) can be bounded as:
Substituting (3.5) into the above inequality yields
The desired result follows by applying Lemma 3.5. ∎
In this section, we propose to incorporate the BB method to SGD (1.2). The BB method does not apply to SGD directly, because SGD never computes the full gradient . In SGD, is an unbiased estimation for when is uniformly sampled (see [15, 30] for studies on importance sampling, which does not sample uniformly). Therefore, one may suggest to use to estimate when computing the BB step size using formula (2.4). However, this approach does not work well because of the variance of the stochastic gradient estimates. The recent work by Sopyła and Drozda  suggested several variants of this idea to compute an estimated BB step size using the stochastic gradients. However, these ideas lack theoretical justifications and the numerical results in  show that these approaches are inferior to existing methods such as averaged SGD .
The SGD-BB algorithm we propose in this paper works in the following manner. We call every iterations of SGD as one epoch. Following the idea of SVRG-BB, SGD-BB also uses the same step size computed by the BB formula in every epoch. Our SGD-BB algorithm is described as in Algorithm 3.
We have a few remarks about SGD-BB (Algorithm 3).
SGD-BB takes the average of the stochastic gradients in one epoch as an estimation of the full gradient.
Note that for computing in Algorithm 3, we actually take the absolute value for the BB formula (2.4). This is because that unlike SVRG-BB, in Algorithm 3 is the average of stochastic gradients at different iterates, not an exact full gradient. As a result, the step size generated by (2.4) can be negative. This can be seen from the following argument. Suppose is chosen such that
By substituting this equality into the equation for computing in Algorithm 3, we have
Without taking the absolute value, the denominator of (4.2) is , which can be negative in stochastic settings.
Moreover, from (4.2) we have the following observations. If , then is smaller than . This is reasonable because indicates that the step size is too large and we need to shrink it. If , then it indicates that we should be more aggressive to take larger step size. We found from our numerical experiments that when the iterates are close to optimum, the size of and do not differentiate much. As a result, is usually increased from by using (4.2). Hence, the way we compute in Algorithm 3 is in a sense to dynamically adjust the step size, by evaluating whether we are moving the iterates along the right direction. This kind of idea can be traced back to .
Note that SGD-BB requires the averaged gradients in two epochs to compute the BB step size, which can only be done starting from the third epoch. Therefore, we need to specify the step sizes and for the first two epochs. From our numerical experiments, we found that the performance of SGD-BB is not sensitive to choices of and .
Due to the randomness of the stochastic gradients, the step size computed in SGD-BB may vibrate drastically sometimes and this may cause instability of the algorithm. Inspired by , we propose the following smoothing technique to stabilize the step size.
It is known that in order to guarantee the convergence of SGD, the step sizes are required to be diminishing. Similar as in , we assume the step sizes are in the form of , where is an unknown constant that needs to be estimated, is a pre-specified function that controls the decreasing rate of the step size, and a typical choice of function is . In the -th epoch of Algorithm 3, we have all the previous step sizes generated by the BB method, while the step sizes generated by the function are given by . In order to ensure that these two sets of step sizes are close to each other, we solve the following optimization problem to determine the unknown parameter :
Here we take the logarithms of the step sizes to ensure that the estimation is not dominated by those ’s with large magnitudes. It is easy to verify that the solution to (4.3) is given by
Therefore, the smoothed step size for the -th epoch of Algorithm 3 is:
In practice, we do not need to store all the ’s and can be computed recursively by
The BB step size and the smoothing technique we used in SGD-BB (Algorithm 3) can also be used in other variants of SGD. In this section, we use SAG as an example to illustrate how to incorporate the BB step size. SAG with BB step size (denoted as SAG-BB) is described as in Algorithm 4. Because SAG does not need diminishing step sizes to ensure convergence, in the smoothing technique we just choose . In this case, the smoothed step size
is equal to the geometric mean of all previous BB step sizes.
In this section, we conduct some numerical experiments to demonstrate the efficacy of our SVRG-BB (Algorithm 2) and SGD-BB (Algorithm 3) algorithms. In particular, we apply SVRG-BB and SGD-BB to solve two standard testing problems in machine learning: logistic regression with -norm regularization
and the squared hinge loss SVM with -norm regularization
are the feature vector and class label of the-th sample, respectively, and is a weighting parameter.
We tested SVRG-BB and SGD-BB for (5.1) and (5.2) for three standard real data sets, which were downloaded from the LIBSVM website111www.csie.ntu.edu.tw/~cjlin/libsvmtools/.. Detailed information of these three data sets are given in Table 1.
In this section, we compare SVRG-BB (Algorithm 2) with SVRG (Algorithm 1) for solving (5.1) and (5.2). We used the best-tuned step size for SVRG, and three different initial step sizes for SVRG-BB. For both SVRG-BB and SVRG, we set as suggested in .
The comparison results of SVRG-BB and SVRG are shown in Figure 1. In all the six sub-figures, the -axis denotes the number of epochs , i.e., the number of outer loops in Algorithm 2. In Figures 0(a), 0(b) and 0(c), the -axis denotes the sub-optimality , and in Figures 0(d), 0(e) and 0(f), the -axis denotes the step size . Note that is obtained by running SVRG with the best-tuned step size until it converges, which is a common practice in the testing of stochastic gradient descent methods. In all the six sub-figures, the dashed lines correspond to SVRG with fixed step sizes given in the legends of the figures. Moreover, the dashed lines in black color always represent SVRG with best-tuned fixed step size, and the green dashed lines use a smaller fixed step size, and the red dashed lines use a larger fixed step size, compared with the best-tuned ones. The solid lines correspond to SVRG-BB with different initial step sizes . The solid lines with blue, purple and yellow colors in Figures 0(a) and 0(d) correspond to , , and , respectively; the solid lines with blue, purple and yellow colors in Figures 0(b) and 0(e) correspond to , , and , respectively; the solid lines with blue, purple and yellow colors in Figures 0(c) and 0(f) correspond to , , and , respectively.
It can be seen from Figures 0(a), 0(b) and 0(c) that, SVRG-BB can always achieve the same level of sub-optimality as SVRG with the best-tuned step size. Although SVRG-BB needs slightly more epochs compared with SVRG with the best-tuned step size, it clearly outperforms SVRG with the other two choices of step sizes. Moreover, from Figures 0(d), 0(e) and 0(f) we see that the step sizes computed by SVRG-BB converge to the best-tuned step sizes after about 10 to 15 epochs. From Figure 1 we also see that SVRG-BB is not sensitive to the choice of . Therefore, SVRG-BB has very promising potential in practice because it generates the best step sizes automatically while running the algorithm.
In this section, we compare SGD-BB with smoothing technique (Algorithm 3) with SGD for solving (5.1) and (5.2). We set , and in our experiments. We used when applying the smoothing technique. Since SGD requires diminishing step size to converge, we tested SGD with diminishing step size in the form with different constants . The comparison results are shown in Figure 2. Similar as Figure 1, the dashed line with black color represents SGD with the best-tuned , and the green and red dashed lines correspond to the other two choices of . The solid lines with blue, purple and yellow colors in Figures 1(a) and 1(d) correspond to , , and , respectively; the solid lines with blue, purple and yellow colors in Figures 1(b) and 1(e) correspond to , , and , respectively; the solid lines with blue, purple and yellow colors in Figures 1(c) and 1(f) correspond to , , and , respectively.
From Figures 1(a), 1(b) and 1(c) we can see that SGD-BB gives comparable or even better sub-optimality than SGD with best-tuned diminishing step size, and SGD-BB is significantly better than SGD with the other two choices of step size. From Figures 1(d), 1(e) and 1(f) we see that after only a few epochs, the step sizes generated by SGD-BB approximately coincide with the best-tuned diminishing step sizes. It can also be seen that after only a few epochs, the step sizes are stabilized by the smoothing technique and they approximately follow the same decreasing trend as the best-tuned diminishing step sizes.
In this section, we compare our SGD-BB (Algorithm 3) and SAG-BB (Algorithm 4) with three existing methods: AdaGrad , SAG with line search (denoted as SAG-L) , and a stochastic quasi-Newton method: oLBFGS . For both SGD-BB and SAG-BB, we set and . Because these methods have very different per-iteration complexity, we compare their CPU time needed to achieve the same sub-optimality.
Figures 2(a), 2(b) and 2(c) show the comparison results of SGD-BB and AdaGrad. From these figures we see that AdaGrad usually has a very quick start, but in many cases the convergence becomes slow in later iterations. Besides, AdaGrad is still somewhat sensitive to the initial step sizes. Especially, when a small initial step size is used, AdaGrad is not able to increase the step size to a suitable level. As a contrast, SGD-BB converges very fast in all three tested problems, and it is not sensitive to the initial step size .
Figures 2(d), 2(e) and 2(f) show the comparison results of SAG-BB and SAG-L. From these figures we see that the SAG-L is quite robust and is not sensitive to the choice of . However, SAG-BB is much faster than SAG-L to reach the same sub-optimality on the tested problems.
In this paper we proposed to use the BB method to compute the step sizes for SGD and SVRG, which leads to two new stochastic gradient methods: SGD-BB and SVRG-BB. We proved the linear convergence of SVRG-BB for strongly convex function, and as a by-product, we proved the linear convergence of the original SVRG with option I for strongly convex function. We also proposed a smoothing technique to stabilize the step sizes generated in SGD-BB, and we showed how to incorporate the BB method to other SGD variants such as SAG. We conducted numerical experiments on real data sets to compare the performance of SVRG-BB and SGD-BB with existing methods. The numerical results showed that the performance of our SVRG-BB and SGD-BB is comparable to and sometimes even better than the original SVRG and SGD with best-tuned step sizes, and is superior to some advanced SGD variants.
International Conference on Artificial Intelligence and Statistics, pages 436–443, 2007.