We study nonconvex finite-sum problems of the form
where neither nor the individual () are necessarily convex; just Lipschitz smooth (i.e., Lipschitz continuous gradients). We use to denote all functions of the form (1). We optimize such functions in the Incremental First-order Oracle (IFO) framework (Agarwal & Bottou, 2014) defined below.
For , an IFO takes an index and a point , and returns the pair .
IFO based complexity analysis was introduced to study lower bounds for finite-sum problems. Algorithms that use IFOs are favored in large-scale applications as they require only a small amount first-order information at each iteration. Two fundamental models in machine learning that profit from IFO algorithms are (i) empirical risk minimization, which typically uses convex finite-sum models; and (ii) deep learning, which uses nonconvex ones.
The prototypical IFO algorithm, stochastic gradient descent (Sgd)111We use ‘incremental gradient’ and ‘stochastic gradient’ interchangeably, though we are only interested in finite-sum problems. has witnessed tremendous progress in the recent years. By now a variety of accelerated, parallel, and faster converging versions are known. Among these, of particular importance are variance reduced (VR) stochastic methods (Schmidt et al., 2013; Johnson & Zhang, 2013; Defazio et al., 2014a), which have delivered exciting progress such as linear convergence rates (for strongly convex functions) as opposed to sublinear rates of ordinary Sgd (Robbins & Monro, 1951; Nemirovski et al., 2009). Similar (but not same) benefits of VR methods can also be seen in smooth convex functions. The Svrg algorithm of (Johnson & Zhang, 2013) is particularly attractive here because of its low storage requirement in comparison to the algorithms in (Schmidt et al., 2013; Defazio et al., 2014a).
Despite the meteoric rise of VR methods, their analysis for general nonconvex problems is largely missing. Johnson & Zhang (2013) remark on convergence of Svrg when is locally strongly convex and provide compelling experimental results (Fig. 4 in (Johnson & Zhang, 2013)). However, problems encountered in practice are typically not even locally convex, let alone strongly convex. The current analysis of Svrg does not extend to nonconvex functions as it relies heavily on convexity for controlling the variance. Given the dominance of stochastic gradient methods in optimizing deep neural nets and other large nonconvex models, theoretical investigation of faster nonconvex stochastic methods is much needed.
|Algorithm||Nonconvex||Convex||Gradient Dominated||Fixed Step Size?|
Convex VR methods are known to enjoy the faster convergence rate of GradientDescent but with a much weaker dependence on , without compromising the rate like Sgd. However, it is not clear if these benefits carry beyond convex problems, prompting the central question of this paper:
For nonconvex functions in , can one achieve convergence rates faster than both Sgd and GradientDescent using an IFO? If so, then how does the rate depend on and on the number of iterations performed by the algorithm?
Perhaps surprisingly, we provide an affirmative answer to this question by showing that a careful selection of parameters in Svrg leads to faster convergence than both Sgd and GradientDescent. To our knowledge, ours is the first work to improve convergence rates of Sgd and GradientDescent for IFO-based nonconvex optimization.
Main Contributions. We summarize our main contributions below and also list the key results in Table 1.
We analyze nonconvex stochastic variance reduced gradient (Svrg), and prove that it has faster rates of convergence than GradientDescent and ordinary Sgd. We show that Svrg is faster than GradientDescent by a factor of (see Table 1).
We provide new theoretical insights into the interplay between step-size, iteration complexity and convergence of nonconvex Svrg (see Corollary 2).
For an interesting nonconvex subclass of called gradient dominated functions (Polyak, 1963; Nesterov & Polyak, 2006), we propose a variant of Svrg that attains a global linear rate of convergence. We improve upon many prior results for this subclass of functions (see Section 3.1). To the best of our knowledge, ours is the first work that shows a stochastic method with linear convergence for gradient dominated functions.
We analyze mini-batch nonconvex Svrg and show that it provably benefits from mini-batching. Specifically, we show theoretical linear speedups in parallel settings for large mini-batch sizes. By using a mini-batch of size , we show that mini-batch nonconvex Svrg is faster by a factor of (Theorem 7). We are not aware of any prior work on mini-batch first-order stochastic methods that shows linear speedup in parallel settings for nonconvex optimization.
Our analysis yields as a byproduct a direct convergence analysis for Svrg for smooth convex functions (Section 4).
We examine a variant of Svrg (called Msvrg) that has faster rates than both GradientDescent and Sgd.
1.1 Related Work
Convex. Bertsekas (2011) surveys several incremental gradient methods for convex problems. A key reference for stochastic convex optimization (for ) is (Nemirovski et al., 2009). Faster rates of convergence are attained for problems in by VR methods, see e.g., (Defazio et al., 2014a; Johnson & Zhang, 2013; Schmidt et al., 2013; Konečný et al., 2015; Shalev-Shwartz & Zhang, 2013; Defazio et al., 2014b). Asynchronous VR frameworks are developed in (Reddi et al., 2015). Agarwal & Bottou (2014); Lan & Zhou (2015) study lower-bounds for convex finite-sum problems. Shalev-Shwartz (2015) prove linear convergence of stochastic dual coordinate ascent when the individual () are nonconvex but is strongly convex. They do not study the general nonconvex case. Moreover, even in their special setting our results improve upon theirs for the high condition number regime.
Nonconvex. Sgd dates at least to the seminal work (Robbins & Monro, 1951); and since then it has been developed in several directions (Poljak & Tsypkin, 1973; Ljung, 1977; Bottou, 1991; Kushner & Clark, 2012). In the (nonsmooth) finite-sum setting, Sra (2012) considers proximal splitting methods, and analyzes asymptotic convergence with nonvanishing gradient errors. Hong (2014) studies a distributed nonconvex incremental ADMM algorithm.
These works, however, only prove expected convergence to stationary points and often lack analysis of rates. The first nonasymptotic convergence rate analysis for Sgd is in (Ghadimi & Lan, 2013), who show that Sgd ensures in iterations. A similar rate for parallel and distributed Sgd was shown recently in (Lian et al., 2015). GradientDescent is known to ensure in iterations (Nesterov, 2003, Chap. 1.2.3).
The first analysis of nonconvex Svrg seems to be due to Shamir (2014)
, who considers the special problem of computing a few leading eigenvectors (e.g., for PCA); see also the follow up work(Shamir, 2015). Finally, we note another interesting example, stochastic optimization of locally quasi-convex functions (Hazan et al., 2015), wherein actually a convergence in function value is shown.
2 Background & Problem Setup
We say is -smooth if there is a constant such that
Throughout, we assume that the functions in (1) are -smooth, so that for all . Such an assumption is very common in the analysis of first-order methods. Here the Lipschitz constant is assumed to be independent of . A function is called -strongly convex if there is such that
The quantity is called the condition number of , whenever is -smooth and -strongly convex. We say is non-strongly convex when is -strongly convex.
where is a global minimizer of . Note that such a function need not be convex; it is also easy to show that a -strongly convex function is -gradient dominated.
We analyze convergence rates for the above classes of functions. Following Nesterov (2003); Ghadimi & Lan (2013) we use to judge when is iterate approximately stationary. Contrast this with Sgd for convex , where one uses or as a convergence criterion. Unfortunately, such criteria cannot be used for nonconvex functions due to the hardness of the problem. While the quantities and or are not comparable in general (see (Ghadimi & Lan, 2013)), they are typically assumed to be of similar magnitude. Throughout our analysis, we do not assume to be constant, and report dependence on it in our results. For our analysis, we need the following definition.
A point is called -accurate if . A stochastic iterative algorithm is said to achieve -accuracy in iterations if , where the expectation is over the stochasticity of the algorithm.
We introduce one more definition useful in the analysis of Sgd methods for bounding the variance.
We say has a -bounded gradient if for all and .
2.1 Nonconvex SGD: Convergence Rate
By using a uniformly randomly chosen (with replacement) index from , Sgd
uses an unbiased estimate of the gradient at each iteration. Under appropriate conditions,Ghadimi & Lan (2013) establish convergence rate of Sgd to a stationary point of . Their results include the following theorem.
For completeness we present a proof in the appendix. Note that our choice of step size requires knowing the total number of iterations in advance. A more practical approach is to use a or . A bound on IFO calls made by Algorithm 1 follows as a corollary of Theorem 1.
Suppose function has -bounded gradient, then the IFO complexity of Algorithm 1 to obtain an -accurate solution is .
As seen in Theorem 1, Sgd has a convergence rate of . This rate is not improvable in general even when the function is (non-strongly) convex (Nemirovski & Yudin, 1983). This barrier is due to the variance introduced by the stochasticity of the gradients, and it is not clear if better rates can be obtained Sgd even for convex .
3 Nonconvex SVRG
We now turn our focus to variance reduced methods. We use Svrg (Johnson & Zhang, 2013), an algorithm recently shown to be very effective for reducing variance in convex problems. As a result, it has gained considerable interest in both machine learning and optimization communities. We seek to understand its benefits for nonconvex optimization. For reference, Algorithm 2 presents Svrg’s pseudocode.
Observe that Algorithm 2
operates in epochs. At the end of epoch, a full gradient is calculated at the point , requiring calls to the IFO. Within its inner loop Svrg performs stochastic updates. The total number of IFO calls for each epoch is thus . For , the algorithm reduces to the classic GradientDescent algorithm. Suppose is chosen to be (typically used in practice), then the total IFO calls per epoch is . To enable a fair comparison with Sgd, we assume that the total number of inner iterations across all epochs in Algorithm 2 is . Also note a simple but important implementation detail: as written, Algorithm 2 requires storing all the iterates (
). This storage can be avoided by keeping a running average with respect to the probability distribution.
Algorithm 2 attains linear convergence for strongly convex (Johnson & Zhang, 2013); for non-strongly convex functions, rates faster than Sgd can be shown by using an indirect perturbation argument—see e.g., (Konečný & Richtárik, 2013; Xiao & Zhang, 2014).
We first state an intermediate result for the iterates of nonconvex Svrg. To ease exposition, we define
for some parameters and (to be defined shortly).
Our first main result is the following theorem that provides convergence rate of Algorithm 2.
Furthermore, we can also show that nonconvex Svrg exhibits expected descent (in objective) after every epoch. The condition that is a multiple of is solely for convenience and can be removed by slight modification of the theorem statement. Note that the value above can depend on . To obtain an explicit dependence, we simplify it using specific choices for and , as formalized below.
By rewriting the above result in terms IFO calls, we get the following general corollary for nonconvex Svrg.
Corollary 2 shows the interplay between step size and the IFO complexity. We observe that the number of IFO calls is minimized in Corollary 2 when . This gives rise to the following key results of the paper.
3.1 Gradient Dominated Functions
Before ending our discussion on convergence of nonconvex Svrg, we prove a linear convergence rate for the class of -gradient dominated functions (2). For ease of exposition, assume that , a property analogous to the “high condition number regime” for strongly convex functions typical in machine learning. Note that gradient dominated functions can be nonconvex.
In fact, for -gradient dominated functions we can prove a stronger result of global linear convergence.
An immediate consequence is the following.
Note that GradientDescent can also achieve linear convergence rate for gradient dominated functions (Polyak, 1963). However, GradientDescent requires IFO calls to obtain an -accurate solution as opposed to for Svrg. Similar (but not the same) gains can be seen for Svrg for strongly convex functions (Johnson & Zhang, 2013). Also notice that we did not assume anything except smoothness on the individual functions in the above results. In particular, the following corollary is also an immediate consequence.
Recall that here denotes the condition number for a -strongly convex function. Corollary 6 follows from Corollary 5 upon noting that -strongly convex function is -gradient dominated. Theorem 5 generalizes the linear convergence result in (Johnson & Zhang, 2013) since it allows nonconvex . Observe that Corollary 6 also applies when is strongly convex for all , though in this case a more refined result can be proved (Johnson & Zhang, 2013).
Finally, we note that our result also improves on a recent result on Sdca in the setting of Corollary 6 when the condition number is reasonably large – a case that typically arises in machine learning. More precisely, for -regularized empirical loss minimization, Shalev-Shwartz (2015) show that Sdca requires iterations when the ’s are possibly nonconvex but their sum is strongly convex. In comparison, we show that Algorithm 3 requires iterations, which is an improvement over Sdca when .
4 Convex Case
In the previous section, we showed nonconvex Svrg converges to a stationary point at the rate . A natural question is whether this rate can be improved if we assume convexity? We provide an affirmative answer. For non-strongly convex functions, this yields a direct analysis (i.e., not based on strongly convex perturbations) for Svrg. While we state our results in terms of stationarity gap for the ease of comparison, our analysis also provides rates with respect to the optimality gap (see the proof of Theorem 6 in the appendix).
We now state corollaries of this theorem that explicitly show the dependence on in the convergence rates.
The above result uses a step size that depends on . For the convex case, we can also use step sizes independent of . The following corollary states the associated result.
We can rewrite these corollaries in terms of IFO complexity to get the following corollaries.
These results follow from Corollary 7 and Corollary 8 and noting that for the total IFO calls made by Algorithm 2 is . It is instructive to quantitatively compare Corollary 9 and Corollary10. With a step size independent of , the convergence rate of Svrg has a dependence that is in the order of (Corollary 8). But this dependence can be reduced to by either carefully selecting a step size that diminishes with (Corollary 7) or by using a good initial point obtained by, say, running iterations of Sgd.
We emphasize that the convergence rate for convex case can be improved significantly by slightly modifying the algorithm (either by adding an appropriate strongly convex perturbation (Xiao & Zhang, 2014) or by using a choice of that changes with epoch (Zhu & Yuan, 2015)). However, it is not clear if these strategies provide any theoretical gains for the general nonconvex case.
5 Mini-batch Nonconvex SVRG
In this section, we study the mini-batch version of Algorithm 2. Mini-batching is a popular strategy, especially in multicore and distributed settings as it greatly helps one exploit parallelism and reduce the communication costs. The pseudocode for mini-batch nonconvex Svrg (Algorithm 4) is provided in the supplement due to lack of space. The key difference between the mini-batch Svrg and Algorithm 2 lies in lines 6 to 8. To use mini-batches we replace line 6 with sampling (with replacement) a mini-batch of size ; lines 7 to 8 are replaced with the following updates:
When , this reduces to Algorithm 2. Mini-batch is typically used to reduce the variance of the stochastic gradient and increase the parallelism. Lemma 4 (in Section G of the appendix) shows the reduction in the variance of stochastic gradients with mini-batch size . Using this lemma, one can derive the mini-batch equivalents of Lemma 1, Theorem 2 and Theorem 3. However, for the sake of brevity, we directly state the following main result for mini-batch Svrg.
It is important to compare this result with mini-batched Sgd. For a batch size of , Sgd obtains a rate of (Dekel et al., 2012) (obtainable by a simple modification of Theorem 1). Specifically, Sgd has a dependence on the batch size. In contrast, Theorem 7 shows that Svrg has a much better dependence of on the batch size. Hence, compared to Sgd, Svrg allows more efficient mini-batching. More formally, in terms of IFO queries we have the following result.
Corollary 11 shows an interesting property of mini-batch Svrg. First, note that IFO calls are required for calculating the gradient on a mini-batch of size . Hence, Svrg does not gain on IFO complexity by using mini-batches. However, if the gradients are calculated in parallel, then this leads to a theoretical linear speedup in multicore and distributed settings. In contrast, Sgd does not yield an efficient mini-batch strategy as it requires IFO calls for achieving an -accurate solution (Li et al., 2014). Thus, the performance of Sgd degrades with mini-batching.
6 Comparison of the convergence rates
In this section, we give a comprehensive comparison of results obtained in this paper. In particular, we compare key aspects of the convergence rates for Sgd, GradientDescent, and Svrg. The comparison is based on IFO complexity to achieve an -accurate solution.
Dependence on : The number of IFO calls of Svrg and GradientDescent depend explicitly on . In contrast, the number of oracle calls of Sgd is independent of (Theorem 1). However, this comes at the expense of worse dependence on . The number of IFO calls in GradientDescent is proportional to . But for Svrg this dependence reduces to for convex (Corollary 7) and for nonconvex (Corollary 3) problems. Whether this difference in dependence on is due to nonconvexity or just an artifact of our analysis is an interesting open problem.
Dependence on : The dependence on (or alternatively ) follows from the convergence rates of the algorithms. Sgd is seen to depend as on , regardless of convexity or nonconvexity. In contrast, for both convex and nonconvex settings, Svrg and GradientDescent converge as . Furthermore, for gradient dominated functions, Svrg and GradientDescent have global linear convergence. This speedup in convergence over Sgd is especially significant when medium to high accuracy solutions are required (i.e., is small).
Assumptions used in analysis: It is important to understand the assumptions used in deriving the convergence rates. All algorithms assume Lipschitz continuous gradients. However, Sgd requires two additional subtle but important assumptions: -bounded gradients and advance knowledge of (since its step sizes depend on ). On the other hand, both Svrg and GradientDescent do not require these assumptions, and thus, are more flexible.
Step size / learning rates: It is valuable to compare the step sizes used by the algorithms. The step sizes of Sgd shrink as the number of iterations increases—an undesirable property. On the other hand, the step sizes of Svrg and GradientDescent are independent of . Hence, both these algorithms can be executed with a fixed step size. However, Svrg uses step sizes that depend on (see Corollary 3 and Corollary 7). A step size independent of can be used for Svrg for convex , albeit at cost of worse dependence on (Corollary 8). GradientDescent does not have this issue as its step size is independent of both and .
Dependence on initial point and mini-batch: Svrg is more sensitive to the initial point in comparison to Sgd. This can be seen by comparing Corollary 3 (of Svrg) to Theorem 1 (of Sgd). Hence, it is important to use a good initial point for Svrg. Similarly, a good mini-batch can be beneficial to Svrg. Moreover, mini-batches not only provides parallelism but also good theoretical guarantees (see Theorem 7). In contrast, the performance gain in Sgd with mini-batches is not very pronounced (see Section 5).
7 Best of two worlds
We have seen in the previous section that Svrg combines the benefits of both GradientDescent and Sgd. We now show that these benefits of Svrg can be made more pronounced by an appropriate step size under additional assumptions. In this case, the IFO complexity of Svrg is lower than those of Sgd and GradientDescent. This variant of Svrg (Msvrg) chooses a step size based on the total number of iterations (or alternatively ). For our discussion below, we assume that .
An almost identical reasoning can be applied when is convex to get the bounds specified in Table 1. Hence, we omit the details and directly state the following result.
Suppose is convex for and has -bounded gradients, then the IFO complexity of Algorithm 2 (with step size , and for and ) to achieve an -accurate solution is .
Msvrg has a convergence rate faster than those of both Sgd and Svrg, though this benefit is not without cost. Msvrg, in contrast to Svrg, uses the additional assumption of -bounded gradients. Furthermore, its step size is not fixed since it depends on the number of iterations . While it is often difficult in practice to compute the step size of Msvrg (Theorem 8), it is typical to try multiple step sizes and choose the one with the best results.
We present our empirical results in this section. For our experiments, we study the problem of multiclass classification using neural networks. This is a typical nonconvex problem encountered in machine learning.
Experimental Setup. We train neural networks with one fully-connected hidden layer of 100 nodes and 10 softmax output nodes. We use -regularization for training. We use CIFAR-10222www.cs.toronto.edu/~kriz/cifar.html, MNIST333http://yann.lecun.com/exdb/mnist/, and STL-10444https://cs.stanford.edu/~acoates/stl10/ datasets for our experiments. These datasets are standard in the neural networks literature. The regularization is 1e-3 for CIFAR-10 and MNIST, and 1e-2 for STL-10. The features in the datasets are normalized to the interval . All the datasets come with a predefined split into training and test datasets.
We compare Sgd (the de-facto algorithm for training neural networks) against nonconvex Svrg. The step size (or learning rate) is critical for Sgd. We set the learning rate of Sgd using the popular inverse schedule , where and are chosen so that Sgd gives the best performance on the training loss. In our experiments, we also use ; this results in a fixed step size for Sgd. For Svrg, we use a fixed step size as suggested by our analysis. Again, the step size is chosen so that Svrg gives the best performance on the training loss.
Initialization & mini-batching. Initialization is critical to training of neural networks. We use the normalized initialization in (Glorot & Bengio, 2010) where parameters are chosen uniformly from , where and are the number of input and output layers of the neural network, respectively.
For Svrg, we use iterations of Sgd for CIFAR-10 and MINST and iterations of Sgd for STL-10 before running Algorithm 2. Such initialization is standard for variance reduced schemes even for convex problems (Johnson & Zhang, 2013; Schmidt et al., 2013). As noted earlier in Section 6, Svrg is more sensitive than Sgd to the initial point, so such an initialization is typically helpful. We use mini-batches of size 10 in our experiments. Sgd with mini-batches is common in training neural networks. Note that mini-batch training is especially beneficial for Svrg, as shown by our analysis in Section 5. Along the lines of theoretical analysis provided by Theorem 7, we use an epoch size in our experiments.
Results. We report objective function (training loss), test error (classification error on the test set), and (convergence criterion throughout our analysis) for the datasets. For all the algorithms, we compare these criteria against the number of effective passes through the data, i.e., IFO calls divided by . This includes the cost of calculating the full gradient at the end of each epoch of Svrg. Due to the Sgd initialization in Svrg and mini-batching, the Svrg plots start from x-axis value of 10 for CIFAR-10 and MNIST and 20 for STL-10. Figure 1 shows the results for our experiment. It can be seen that the for Svrg is lower compared to Sgd, suggesting faster convergence to a stationary point. Furthermore, the training loss is also lower compared to Sgd in all the datasets. Notably, the test error for CIFAR-10 is lower for Svrg, indicating better generalization; we did not notice substantial difference in test error for MNIST and STL-10 (see Section H in the appendix). Overall, these results on a network with one hidden layer are promising; it will be interesting to study Svrg for deep neural networks in the future.
In this paper, we examined a VR scheme for nonconvex optimization. We showed that by employing VR in stochastic methods, one can perform better than both Sgd and GradientDescent in the context of nonconvex optimization. When the function in (1) is gradient dominated, we proposed a variant of Svrg that has linear convergence to the global minimum. Our analysis shows that Svrg has a number of interesting properties that include convergence with fixed step size, descent property after every epoch; a property that need not hold for Sgd. We also showed that Svrg, in contrast to Sgd, enjoys efficient mini-batching, attaining speedups linear in the size of the mini-batches in parallel settings. Our analysis also reveals that the initial point and use of mini-batches are important to Svrg.
Before concluding the paper, we would like to discuss the implications of our work and few caveats. One should exercise some caution while interpreting the results in the paper. All our theoretical results are based on the stationarity gap. In general, this does not necessarily translate to optimality gap or low training loss and test error. One criticism against VR schemes in nonconvex optimization is the general wisdom that variance in the stochastic gradients of Sgd can actually help it escape local minimum and saddle points. In fact, Ge et al. (2015) add additional noise to the stochastic gradient in order to escape saddle points. However, one can reap the benefit of VR schemes even in such scenarios. For example, one can envision an algorithm which uses Sgd as an exploration tool to obtain a good initial point and then uses a VR algorithm as an exploitation tool to quickly converge to a good local minimum. In either case, we believe variance reduction can be used as an important tool alongside other tools like momentum, adaptive learning rates for faster and better nonconvex optimization.
- Agarwal & Bottou (2014) Agarwal, Alekh and Bottou, Leon. A lower bound for the optimization of finite sums. arXiv:1410.0723, 2014.
- Bertsekas (2011) Bertsekas, Dimitri P. Incremental gradient, subgradient, and proximal methods for convex optimization: A survey. In S. Sra, S. Nowozin, S. Wright (ed.), Optimization for Machine Learning. MIT Press, 2011.
- Bottou (1991) Bottou, Léon. Stochastic gradient learning in neural networks. Proceedings of Neuro-Nımes, 91(8), 1991.
- Defazio et al. (2014a) Defazio, Aaron, Bach, Francis, and Lacoste-Julien, Simon. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In NIPS 27, pp. 1646–1654, 2014a.
- Defazio et al. (2014b) Defazio, Aaron J, Caetano, Tibério S, and Domke, Justin. Finito: A faster, permutable incremental gradient method for big data problems. arXiv:1407.2710, 2014b.
- Dekel et al. (2012) Dekel, Ofer, Gilad-Bachrach, Ran, Shamir, Ohad, and Xiao, Lin. Optimal distributed online prediction using mini-batches. The Journal of Machine Learning Research, 13(1):165–202, January 2012. ISSN 1532-4435.
Ge et al. (2015)
Ge, Rong, Huang, Furong, Jin, Chi, and Yuan, Yang.
Escaping from saddle points - online stochastic gradient for tensor decomposition.In Proceedings of The 28th Conference on Learning Theory, COLT 2015, pp. 797–842, 2015.
- Ghadimi & Lan (2013) Ghadimi, Saeed and Lan, Guanghui. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013. doi: 10.1137/120880811.
Glorot & Bengio (2010)
Glorot, Xavier and Bengio, Yoshua.
Understanding the difficulty of training deep feedforward neural
In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’10), 2010.
- Hazan et al. (2015) Hazan, Elad, Levy, Kfir, and Shalev-Shwartz, Shai. Beyond convexity: Stochastic quasi-convex optimization. In Advances in Neural Information Processing Systems, pp. 1585–1593, 2015.
- Hong (2014) Hong, Mingyi. A distributed, asynchronous and incremental algorithm for nonconvex optimization: An admm based approach. arXiv preprint arXiv:1412.6058, 2014.
- Johnson & Zhang (2013) Johnson, Rie and Zhang, Tong. Accelerating stochastic gradient descent using predictive variance reduction. In NIPS 26, pp. 315–323, 2013.
- Konečný & Richtárik (2013) Konečný, Jakub and Richtárik, Peter. Semi-Stochastic Gradient Descent Methods. arXiv:1312.1666, 2013.
- Konečný et al. (2015) Konečný, Jakub, Liu, Jie, Richtárik, Peter, and Takáč, Martin. Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting. arXiv:1504.04407, 2015.
- Kushner & Clark (2012) Kushner, Harold Joseph and Clark, Dean S. Stochastic approximation methods for constrained and unconstrained systems, volume 26. Springer Science & Business Media, 2012.
- Lan & Zhou (2015) Lan, Guanghui and Zhou, Yi. An optimal randomized incremental gradient method. arXiv:1507.02000, 2015.
- Li et al. (2014) Li, Mu, Zhang, Tong, Chen, Yuqiang, and Smola, Alexander J. Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, pp. 661–670. ACM, 2014.
- Lian et al. (2015) Lian, Xiangru, Huang, Yijun, Li, Yuncheng, and Liu, Ji. Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization. In NIPS, 2015.
- Ljung (1977) Ljung, Lennart. Analysis of recursive stochastic algorithms. Automatic Control, IEEE Transactions on, 22(4):551–575, 1977.
- Nemirovski et al. (2009) Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574–1609, 2009.
- Nemirovski & Yudin (1983) Nemirovski, Arkadi and Yudin, D. Problem Complexity and Method Efficiency in Optimization. John Wiley and Sons, 1983.
- Nesterov (2003) Nesterov, Yurii. Introductory Lectures On Convex Optimization: A Basic Course. Springer, 2003.
- Nesterov & Polyak (2006) Nesterov, Yurii and Polyak, Boris T. Cubic regularization of newton method and its global performance. Mathematical Programming, 108(1):177–205, 2006.
- Poljak & Tsypkin (1973) Poljak, BT and Tsypkin, Ya Z. Pseudogradient adaptation and training algorithms. Automation and Remote Control, 34:45–67, 1973.
- Polyak (1963) Polyak, B.T. Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics, 3(4):864–878, January 1963.
- Reddi et al. (2015) Reddi, Sashank, Hefny, Ahmed, Sra, Suvrit, Poczos, Barnabas, and Smola, Alex J. On variance reduction in stochastic gradient descent and its asynchronous variants. In NIPS 28, pp. 2629–2637, 2015.
- Robbins & Monro (1951) Robbins, H. and Monro, S. A stochastic approximation method. Annals of Mathematical Statistics, 22:400–407, 1951.
- Schmidt et al. (2013) Schmidt, Mark W., Roux, Nicolas Le, and Bach, Francis R. Minimizing Finite Sums with the Stochastic Average Gradient. arXiv:1309.2388, 2013.
- Shalev-Shwartz (2015) Shalev-Shwartz, Shai. SDCA without duality. CoRR, abs/1502.06177, 2015.
- Shalev-Shwartz & Zhang (2013) Shalev-Shwartz, Shai and Zhang, Tong. Stochastic dual coordinate ascent methods for regularized loss. The Journal of Machine Learning Research, 14(1):567–599, 2013.
- Shamir (2014) Shamir, Ohad. A stochastic PCA and SVD algorithm with an exponential convergence rate. arXiv:1409.2848, 2014.
- Shamir (2015) Shamir, Ohad. Fast stochastic algorithms for SVD and PCA: Convergence properties and convexity. arXiv:1507.08788, 2015.
- Sra (2012) Sra, Suvrit. Scalable nonconvex inexact proximal splitting. In NIPS, pp. 530–538, 2012.
- Xiao & Zhang (2014) Xiao, Lin and Zhang, Tong. A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization, 24(4):2057–2075, 2014.
- Zhu & Yuan (2015) Zhu, Zeyuan Allen and Yuan, Yang. Univr: A universal variance reduction framework for proximal stochastic gradient method. CoRR, abs/1506.01972, 2015.
Appendix A Nonconvex SGD: Convergence Rate
Proof of Theorem 1
We include the proof here for completeness. Please refer to (Ghadimi & Lan, 2013) for a more general result.
The iterates of Algorithm 1 satisfy the following bound:
The first inequality follows from Lipschitz continuity of . The second inequality follows from the update in Algorithm 1 and since (unbiasedness of the stochastic gradient). The last step uses our assumption on gradient boundedness. Rearranging Equation (5) we obtain
Summing Equation (6) from to and using that is constant we obtain
The first step holds because the minimum is less than the average. The second and third steps are obtained from Equation (6) and the fact that , respectively. The final inequality follows upon using . By setting
in the above inequality, we get the desired result. ∎
Appendix B Nonconvex SVRG
In this section, we provide the proofs of the results for nonconvex Svrg. We first start with few useful lemmas and then proceed towards the main results.
Since is -smooth we have
Using the Svrg update in Algorithm 2 and its unbiasedness, the right hand side above is further upper bounded by
Consider now the Lyapunov function
For bounding it we will require the following: