Non-ergodic Convergence Analysis of Heavy-Ball Algorithms

by   Tao Sun, et al.
NetEase, Inc

In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a relaxed strongly convex condition, the linear convergence is established under weaker assumptions on the step size and inertial parameter than made in the existing literature. We extend our results to multi-block version of the algorithm with both the cyclic and stochastic update rules. In addition, our results can also be extended to decentralized optimization, where the ergodic analysis is not applicable.


page 1

page 2

page 3

page 4


Linearly convergent stochastic heavy ball method for minimizing generalization error

In this work we establish the first linear convergence result for the st...

Gradient Temporal Difference with Momentum: Stability and Convergence

Gradient temporal difference (Gradient TD) algorithms are a popular clas...

Stochastic Heavy Ball

This paper deals with a natural stochastic optimization procedure derive...

From Averaging to Acceleration, There is Only a Step-size

We show that accelerated gradient descent, averaged gradient descent and...

Sequential convergence of AdaGrad algorithm for smooth convex optimization

We prove that the iterates produced by, either the scalar step size vari...

Linear Convergence of Adaptive Stochastic Gradient Descent

We prove that the norm version of the adaptive stochastic gradient metho...

The (1+1)-ES Reliably Overcomes Saddle Points

It is known that step size adaptive evolution strategies (ES) do not con...


In this paper, we study the Heavy-ball algorithm first proposed by Polyak (1964), for solving the following unconstrained minimization problem


where is convex and differentiable, and is Lipschitz continuous with constant . The Heavy-ball method iterates


where is the step size and is the inertial parameter. Different from the gradient descent algorithm, the sequence generated by Heavy-ball method is not Fejér monotone due to the inertial term . This poses a challenge in proving the convergence rate of in the convex case. In the existing literature, the sublinear convergence rate of the Heavy-ball has been proved only in the sense of ergodicity.

When the objective function is twice continuously differentiable and strongly convex (i.e., almost quadratic), the Heavy-ball method provably converges linearly. Under a weaker assumption that the objective function is nonconvex but Lipschitz differentiable, Zavriev and Kostyuk (1993) proved that the sequence generated by the Heavy-ball method will converge to a critical point, yet without specifying the convergence rate. The smoothness of objective function is crucial for convergence of the Heavy-ball. Indeed, it can be divergent for a strongly convex but nonsmooth function as suggested by Lessard, Recht, and Packard (2016). Different from the classical gradient descent methods, the Heavy-ball algorithm fails to generate a Fejér monotone sequence. In the convex and smooth case, the only result about convergence rate, to our knowledge, is the ergodic rate in terms of the objective value (Ghadimi, Feyzmahdavian, and Johansson, 2015), i.e., . The linear convergence of Heavy-ball algorithm was proved under the strongly convexity assumption by Ghadimi, Feyzmahdavian, and Johansson (2015). But the authors imposed a restrictive assumption on the inertial parameter . Specifically, when the strongly convex constant is tiny, the convergence result holds only for a small range of values. By incorporating the idea of proximal mapping, the inertial proximal gradient algorithm (iPiano) was proposed in (Ochs et al., 2014), whose convergence in nonconvex case was thoroughly discussed. Locally linear convergence of iPiano and Heavy-ball method was later proved in (Ochs, 2016). In the strongly convex case, the linear convergence was proved for iPiano with fixed (Ochs, Brox, and Pock, 2015). In the paper (Pock and Sabach, 2016), inertial Proximal Alternating Linearized Minimization (iPALM) was introduced as a variant of iPiano for solving the two-block regularized problem. Xu and Yin (2013)

analyzed the Heavy-ball algorithm in tensor minimization problems. Stochastic versions of heavy-ball have also been introduced

(Loizou and Richtárik, 2017b, a). A multi-step heavy-ball algorithm was analyzed in (Liang, Fadili, and Peyré, 2016). The inertial methods are also developed and studied in the operator research by Combettes and Glaudin (2017). None of the aforementioned Heavy-ball based algorithms, however, provides a non-ergodic convergence rate.


In this paper, we establish the first non-ergodic convergence result in general convex case. More precisely, we prove that for convex and coercive 111We say is coercive, if as .. Compared with existing result in (Ghadimi, Feyzmahdavian, and Johansson, 2015), ours allows a larger step size . We also prove a linear convergence result under a restricted strongly convex condition, weaker than the strong convexity assumption. In short, we make weaker assumptions on the step size, on the inertial parameter, as well as on the convexity of the objective function. The convergence of multi-block extensions of Heavy-ball method is studied. The sublinear and linear convergence rates are proved for the cyclic and stochastic update rules, respectively. In addition, we extend our analysis to the decentralized Heavy-ball method, where the ergodic analysis is not applicable. Our theoretical results are based on a novel Lyapunov function, which is motivated by a modified dynamical system.

A dynamical system interpretation

It has been long known that the Heavy-ball method is equivalent to the discretization of the following second-order ODE (Alvarez, 2000):


for some . In the case , the Heavy-ball method boils down to the standard gradient descent, which is known to be the discretization of the following first-order ODE


The dynamical system (3), however, misses essential information about relation between and . Specifically, if we replace by with being the discretization step size, then it holds that

Since both and can be viewed as the discretization of , we propose to modify (3) by adding the following constraint


where . In next section, we will devise a useful Lyapunov function by exploiting the additive constraint (5) and establish the asymptotic non-ergodic sublinear convergence rate in the continuous setting. Finally, we will “translate” this analysis into that in discretized setting.

Analysis of the dynamical system

We analyze the modified dynamical system (3) + (5). The existence of the solution is beyond the scope of this paper and will not be discussed here. Let us assume that is coercive, , and . We consider the Lyapunov function


and refer the readers to the relevant equations (3)-(6). A direct calculation gives


which means is non-increasing. As a result,

By the coercivity of , is bounded. Then by the continuity of , is bounded; using (3), is also bounded. By the triangle inequality, we have


Since , we obtain the boundedness of ; by (5), is also bounded. Let , we have


where is due to the convexity of ; is due to the Young’s inequality; is due to (3); is due to the triangle inequality; is because of (5). Denote

Since and are both bounded, we have . Using (Analysis of the dynamical system), we have


Combining (Analysis of the dynamical system) and (Analysis of the dynamical system), we have , or equivalently,


Taking the integral of both sides from to and noting that (we have assumed that ), we get

and thus . Since , we have

Thus we have derived the asymptotic sublinear convergence rate for .

Convergence analysis of Heavy-ball

In this section, we prove convergence rates of Heavy-ball method. The core of the proof is to construct a proper Lyapunov function. The expression of in (6) suggests the Lyapunov function be of the form for some . In fact, we have the following sufficient descent lemma. All the technical proofs for the rest of the paper, will be provided in the supplementary materials.

Lemma 1

Suppose is convex with -Lipschitz gradient and . Let be generated by the Heavy-ball method with non-increasing . By choosing the step size

with fixed , we have


According to Lemma 1, a potentially useful Lyapunov function is , as it has the descent property shown in (1). However, it does not fulfill the relation in (Analysis of the dynamical system)  222That is, we are not able build a useful error relation for .. Therefore, we rewrite (1), so that the new right-hand-side contains something like . It turns out that a better Lyapunov function reads




We can see is in line with the discretization of (6).

Given the Lyapunov function in (13), we present a key technical lemma.

Lemma 2

Suppose the assumptions of Lemma 1 hold. Let denote the projection of onto , assumed to exist, and define


Then it holds that


We see that (2) is the discretization of (11) if .

Sublinear convergence

We present the non-ergodic convergence rate of the function value. This rate holds when . We define


In our following settings, we can see it actually holds that .

Theorem 1

Under the assumptions of Lemma 1 and assumptions that and is coercive, We have


To our best knowledge, this is the first non-ergodic result established for Heavy-ball algorithm in convex case. The definition of implies , so it holds that

which is on the same order of complexity as that in gradient descent.

The coercivity assumption on is crucial for Theorem 1. When the function fails to be coercive, we need to assume summable instead.

Corollary 1

Suppose the assumptions of Lemma 1 hold, and .333A classical example is , where . Let be generated by the Heavy-ball algorithm and be coercive. Then,

Linear convergence with restricted strong convexity

We say the function satisfies a restricted strongly convex condition (Lai and Yin, 2013), if


where is the projection of onto the set , and . Restricted strong convexity is weaker than the strong convexity. For example, let us consider the function with . When fails to be full row-rank, is not strongly convex but restricted strongly convex.

Theorem 2

Suppose the assumptions of Theorem 1 hold, and that satisfies condition (19). Then we have

for and .

Our result improves the linear convergence established by Ghadimi, Feyzmahdavian, and Johansson (2015) in two aspects: Firstly, The strongly convex assumption is weakened to (19). Secondly, The step size and inertial parameter are chosen independent of the strongly convex constants.

Cyclic coordinate descent Heavy-ball algorithm

In this section, we consider the multi-block version of Heavy-ball algorithm and prove its convergence rates under convexity assumption. The minimization problem reads


The function is assumed to satisfy


With (21), we can easily obtain


The proof is similar to [Lemma 1.2.3,Nesterov (2013)], and we shall skip it here. We denote

with the convention . The cyclic coordinate descent inertial algorithm iterates: for from to ,


where . Our analysis relies on the following assumption:

A1: for any , the parameters is non-increasing.

Lemma 3

Let be a convex function satisfying (21), and finite . Let be generated by scheme (23) and Assumption A1 hold. Choosing the step size

for arbitrary fixed , we have


where .

We consider the following similar Lyapunov function in the analysis of cyclic coordinate descent Heavy-ball algorithm




Then we have the following lemma.

Lemma 4

Suppose the conditions Lemma 3 hold. Let denote the projection of onto , assumed to exist, and define


It holds that


Sublinear convergence of cyclic coordinate descent Heavy-ball algorithm

We show the convergence rate of cyclic coordinate descent Heavy-ball algorithm for coercive .

Theorem 3

Suppose the conditions of Lemma 3 hold, is coercive, and

Then we have


where is given by (17).

We readily check that , where is number of the block. Therefore, the cyclic inertial algorithm converges with the rate . Compared with the results in (Sun and Hong, 2015), this rate is on the same order as that of cyclic block coordinate descent in general convex setting.

Linear convergence of cyclic coordinate descent Heavy-ball algorithm

Under the same assumption of restricted strong convexity, we derive the linear convergence rate for cyclic coordinate descent Heavy-ball algorithm.

Theorem 4

Suppose the conditions of Lemma 3 hold, satisfies (19), and

Then we have


for some , and .

This result can be extended to the essentially cyclic Heavy-ball algorithm. The essentially cyclic index selection strategy (Sun, Hannah, and Yin, 2017), which generalizes the cyclic update rule, is defined as follows: there is an , , such that each block is updated at least once in a window of .

Stochastic coordinate descent Heavy-ball algorithm

For the stochastic index selection strategy, in the -th iteration, we pick uniformly from and iterate


In this section, we make the following assumption

A2: the parameters is non-increasing.

Assumption A2 is quite different from previous requirement that which are constrained on . This difference comes from the uniformly stochastic selection of the index.

Lemma 5

Let be a convex function whose gradient is Lipschitz continuous with , and finite . Let be generated by scheme (31) and Assumption A2 be satisfied. Choose the step size

for arbitrary fixed . Then, we can obtain


Similarly, we consider the following function




Different from the previous analyses, the Lyapunov function considered here is instead of . Naturally, the sufficient descent property is established in the sense of expectation.

Lemma 6

Suppose the conditions of Lemma 5 hold. Let denote the projection of onto , assumed to exist, and define


Then it holds


Sublinear convergence of stochastic coordinate descent Heavy-ball algorithm

Due to that the sufficient descent condition involves expectations, even using the coercivity of , we cannot obtain the boundedness of the generated points. Therefore, we first present a result by assuming the smoothness of only.

Theorem 5

Suppose that the assumptions of Lemma 5 hold. Then we have


We remark that Theorem (5) also holds for nonconvex functions. To obtain the sublinear convergence rate on the function values, we need a boundedness assumption. Precisely, the assumption is

A3: the sequence satisfies

Under assumption A3, we are able to show the non-ergodic convergence sublinear convergence rates of the expected objective values.

Theorem 6

Suppose that the assumptions of Lemma 5 and A3 hold. Then we have

Figure 1:

Objective values of inertial algorithms v.s. number of epochs for linear regression and logistic regression tasks. Panels (a)-(d): Heavy-ball algorithms comparisons for linear regression with Gaussian data (a); for linear regression with Bernoulli data (b); for logistic regression with Gaussian data (c); for logistic regression with Bernoulli data (d). Panels (e)-(h): Cyclic coordinate descent Heavy-ball algorithms comparisons for linear regression with Gaussian data (e); for linear regression with Bernoulli data (f); for logistic regression with Gaussian data (g); for logistic regression with Bernoulli data (h). Panels (i)-(l): Stochastic coordinate descent Heavy-ball algorithms comparisons for linear regression with Gaussian data (i); for linear regression with Bernoulli data (j); for logistic regression with Gaussian data (k); for logistic regression with Bernoulli data (l).

Linear convergence of stochastic coordinate descent Heavy-ball algorithm

The linear convergence rate of stochastic coordinate descent Heavy-ball algorithm is similar to previous ones. By assuming the restricted strongly convex condition, the linear convergence rate of the expected objective values can be proved.

Theorem 7

Suppose that the assumptions in Lemma 5 hold, and the function satisfies the restricted strongly convex condition (19). Let be generated by the scheme (31). Then we have


where , and .

While we only consider the uniform probability selection strategy here, the same convergence results can be easily extended to the non-uniform probability selection strategy.

Applications to decentralized optimization

We apply the analysis to the following decentralized optimization problem

where is differentiable and is -Lipschitz. Denote by the local copy of at node and . In the community of decentralized algorithms, rather than directly solving the problem, following penalty formulation instead has been proposed


where is the mixing matrix, and , and is the unit matrix. It is easy to see that is Lipschitz with the constant , here

is minimum eigenvalue of

. Researchers consider the decentralized gradient descent (DGD) (Nedic and Ozdaglar, 2009), which is essentially the gradient descent applied to (40) with stepsize being equal to . This algorithm can be implemented over a connected network, in which the agents communicate with their neighbors and make full use of the computing resources of all nodes. Alternatively, we can use the Heavy-ball method by choosing the stepsize , that is,

For node , the local scheme is then

where is the copy of the variable in node in the th iteration and denotes the neighbors of node . In the global scheme, it is basically Heavy-ball algorithm. Thus, we can apply our theoretical findings to this algorithm. To guarantee the convergence, we just need

After simplification, we then get

In a word, we need the requirements

The convergence result for decentralized Heavy-ball method directly follows from our previous theoretical findings and can be summarized as below.

Corollary 2

Assume that is convex and differentiable, and is Lipschitz with . Let , and the sequence be generated by the decentralized Heavy-ball method. For any fixed stepsize , we have


This justifies the superiority of our non-ergodic analysis. As aforementioned in the introduction, all the existing convergence results are about the sequence . However, for decentralized Heavy-ball algorithm, it is meaningless to discuss the ergodic rates, because the nodes only communicate with their neighbors. However, our results, in this case, still hold.

Experimental results

We report the numerical simulations of Heavy-ball method applied to the linear regression problem


and the logistic regression problem


where , . All experiments were performed using MATLAB on an desktop with an Intel 3.4 GHz CPU. We tested the three Heavy-ball algorithms with different inertial parameters. We fixed the stepsize as in all numerical tests. For the stepsize, we need , i.e., . Therefore, inertial parameters are set to . For linear regression problem, , where denotes the largest eigenvalue of a matrix; whereas for logistic regression, we have . With schemes of the algorithms, for cyclic coordinate gradient descent, the function values are recorded after the whole epoch is updated; while for stochastic coordinate gradient descent, functions values are updated after per iteration. The special case corresponds to the gradient descent, or cyclic coordinate gradient descent, or stochastic coordinate gradient descent. And we set and . The data and were generated by the Gaussian random and Bernoulli random distributions, respectively. The maximum number of iterations was set to . For logistic regression, we set . We tested the three Heavy-ball algorithms for both two regression tasks with Gaussian and Bernoulli data.

As illustrated by Figure 1, larger leads to faster convergence for both Heavy-ball algorithm and cyclic coordinate descent algorithm when . However, for the stochastic block coordinate descent scheme, the inertial method helps insignifically. This is because for the stochastic case, in the th iteration, the inertial terms contribute only when . This case, however, happens with probability and is the number of the blocks; as is large, happens at low probability for just one iteration, let alone the whole iterations. Therefore, the inertial method is actually inactive at most iterations for stochastic block coordinate descent scheme.

To improve the practical performance of stochastic block coordinate descent Heavy-ball algorithm, another inertial scheme proposed in Xu and Yin (2013) can be recruited, in which, a new storage is used. In each iteration, the algorithm employs to replace in scheme (31) and then updates with keeping other coordinates of . In this scheme, the inertial term can be active for all iterations. However, the convergence of such algorithm is beyond the proof techniques proposed in this paper, and of course, deserves further study.


In this paper, we studied the non-ergodic computational complexity of the Heavy-ball methods in the convex setting. Under different assumptions, we proved the non-ergodic sublinear and linear convergence rates for the algorithm, respectively. In both cases, we made much more relaxed assumptions than appeared in the existing literatures. Our proof was motivated by the analysis on a novel dynamical system. We extended our results to the multi-block coordinate descent Heavy-ball algorithm for both cyclic and stochastic update rules. The application to decentralized optimization demonstrated the advantage of our analysis techniques.

Acknowledgments: The authors are indebted to anonymous referees for their useful suggestions. We are grateful for the support from the the Major State Research Development Program (2016YFB0201305), and National Key Research and Development Program of China (2017YFB0202003), and National Natural Science Foundation of Hunan (2018JJ3616).


  • Alvarez (2000) Alvarez, F. 2000. On the minimizing property of a second order dissipative system in hilbert spaces. SIAM Journal on Control and Optimization 38(4):1102–1119.
  • Combettes and Glaudin (2017) Combettes, P. L., and Glaudin, L. E. 2017. Quasinonexpansive iterations on the affine hull of orbits: From mann’s mean value algorithm to inertial methods. Siam Journal on Optimization 27(4).
  • Ghadimi, Feyzmahdavian, and Johansson (2015) Ghadimi, E.; Feyzmahdavian, H. R.; and Johansson, M. 2015. Global convergence of the heavy-ball method for convex optimization. In Control Conference (ECC), 2015 European, 310–315. IEEE.
  • Lai and Yin (2013) Lai, M.-J., and Yin, W. 2013. Augmented and nuclear-norm models with a globally linearly convergent algorithm. SIAM Journal on Imaging Sciences 6(2):1059–1091.
  • Lessard, Recht, and Packard (2016) Lessard, L.; Recht, B.; and Packard, A. 2016. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization 26(1):57–95.
  • Liang, Fadili, and Peyré (2016) Liang, J.; Fadili, J.; and Peyré, G. 2016. A multi-step inertial forward-backward splitting method for non-convex optimization. In Advances in Neural Information Processing Systems, 4035–4043.
  • Loizou and Richtárik (2017a) Loizou, N., and Richtárik, P. 2017a. Linearly convergent stochastic heavy ball method for minimizing generalization error. arXiv preprint arXiv:1710.10737.
  • Loizou and Richtárik (2017b) Loizou, N., and Richtárik, P. 2017b. Momentum and stochastic momentum for stochastic gradient, newton, proximal point and subspace descent methods. arXiv preprint arXiv:1712.09677.
  • Nedic and Ozdaglar (2009) Nedic, A., and Ozdaglar, A. 2009. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control 54(1):48–61.
  • Nesterov (2013) Nesterov, Y. 2013. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media.
  • Ochs et al. (2014) Ochs, P.; Chen, Y.; Brox, T.; and Pock, T. 2014. ipiano: Inertial proximal algorithm for nonconvex optimization. SIAM Journal on Imaging Sciences 7(2):1388–1419.
  • Ochs, Brox, and Pock (2015) Ochs, P.; Brox, T.; and Pock, T. 2015. ipiasco: Inertial proximal algorithm for strongly convex optimization. Journal of Mathematical Imaging and Vision 53(2):171–181.
  • Ochs (2016) Ochs, P. 2016. Local convergence of the heavy-ball method and ipiano for non-convex optimization. arXiv preprint arXiv:1606.09070.
  • Pock and Sabach (2016) Pock, T., and Sabach, S. 2016. Inertial proximal alternating linearized minimization (ipalm) for nonconvex and nonsmooth problems. SIAM Journal on Imaging Sciences 9(4):1756–1787.
  • Polyak (1964) Polyak, B. T. 1964. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4(5):1–17.
  • Sun and Hong (2015) Sun, R., and Hong, M. 2015. Improved iteration complexity bounds of cyclic block coordinate descent for convex problems. In Advances in Neural Information Processing Systems, 1306–1314.
  • Sun, Hannah, and Yin (2017) Sun, T.; Hannah, R.; and Yin, W. 2017. Asynchronous coordinate descent under more realistic assumptions. NIPS.
  • Xu and Yin (2013) Xu, Y., and Yin, W. 2013. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM Journal on imaging sciences 6(3):1758–1789.
  • Zavriev and Kostyuk (1993) Zavriev, S., and Kostyuk, F. 1993. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling 4(4):336–341.

Proof of Lemma 1

By the scheme for updating ,


By Lipschitz continuity of ,


Combining (44) and (45), we have


where uses the Cauchy-Schwarz inequality . A simple calculation gives