A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)

10/25/2017 ∙ by Prateek Jain, et al. ∙ Microsoft University of Washington Stanford University 0

This work provides a simplified proof of the statistical minimax optimality of (iterate averaged) stochastic gradient descent (SGD), for the special case of least squares. This result is obtained by analyzing SGD as a stochastic process and by sharply characterizing the stationary covariance matrix of this process. The finite rate optimality characterization captures the constant factors and addresses model mis-specification.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Stochastic gradient descent is among the most commonly used practical algorithms for large scale stochastic optimization. The seminal result of  [9, 8] formalized this effectiveness, showing that for certain (locally quadric) problems, asymptotically, stochastic gradient descent is statistically minimax optimal (provided the iterates are averaged). There are a number of more modern proofs [1, 3, 2, 5] of this fact, which provide finite rates of convergence. Other recent algorithms also achieve the statistically optimal minimax rate, with finite convergence rates [4].

This work provides a short proof of this minimax optimality for SGD for the special case of least squares through a characterization of SGD as a stochastic process. The proof builds on ideas developed in [2, 5].

SGD for least squares. The expected square loss for over input-output pairs , where and are sampled from a distribution , is:

The optimal weight is denoted by:

Assume the argmin in unique.

Stochastic gradient descent proceeds as follows: at each iteration , using an i.i.d. sample , the update of is:

where is a fixed stepsize.

Notation. For a symmetric positive definite matrix

and a vector

, define:

For a symmetric matrix , define the induced matrix norm under as:

The statistically optimal rate. Using samples (and for large enough

), the minimax optimal rate is achieved by the maximum likelihood estimator (the MLE), or, equivalently, the empirical risk minimizer. Given

i.i.d. samples , define

where denotes the MLE estimator over the samples.

This rate can be characterized as follows: define

and the (asymptotic) rate of the MLE is  [7, 10]. Precisely,

The works of [9, 8] proved that a certain averaged stochastic gradient method achieves this minimax rate, in the limit.

For the case of additive noise models (i.e. the “well-specified” case), the assumption is that , with being independent of . Here, it is straightforward to see that:

The rate of is still minimax optimal even among mis-specified models, where the additive noise assumption may not hold [6, 7, 10].


Assume the fourth moment of

is finite. Denote the second moment matrix of as

and suppose

is strictly positive definite with minimal eigenvalue:

Define as the smallest value which satisfies:

This implies .

2 Statistical Risk Bounds


and so the optimal constant in the rate can be written as:

For the mis-specified case, it is helpful to define:

which can be viewed as a measure of how mis-specified the model is. Note if the model is well-specified, then .

Denote the average iterate, averaged from iteration to , by:

Theorem 1.

Suppose . The risk is bounded as:

The bias term (the first term) decays at a geometric rate (one can set or maintain multiple running averages if is not known in advance). If and the model is well-specified (

), then the variance term is

, and the rate of the bias contraction is . If the model is not well specified, then using a smaller stepsize of , leads to the same minimax optimal rate (up to a constant factor of 2), albeit at a slower bias contraction rate. In the mis-specified case, an example in [5] shows that such a smaller stepsize is required in order to be within a constant factor of the minimax rate. An even smaller stepsize leads to a constant even closer to that of the optimal rate.

3 Analysis

The analysis first characterizes a bias/variance decomposition, where the variance is bounded in terms of properties of the stationary covariance of . Then this asymptotic covariance matrix is analyzed.

Throughout assume:

3.1 The Bias-Variance Decomposition

The gradient at in iteration is:

which is a mean quantity. Also define:

The update rule can be written as:

Roughly speaking, the above shows how the process on consists of a contraction along with an addition of a zero mean quantity.

From recursion,


It is helpful to consider a certain bias and variance decomposition. Let us write:


(The first conditional expectation notation slightly abuses notation, and should be taken as a definition111The abuse is due that the right hand side drops the conditioning.).

Lemma 1.

The error is bounded as:


Equation 1 implies that:

Now observe that for vector valued random variables

, implies

the proof of the lemma follows by noting that . ∎

Bias. The bias term is characterized as follows:

Lemma 2.

For all ,


Assume for all . Observe:

which completes the proof. ∎

Variance. Now suppose . Define the covariance matrix:

Using the recursion, ,


which follows from:

(these hold since is mean and both and are independent of ).

Lemma 3.

Suppose . There exists a unique such that:

where satisfies:


By recursion,

Using that is mean zero and independent of and for ,

Now using that and that and are independent (for ),

which proves .

To prove the limit exists, it suffices to first argue the trace of is uniformly bounded from above, for all . By taking the trace of update rule, Equation 2, for ,



and, using ,

proving the uniform boundedness of the trace of . Now, for any fixed , the limit of exists, by the monotone convergence theorem. From this, it follows that every entry of the matrix converges. ∎

Lemma 4.


and so:



double counting the diagonal terms . For , . To see why, consider the recursion and take expectations to get since the sample is independent of the . From this,

and so,

Notice that for any non-negative integer . Since and , because the product of two commuting PSD matrices is PSD. Also note that for PSD matrices , . Hence,

from lemma 3 where followed from

and the series converges because . ∎

3.2 Stationary Distribution Analysis

Define two linear operators on symmetric matrices, and — where and can be viewed as matrices acting on dimensions — as follows:

With this, is the solution to:


(due to Equation 3).

Lemma 5.

(Crude bound) is bounded as:


Define one more linear operator as follows:

The inverse of this operator can be written as:

which exists since the sum converges due to fact that .

A few inequalities are helpful: If , then



(which follows since ). Also, if , then


which implies:


The following inequality is also of use:

By definition of ,

Using this and Equation 6,

Proceeding recursively by using Equation 8,



leads to

which completes the proof. ∎

Lemma 6.

(Refined bound) The is bounded as:


From Lemma 5 and Equation 7,

Also, from Equation 3, satisfies:

Multiplying this by and taking the trace leads to:

which completes the proof. ∎

3.3 Completing the proof of Theorem 1


The proof of the theorem is completed by applying the developed lemmas. For the bias term, using convexity leads to:

For the variance term, observe that

which completes the proof. ∎


Sham Kakade acknowledges funding from the Washington Research Foundation Fund for Innovation in Data-Intensive Discovery. The authors also thank Zaid Harchaoui for helpful discussions.


  • [1] Francis R. Bach.

    Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression.

    Journal of Machine Learning Research (JMLR)

    , volume 15, 2014.
  • [2] Alexandre Défossez and Francis R. Bach. Averaged least-mean-squares: Bias-variance trade-offs and optimal sampling distributions. In AISTATS, volume 38, 2015.
  • [3] Aymeric Dieuleveut and Francis R. Bach. Non-parametric stochastic approximation with large step sizes. The Annals of Statistics, 2015.
  • [4] Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford. Competing with the empirical risk minimizer in a single pass. In COLT, 2015.
  • [5] Prateek Jain, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic approximation through mini-batching and tail-averaging. CoRR, abs/1610.03774, 2016.
  • [6] Harold J. Kushner and Dean S. Clark. Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer-Verlag, 1978.
  • [7] Erich L. Lehmann and George Casella. Theory of Point Estimation. Springer Texts in Statistics. Springer, 1998.
  • [8] Boris T. Polyak and Anatoli B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, volume 30, 1992.
  • [9] David Ruppert. Efficient estimations from a slowly convergent robbins-monro process. Tech. Report, ORIE, Cornell University, 1988.
  • [10] Aad W. van der Vaart. Asymptotic Statistics. Cambridge University Publishers, 2000.