Finite-Time Analysis of Stochastic Gradient Descent under Markov Randomness

03/24/2020
by   Thinh T. Doan, et al.
ibm
University of North Carolina at Chapel Hill
Georgia Institute of Technology
0

Motivated by broad applications in reinforcement learning and machine learning, this paper considers the popular stochastic gradient descent (SGD) when the gradients of the underlying objective function are sampled from Markov processes. This Markov sampling leads to the gradient samples being biased and not independent. The existing results for the convergence of SGD under Markov randomness are often established under the assumptions on the boundedness of either the iterates or the gradient samples. Our main focus is to study the finite-time convergence of SGD for different types of objective functions, without requiring these assumptions. We show that SGD converges nearly at the same rate with Markovian gradient samples as with independent gradient samples. The only difference is a logarithmic factor that accounts for the mixing time of the Markov chain.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/28/2023

Stochastic Gradient Descent under Markovian Sampling Schemes

We study a variation of vanilla stochastic gradient descent where the op...
10/31/2019

Mixing of Stochastic Accelerated Gradient Descent

We study the mixing properties for stochastic accelerated gradient desce...
09/23/2019

Decentralized Markov Chain Gradient Descent

Decentralized stochastic gradient method emerges as a promising solution...
06/16/2020

Least Squares Regression with Markovian Data: Fundamental Limits and Algorithms

We study the problem of least squares linear regression where the data-p...
11/05/2019

A Rule for Gradient Estimator Selection, with an Application to Variational Inference

Stochastic gradient descent (SGD) is the workhorse of modern machine lea...
06/11/2020

Borrowing From the Future: Addressing Double Sampling in Model-free Control

In model-free reinforcement learning, the temporal difference method and...

Please sign up or login with your details

Forgot password? Click here to reset