On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration

04/09/2020
by   Wenlong Mou, et al.
3

We undertake a precise study of the asymptotic and non-asymptotic properties of stochastic approximation procedures with Polyak-Ruppert averaging for solving a linear system A̅θ = b̅. When the matrix A̅ is Hurwitz, we prove a central limit theorem (CLT) for the averaged iterates with fixed step size and number of iterations going to infinity. The CLT characterizes the exact asymptotic covariance matrix, which is the sum of the classical Polyak-Ruppert covariance and a correction term that scales with the step size. Under assumptions on the tail of the noise distribution, we prove a non-asymptotic concentration inequality whose main term matches the covariance in CLT in any direction, up to universal constants. When the matrix A̅ is not Hurwitz but only has non-negative real parts in its eigenvalues, we prove that the averaged LSA procedure actually achieves an O(1/T) rate in mean-squared error. Our results provide a more refined understanding of linear stochastic approximation in both the asymptotic and non-asymptotic settings. We also show various applications of the main results, including the study of momentum-based stochastic gradient methods as well as temporal difference algorithms in reinforcement learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/15/2021

On Riemannian Stochastic Approximation Schemes with Fixed Step-Size

This paper studies fixed step-size stochastic approximation (SA) schemes...
research
09/06/2023

The Curse of Memory in Stochastic Approximation: Extended Version

Theory and application of stochastic approximation (SA) has grown within...
research
09/28/2020

Non asymptotic controls on a recursive superquantile approximation

In this work, we study a new recursive stochastic algorithm for the join...
research
10/29/2020

A non-asymptotic version of Cressie's refined continuity correction for the binomial distribution

In this paper, we prove a non-asymptotic version of the refined continui...
research
06/26/2019

An urn model with local reinforcement: a theoretical framework for a chi-squared goodness of fit test with a big sample

Motivated by recent studies of big samples, this work aims at constructi...
research
01/16/2023

Asymptotic normality and optimality in nonsmooth stochastic approximation

In their seminal work, Polyak and Juditsky showed that stochastic approx...
research
07/10/2022

Finite-time High-probability Bounds for Polyak-Ruppert Averaged Iterates of Linear Stochastic Approximation

This paper provides a finite-time analysis of linear stochastic approxim...

Please sign up or login with your details

Forgot password? Click here to reset