Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning

02/03/2019
by   R. Srikant, et al.
0

We consider the dynamics of a linear stochastic approximation algorithm driven by Markovian noise, and derive finite-time bounds on the moments of the error, i.e., deviation of the output of the algorithm from the equilibrium point of an associated ordinary differential equation (ODE). To obtain finite-time bounds on the mean-square error in the case of constant step-size algorithms, our analysis uses Stein's method to identify a Lyapunov function that can potentially yield good steady-state bounds, and uses this Lyapunov function to obtain finite-time bounds by mimicking the corresponding steps in the analysis of the associated ODE. We also provide a comprehensive treatment of the moments of the square of the 2-norm of the approximation error. Our analysis yields the following results: (i) for a given step-size, we show that the lower-order moments can be made small as a function of the step-size and can be upper-bounded by the moments of a Gaussian random variable; (ii) we show that the higher-order moments beyond a threshold may be infinite in steady-state; and (iii) we characterize the number of samples needed for the finite-time bounds to be of the same order as the steady-state bounds. As a by-product of our analysis, we also solve the open problem of obtaining finite-time bounds for the performance of temporal difference learning algorithms with linear function approximation and a constant step-size, without requiring a projection step or an i.i.d. noise assumption.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/24/2021

Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

This paper considers a novel multi-agent linear stochastic approximation...
research
05/27/2019

Finite-Time Analysis of Q-Learning with Linear Function Approximation

In this paper, we consider the model-free reinforcement learning problem...
research
11/10/2022

Error bound analysis of the stochastic parareal algorithm

Stochastic parareal (SParareal) is a probabilistic variant of the popula...
research
09/13/2021

On the Correlation between the Noise and a Priori Error Vectors in Affine Projection Algorithms

This paper analyzes the correlation matrix between the a priori error an...
research
02/04/2020

Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noise

Linear two-timescale stochastic approximation (SA) scheme is an importan...
research
03/21/2018

Stochastic Learning under Random Reshuffling

In empirical risk optimization, it has been observed that stochastic gra...
research
04/04/2020

Tracking Performance of Online Stochastic Learners

The utilization of online stochastic algorithms is popular in large-scal...

Please sign up or login with your details

Forgot password? Click here to reset