Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noise

03/28/2023
by   Zaiwei Chen, et al.
0

In this work, we study the concentration behavior of a stochastic approximation (SA) algorithm under a contractive operator with respect to an arbitrary norm. We consider two settings where the iterates are potentially unbounded: (1) bounded multiplicative noise, and (2) additive sub-Gaussian noise. We obtain maximal concentration inequalities on the convergence errors, and show that these errors have sub-Gaussian tails in the additive noise setting, and super-polynomial tails (faster than polynomial decay) in the multiplicative noise setting. In addition, we provide an impossibility result showing that it is in general not possible to achieve sub-exponential tails for SA with multiplicative noise. To establish these results, we develop a novel bootstrapping argument that involves bounding the moment generating function of the generalized Moreau envelope of the error and the construction of an exponential supermartingale to enable using Ville's maximal inequality. To demonstrate the applicability of our theoretical results, we use them to provide maximal concentration bounds for a large class of reinforcement learning algorithms, including but not limited to on-policy TD-learning with linear function approximation, off-policy TD-learning with generalized importance sampling factors, and Q-learning. To the best of our knowledge, super-polynomial concentration bounds for off-policy TD-learning have not been established in the literature due to the challenge of handling the combination of unbounded iterates and multiplicative noise.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/27/2021

Concentration of Contractive Stochastic Approximation and Reinforcement Learning

Using a martingale concentration inequality, concentration bounds `from ...
research
02/09/2021

The Multiplicative Version of Azuma's Inequality, with an Application to Contention Analysis

Azuma's inequality is a tool for proving concentration bounds on random ...
research
05/25/2022

Transportation-Inequalities, Lyapunov Stability and Sampling for Dynamical Systems on Continuous State Space

We study the concentration phenomenon for discrete-time random dynamical...
research
10/31/2019

Multiplicative noise in Bayesian inverse problems: Well-posedness and consistency of MAP estimators

Multiplicative noise arises in inverse problems when, for example, uncer...
research
12/15/2020

An exact solution in Markov decision process with multiplicative rewards as a general framework

We develop an exactly solvable framework of Markov decision process with...
research
06/07/2022

Concentration analysis of multivariate elliptic diffusion processes

We prove concentration inequalities and associated PAC bounds for contin...
research
07/11/2019

Computational Concentration of Measure: Optimal Bounds, Reductions, and More

Product measures of dimension n are known to be concentrated in Hamming ...

Please sign up or login with your details

Forgot password? Click here to reset