
Zap Meets Momentum: Stochastic Approximation Algorithms with Optimal Convergence Rate
There are two well known Stochastic Approximation techniques that are kn...
read it

Can speed up the convergence rate of stochastic gradient methods to O(1/k^2) by a gradient averaging strategy?
In this paper we consider the question of whether it is possible to appl...
read it

ROOTSGD: Sharp Nonasymptotics and Asymptotic Efficiency in a Single Algorithm
The theory and practice of stochastic optimization has focused on stocha...
read it

The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learning
The paper concerns convergence and asymptotic statistics for stochastic ...
read it

A Law of Iterated Logarithm for MultiAgent Reinforcement Learning
In MultiAgent Reinforcement Learning (MARL), multiple agents interact w...
read it

Geometric Bounds for Convergence Rates of Averaging Algorithms
We develop a generic method for bounding the convergence rate of an aver...
read it

Deviation inequalities for stochastic approximation by averaging
We introduce a class of Markov chains, that contains the model of stocha...
read it
Accelerating Optimization and Reinforcement Learning with QuasiStochastic Approximation
The ODE method has been a workhorse for algorithm design and analysis since the introduction of the stochastic approximation. It is now understood that convergence theory amounts to establishing robustness of Euler approximations for ODEs, while theory of rates of convergence requires finer analysis. This paper sets out to extend this theory to quasistochastic approximation, based on algorithms in which the "noise" is based on deterministic signals. The main results are obtained under minimal assumptions: the usual Lipschitz conditions for ODE vector fields, and it is assumed that there is a well defined linearization near the optimal parameter θ^*, with Hurwitz linearization matrix A^*. The main contributions are summarized as follows: (i) If the algorithm gain is a_t=g/(1+t)^ρ with g>0 and ρ∈(0,1), then the rate of convergence of the algorithm is 1/t^ρ. There is also a well defined "finitet" approximation: a_t^1{Θ_tθ^*}=Y̅+Ξ^I_t+o(1) where Y̅∈ℝ^d is a vector identified in the paper, and {Ξ^I_t} is bounded with zero temporal mean. (ii) With gain a_t = g/(1+t) the results are not as sharp: the rate of convergence 1/t holds only if I + g A^* is Hurwitz. (iii) Based on the RuppertPolyak averaging of stochastic approximation, one would expect that a convergence rate of 1/t can be obtained by averaging: Θ^RP_T=1/T∫_0^T Θ_t dt where the estimates {Θ_t} are obtained using the gain in (i). The preceding sharp bounds imply that averaging results in 1/t convergence rate if and only if Y̅= 0. This condition holds if the noise is additive, but appears to fail in general. (iv) The theory is illustrated with applications to gradientfree optimization and policy gradient algorithms for reinforcement learning.
READ FULL TEXT
Comments
There are no comments yet.