DeepAI AI Chat
Log In Sign Up

Sample Complexity and Overparameterization Bounds for Projection-Free Neural TD Learning

by   Semih Cayci, et al.

We study the dynamics of temporal-difference learning with neural network-based value function approximation over a general state space, namely, Neural TD learning. Existing analysis of neural TD learning relies on either infinite width-analysis or constraining the network parameters in a (random) compact set; as a result, an extra projection step is required at each iteration. This paper establishes a new convergence analysis of neural TD learning without any projection. We show that the projection-free TD learning equipped with a two-layer ReLU network of any width exceeding poly(ν,1/ϵ) converges to the true value function with error ϵ given poly(ν,1/ϵ) iterations or samples, where ν is an upper bound on the RKHS norm of the value function induced by the neural tangent kernel. Our sample complexity and overparameterization bounds are based on a drift analysis of the network parameters as a stopped random process in the lazy training regime.


page 1

page 2

page 3

page 4


On the Sample Complexity of Reinforcement Learning with a Generative Model

We consider the problem of learning the optimal action-value function in...

PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method

Emphatic temporal difference (ETD) learning (Sutton et al., 2016) is a s...

On the Global Convergence of Fitted Q-Iteration with Two-layer Neural Network Parametrization

Deep Q-learning based algorithms have been applied successfully in many ...

Tighter Sparse Approximation Bounds for ReLU Neural Networks

A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski Ba...

On the Optimization Dynamics of Wide Hypernetworks

Recent results in the theoretical study of deep learning have shown that...

Computing Transience Bounds of Emergency Call Centers: a Hierarchical Timed Petri Net Approach

A fundamental issue in the analysis of emergency call centers is to esti...