Regularized Q-learning

02/11/2022
by   Han-Dong Lim, et al.
0

Q-learning is widely used algorithm in reinforcement learning community. Under the lookup table setting, its convergence is well established. However, its behavior is known to be unstable with the linear function approximation case. This paper develops a new Q-learning algorithm that converges when linear function approximation is used. We prove that simply adding an appropriate regularization term ensures convergence of the algorithm. We prove its stability using a recent analysis tool based on switching system models. Moreover, we experimentally show that it converges in environments where Q-learning with linear function approximation has known to diverge. We also provide an error bound on the solution where the algorithm converges.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/08/2021

Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Natural policy gradient (NPG) methods with function approximation achiev...
research
06/30/2023

TD Convergence: An Optimization Perspective

We study the convergence behavior of the celebrated temporal-difference ...
research
02/14/2022

On the Chattering of SARSA with Linear Function Approximation

SARSA, a classical on-policy control algorithm for reinforcement learnin...
research
02/20/2020

Adaptive Temporal Difference Learning with Linear Function Approximation

This paper revisits the celebrated temporal difference (TD) learning alg...
research
06/13/2012

Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping

We consider the problem of efficiently learning optimal control policies...
research
01/21/2021

Breaking the Deadly Triad with a Target Network

The deadly triad refers to the instability of a reinforcement learning a...
research
07/05/2023

Stability of Q-Learning Through Design and Optimism

Q-learning has become an important part of the reinforcement learning to...

Please sign up or login with your details

Forgot password? Click here to reset