A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithms

12/04/2019
by   Donghwan Lee, et al.
0

In this paper, we introduce a unified framework for analyzing a large family of Q-learning algorithms, based on switching system perspectives and ODE-based stochastic approximation. We show that the nonlinear ODE models associated with these Q-learning algorithms can be formulated as switched linear systems, and analyze their asymptotic stability by leveraging existing switching system theories. Our approach provides the first O.D.E. analysis of the asymptotic convergences of various Q-learning algorithms, including asynchronous Q-learning, averaging Q-learning, double Q-learning with or without regularization. We also extend the approach to analyze Q-learning with linear function approximation and derive a new sufficient condition for its convergence.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/24/2019

Target-Based Temporal Difference Learning

The use of target networks has been a popular and key component of recen...
research
07/25/2022

Finite-Time Analysis of Asynchronous Q-learning under Diminishing Step-Size from Control-Theoretic View

Q-learning has long been one of the most popular reinforcement learning ...
research
09/29/2020

Finite-Time Analysis for Double Q-learning

Although Q-learning is one of the most successful algorithms for finding...
research
11/25/2021

A Letter on Convergence of In-Parameter-Linear Nonlinear Neural Architectures with Gradient Learnings

This letter summarizes and proves the concept of bounded-input bounded-s...
research
11/15/2020

Functorial Manifold Learning and Overlapping Clustering

We adapt previous research on topological unsupervised learning to devel...
research
02/16/2021

From Majorization to Interpolation: Distributionally Robust Learning using Kernel Smoothing

We study the function approximation aspect of distributionally robust op...
research
06/08/2022

Learning in games from a stochastic approximation viewpoint

We develop a unified stochastic approximation framework for analyzing th...

Please sign up or login with your details

Forgot password? Click here to reset