Provably-Efficient Double Q-Learning

07/09/2020
by   Wentao Weng, et al.
0

In this paper, we establish a theoretical comparison between the asymptotic mean-squared error of Double Q-learning and Q-learning. Our result builds upon an analysis for linear stochastic approximation based on Lyapunov equations and applies to both tabular setting and with linear function approximation, provided that the optimal policy is unique and the algorithms converge. We show that the asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning if Double Q-learning uses twice the learning rate of Q-learning and outputs the average of its two estimators. We also present some practical implications of this theoretical observation using simulations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/16/2018

Optimal mean squared error bandwidth for spectral variance estimators in MCMC simulations

This paper proposes optimal mean squared error bandwidths for a family o...
research
09/29/2020

Finite-Time Analysis for Double Q-learning

Although Q-learning is one of the most successful algorithms for finding...
research
03/29/2021

A bandit-learning approach to multifidelity approximation

Multifidelity approximation is an important technique in scientific comp...
research
03/25/2021

Smoothing methods to estimate the hazard rate under double truncation

In Survival Analysis, the observed lifetimes often correspond to individ...
research
08/04/2021

The Theory of Perfect Learning

The perfect learning exists. We mean a learning model that can be genera...
research
09/19/2012

Comunication-Efficient Algorithms for Statistical Optimization

We analyze two communication-efficient algorithms for distributed statis...
research
04/20/2022

Exact Formulas for Finite-Time Estimation Errors of Decentralized Temporal Difference Learning with Linear Function Approximation

In this paper, we consider the policy evaluation problem in multi-agent ...

Please sign up or login with your details

Forgot password? Click here to reset