Low-rank State-action Value-function Approximation

04/18/2021
by   Sergio Rozada, et al.
0

Value functions are central to Dynamic Programming and Reinforcement Learning but their exact estimation suffers from the curse of dimensionality, challenging the development of practical value-function (VF) estimation algorithms. Several approaches have been proposed to overcome this issue, from non-parametric schemes that aggregate states or actions to parametric approximations of state and action VFs via, e.g., linear estimators or deep neural networks. Relevantly, several high-dimensional state problems can be well-approximated by an intrinsic low-rank structure. Motivated by this and leveraging results from low-rank optimization, this paper proposes different stochastic algorithms to estimate a low-rank factorization of the Q(s, a) matrix. This is a non-parametric alternative to VF approximation that dramatically reduces the computational and sample complexities relative to classical Q-learning methods that estimate Q(s,a) separately for each state-action pair.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/21/2022

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

Value-function (VF) approximation is a central problem in Reinforcement ...
research
11/19/2021

Uncertainty-aware Low-Rank Q-Matrix Estimation for Deep Reinforcement Learning

Value estimation is one key problem in Reinforcement Learning. Albeit ma...
research
06/11/2020

Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimation

We consider the question of learning Q-function in a sample efficient ma...
research
09/26/2019

Harnessing Structures for Value-Based Planning and Reinforcement Learning

Value-based methods constitute a fundamental methodology in planning and...
research
11/28/2018

Toward breaking the curse of dimensionality: an FPTAS for stochastic dynamic programs with multidimensional actions and scalar states

We propose a Fully Polynomial-Time Approximation Scheme (FPTAS) for stoc...
research
12/04/2015

Q-Networks for Binary Vector Actions

In this paper reinforcement learning with binary vector actions was inve...
research
11/16/2022

Global Optimization with Parametric Function Approximation

We consider the problem of global optimization with noisy zeroth order o...

Please sign up or login with your details

Forgot password? Click here to reset