DeepAI
Log In Sign Up

Adaptive Discretization for Episodic Reinforcement Learning in Metric Spaces

10/17/2019
by   Sean R. Sinclair, et al.
0

We present an efficient algorithm for model-free episodic reinforcement learning on large (potentially continuous) state-action spaces. Our algorithm is based on a novel Q-learning policy with adaptive data-driven discretization. The central idea is to maintain a finer partition of the state-action space in regions which are frequently visited in historical trajectories, and have higher payoff estimates. We demonstrate how our adaptive partitions take advantage of the shape of the optimal Q-function and the joint space, without sacrificing the worst-case performance. In particular, we recover the regret guarantees of prior algorithms for continuous state-action spaces, which however require either an optimal discretization as input, and/or access to a simulation oracle. Moreover, experiments demonstrate how our algorithm automatically adapts to the underlying structure of the problem, resulting in much better performance compared both to heuristics, as well as Q-learning with uniform discretization.

READ FULL TEXT
07/01/2020

Adaptive Discretization for Model-Based Reinforcement Learning

We introduce the technique of adaptive discretization to design efficien...
03/09/2020

Zooming for Efficient Model-Free Reinforcement Learning in Metric Spaces

Despite the wealth of research into provably efficient reinforcement lea...
06/22/2020

Adaptive Discretization for Adversarial Bandits with Continuous Action Spaces

Lipschitz bandits is a prominent version of multi-armed bandits that stu...
03/16/2020

Particle-Based Adaptive Discretization for Continuous Control using Deep Reinforcement Learning

Learning controls in high-dimensional continuous action spaces, such as ...
10/29/2021

Adaptive Discretization in Online Reinforcement Learning

Discretization based approaches to solving online reinforcement learning...
07/16/2018

Discrete linear-complexity reinforcement learning in continuous action spaces for Q-learning algorithms

In this article, we sketch an algorithm that extends the Q-learning algo...
03/07/2006

Metric State Space Reinforcement Learning for a Vision-Capable Mobile Robot

We address the problem of autonomously learning controllers for vision-c...