Log In Sign Up

Adaptive Discretization for Episodic Reinforcement Learning in Metric Spaces

by   Sean R. Sinclair, et al.

We present an efficient algorithm for model-free episodic reinforcement learning on large (potentially continuous) state-action spaces. Our algorithm is based on a novel Q-learning policy with adaptive data-driven discretization. The central idea is to maintain a finer partition of the state-action space in regions which are frequently visited in historical trajectories, and have higher payoff estimates. We demonstrate how our adaptive partitions take advantage of the shape of the optimal Q-function and the joint space, without sacrificing the worst-case performance. In particular, we recover the regret guarantees of prior algorithms for continuous state-action spaces, which however require either an optimal discretization as input, and/or access to a simulation oracle. Moreover, experiments demonstrate how our algorithm automatically adapts to the underlying structure of the problem, resulting in much better performance compared both to heuristics, as well as Q-learning with uniform discretization.


Adaptive Discretization for Model-Based Reinforcement Learning

We introduce the technique of adaptive discretization to design efficien...

Zooming for Efficient Model-Free Reinforcement Learning in Metric Spaces

Despite the wealth of research into provably efficient reinforcement lea...

Adaptive Discretization for Adversarial Bandits with Continuous Action Spaces

Lipschitz bandits is a prominent version of multi-armed bandits that stu...

Particle-Based Adaptive Discretization for Continuous Control using Deep Reinforcement Learning

Learning controls in high-dimensional continuous action spaces, such as ...

Adaptive Discretization in Online Reinforcement Learning

Discretization based approaches to solving online reinforcement learning...

Discrete linear-complexity reinforcement learning in continuous action spaces for Q-learning algorithms

In this article, we sketch an algorithm that extends the Q-learning algo...

Metric State Space Reinforcement Learning for a Vision-Capable Mobile Robot

We address the problem of autonomously learning controllers for vision-c...