Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo

05/29/2023
by   Haque Ishfaq, et al.
3

We present a scalable and effective exploration strategy based on Thompson sampling for reinforcement learning (RL). One of the key shortcomings of existing Thompson sampling algorithms is the need to perform a Gaussian approximation of the posterior distribution, which is not a good surrogate in most practical settings. We instead directly sample the Q function from its posterior distribution, by using Langevin Monte Carlo, an efficient type of Markov Chain Monte Carlo (MCMC) method. Our method only needs to perform noisy gradient descent updates to learn the exact posterior distribution of the Q function, which makes our approach easy to deploy in deep RL. We provide a rigorous theoretical analysis for the proposed method and demonstrate that, in the linear Markov decision process (linear MDP) setting, it has a regret bound of Õ(d^3/2H^5/2√(T)), where d is the dimension of the feature mapping, H is the planning horizon, and T is the total number of steps. We apply this approach to deep RL, by using Adam optimizer to perform gradient updates. Our approach achieves better or similar results compared with state-of-the-art deep RL algorithms on several challenging exploration tasks from the Atari57 suite.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/22/2022

Langevin Monte Carlo for Contextual Bandits

We study the efficiency of Thompson sampling for contextual bandits. Exi...
research
11/11/2018

Langevin-gradient parallel tempering for Bayesian neural learning

Bayesian neural learning feature a rigorous approach to estimation and u...
research
12/27/2022

Model-Based Reinforcement Learning with Multinomial Logistic Function Approximation

We study model-based reinforcement learning (RL) for episodic Markov dec...
research
05/15/2018

The Hierarchical Adaptive Forgetting Variational Filter

A common problem in Machine Learning and statistics consists in detectin...
research
02/11/2022

A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search

Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex...
research
02/27/2023

Taylor TD-learning

Many reinforcement learning approaches rely on temporal-difference (TD) ...
research
11/26/2012

Bayesian learning of noisy Markov decision processes

We consider the inverse reinforcement learning problem, that is, the pro...

Please sign up or login with your details

Forgot password? Click here to reset