Statistics and Samples in Distributional Reinforcement Learning

02/21/2019
by   Mark Rowland, et al.
64

We present a unifying framework for designing and analysing distributional reinforcement learning (DRL) algorithms in terms of recursively estimating statistics of the return distribution. Our key insight is that DRL algorithms can be decomposed as the combination of some statistical estimator and a method for imputing a return distribution consistent with that set of statistics. With this new understanding, we are able to provide improved analyses of existing DRL algorithms as well as construct a new algorithm (EDRL) based upon estimation of the expectiles of the return distribution. We compare EDRL with existing methods on a variety of MDPs to illustrate concrete aspects of our analysis, and develop a deep RL variant of the algorithm, ER-DQN, which we evaluate on the Atari-57 suite of games.

READ FULL TEXT

page 7

page 8

page 22

page 24

page 25

research
02/01/2022

Distributional Reinforcement Learning via Sinkhorn Iterations

Distributional reinforcement learning (RL) is a class of state-of-the-ar...
research
07/24/2020

Distributional Reinforcement Learning with Maximum Mean Discrepancy

Distributional reinforcement learning (RL) has achieved state-of-the-art...
research
03/23/2023

Policy Evaluation in Distributional LQR

Distributional reinforcement learning (DRL) enhances the understanding o...
research
07/21/2017

A Distributional Perspective on Reinforcement Learning

In this paper we argue for the fundamental importance of the value distr...
research
04/19/2023

Robust Route Planning with Distributional Reinforcement Learning in a Stochastic Road Network Environment

Route planning is essential to mobile robot navigation problems. In rece...
research
06/12/2023

Diverse Projection Ensembles for Distributional Reinforcement Learning

In contrast to classical reinforcement learning, distributional reinforc...
research
09/21/2014

Distributed Robust Learning

We propose a framework for distributed robust statistical learning on b...

Please sign up or login with your details

Forgot password? Click here to reset