Distributional Multivariate Policy Evaluation and Exploration with the Bellman GAN

08/06/2018
by   Dror Freirich, et al.
0

The recently proposed distributional approach to reinforcement learning (DiRL) is centered on learning the distribution of the reward-to-go, often referred to as the value distribution. In this work, we show that the distributional Bellman equation, which drives DiRL methods, is equivalent to a generative adversarial network (GAN) model. In this formulation, DiRL can be seen as learning a deep generative model of the value distribution, driven by the discrepancy between the distribution of the current value, and the distribution of the sum of current reward and next value. We use this insight to propose a GAN-based approach to DiRL, which leverages the strengths of GANs in learning distributions of high-dimensional data. In particular, we show that our GAN approach can be used for DiRL with multivariate rewards, an important setting which cannot be tackled with prior methods. The multivariate setting also allows us to unify learning the distribution of values and state transitions, and we exploit this idea to devise a novel exploration method that is driven by the discrepancy in estimating both values and states.

READ FULL TEXT
research
06/28/2022

GAN-based Intrinsic Exploration For Sample Efficient Reinforcement Learning

In this study, we address the problem of efficient exploration in reinfo...
research
05/13/2018

GAN Q-learning

Distributional reinforcement learning (distributional RL) has seen empir...
research
10/26/2021

Distributional Reinforcement Learning for Multi-Dimensional Reward Functions

A growing trend for value-based reinforcement learning (RL) algorithms i...
research
05/10/2019

GAN-based Deep Distributional Reinforcement Learning for Resource Management in Network Slicing

Network slicing is a key technology in 5G communications system, which a...
research
01/31/2022

On solutions of the distributional Bellman equation

In distributional reinforcement learning not only expected returns but t...
research
02/28/2020

Distributionally Robust Chance Constrained Programming with Generative Adversarial Networks (GANs)

This paper presents a novel deep learning based data-driven optimization...
research
02/20/2021

Strata-based Quantification of Distributional Uncertainty in Socio-Economic Indicators: A Comparative Study of Indian States

This paper reports a comprehensive study of distributional uncertainty i...

Please sign up or login with your details

Forgot password? Click here to reset