DeepAI AI Chat
Log In Sign Up

Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning

by   Yingjie Fei, et al.
University of Wisconsin-Madison
Princeton University
cornell university

We study risk-sensitive reinforcement learning (RL) based on the entropic risk measure. Although existing works have established non-asymptotic regret guarantees for this problem, they leave open an exponential gap between the upper and lower bounds. We identify the deficiencies in existing algorithms and their analysis that result in such a gap. To remedy these deficiencies, we investigate a simple transformation of the risk-sensitive Bellman equations, which we call the exponential Bellman equation. The exponential Bellman equation inspires us to develop a novel analysis of Bellman backup procedures in risk-sensitive RL algorithms, and further motivates the design of a novel exploration mechanism. We show that these analytic and algorithmic innovations together lead to improved regret upper bounds over existing ones.


page 1

page 2

page 3

page 4


Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learning

In this paper, we study gap-dependent regret guarantees for risk-sensiti...

Risk-Sensitive Reinforcement Learning: Iterated CVaR and the Worst Path

In this paper, we study a novel episodic risk-sensitive Reinforcement Le...

Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret

We study risk-sensitive reinforcement learning in episodic Markov decisi...

Regret Bounds for Risk-Sensitive Reinforcement Learning

In safety-critical applications of reinforcement learning such as health...

Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures

We study finite episodic Markov decision processes incorporating dynamic...

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds

We study the regret guarantee for risk-sensitive reinforcement learning ...

Quantum Computing Provides Exponential Regret Improvement in Episodic Reinforcement Learning

In this paper, we investigate the problem of episodic reinforcement lear...