Hyperbolic Discounting and Learning over Multiple Horizons

by   William Fedus, et al.
Montréal Institute of Learning Algorithms

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.


Multi-Agent Learning of Numerical Methods for Hyperbolic PDEs with Factored Dec-MDP

Factored decentralized Markov decision process (Dec-MDP) is a framework ...

Hyperbolic Deep Reinforcement Learning

We propose a new class of deep reinforcement learning (RL) algorithms th...

Reinforcement Learning with Non-Exponential Discounting

Commonly in reinforcement learning (RL), rewards are discounted over tim...

Analysis of Temporal Difference Learning: Linear System Approach

The goal of this technical note is to introduce a new finite-time conver...

Experimental Analysis of Reinforcement Learning Techniques for Spectrum Sharing Radar

In this work, we first describe a framework for the application of Reinf...

Taylor Expansion of Discount Factors

In practical reinforcement learning (RL), the discount factor used for e...

Bridging the gap between QP-based and MPC-based RL

Reinforcement learning methods typically use Deep Neural Networks to app...

Please sign up or login with your details

Forgot password? Click here to reset