-
Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods
This paper investigates estimating the variance of a temporal-difference...
read it
-
Efficient exploration with Double Uncertain Value Networks
This paper studies directed exploration for reinforcement learning agent...
read it
-
The Uncertainty Bellman Equation and Exploration
We consider the exploration/exploitation problem in reinforcement learni...
read it
-
Parameterized Indexed Value Function for Efficient Exploration in Reinforcement Learning
It is well known that quantifying uncertainty in the action-value estima...
read it
-
Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
Value functions are crucial for model-free Reinforcement Learning (RL) t...
read it
-
Separating value functions across time-scales
In many finite horizon episodic reinforcement learning (RL) settings, it...
read it
-
Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
Policy evaluation is a key process in Reinforcement Learning (RL). It as...
read it
Leveraging the Variance of Return Sequences for Exploration Policy
This paper introduces a method for constructing an upper bound for exploration policy using either the weighted variance of return sequences or the weighted temporal difference (TD) error. We demonstrate that the variance of the return sequence for a specific state-action pair is an important information source that can be leveraged to guide exploration in reinforcement learning. The intuition is that fluctuation in the return sequence indicates greater uncertainty in the near future returns. This divergence occurs because of the cyclic nature of value-based reinforcement learning; the evolving value function begets policy improvements which in turn modify the value function. Although both variance and TD errors capture different aspects of this uncertainty, our analysis shows that both can be valuable to guide exploration. We propose a two-stream network architecture to estimate weighted variance/TD errors within DQN agents for our exploration method and show that it outperforms the baseline on a wide range of Atari games.
READ FULL TEXT
Comments
There are no comments yet.