-
Deterministic Value-Policy Gradients
Reinforcement learning algorithms such as the deep deterministic policy ...
09/09/2019 ∙ by Qingpeng Cai, et al. ∙
0
∙
share
read it
-
Quantile Regression Deep Reinforcement Learning
Policy gradient based reinforcement learning algorithms coupled with neu...
06/27/2019 ∙ by Oliver Richter, et al. ∙
6
∙
share
read it
-
Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
Recent analyses of certain gradient descent optimization methods have sh...
10/05/2018 ∙ by Peter Henderson, et al. ∙
12
∙
share
read it
-
Imitation-Projected Policy Gradient for Programmatic Reinforcement Learning
We present Imitation-Projected Policy Gradient (IPPG), an algorithmic fr...
07/11/2019 ∙ by Abhinav Verma, et al. ∙
3
∙
share
read it
-
Divide-and-Conquer Reinforcement Learning
Standard model-free deep reinforcement learning (RL) algorithms sample a...
11/27/2017 ∙ by Dibya Ghosh, et al. ∙
0
∙
share
read it
-
Computational Flight Control: A Domain-Knowledge-Aided Deep Reinforcement Learning Approach
This papers aims to examine the potential of using the emerging deep rei...
08/19/2019 ∙ by Hyo-Sang Shin, et al. ∙
2
∙
share
read it
-
Batch Policy Gradient Methods for Improving Neural Conversation Models
We study reinforcement learning of chatbots with recurrent neural networ...
02/10/2017 ∙ by Kirthevasan Kandasamy, et al. ∙
0
∙
share
read it
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning
06/01/2017 ∙ by Shixiang Gu, et al. ∙
0
∙
share
Off-policy model-free deep reinforcement learning methods using previously collected data can improve sample efficiency over on-policy policy gradient techniques. On the other hand, on-policy algorithms are often more stable and easier to use. This paper examines, both theoretically and empirically, approaches to merging on- and off-policy updates for deep reinforcement learning. Theoretical results show that off-policy updates with a value function estimator can be interpolated with on-policy policy gradient updates whilst still satisfying performance bounds. Our analysis uses control variate methods to produce a family of policy gradient algorithms, with several recently proposed algorithms being special cases of this family. We then provide an empirical comparison of these techniques with the remaining algorithmic details fixed, and show how different mixing of off-policy gradient estimates with on-policy samples contribute to improvements in empirical performance. The final algorithm provides a generalization and unification of existing deep policy gradient techniques, has theoretical guarantees on the bias introduced by off-policy updates, and improves on the state-of-the-art model-free deep RL methods on a number of OpenAI Gym continuous control benchmarks.
READ FULL TEXT