1 Introduction
The goal of modelfree reinforcement learning (RL) is to optimize an agent’s behavior policy through trial and error interaction with a black box environment. Valuebased RL algorithms such as Qlearning (Watkins, 1989) and policybased algorithms such as actorcritic (Konda & Tsitsiklis, 2000) have achieved wellknown successes in environments with enumerable action spaces and predictable but possibly complex dynamics, e.g., as in Atari games (Mnih et al., 2013; Van Hasselt et al., 2016; Mnih et al., 2016). However, when applied to environments with more sophisticated action spaces and dynamics (e.g., continuous control and robotics), success has been far more limited.
In an attempt to improve the applicability of Qlearning to continuous control, Silver et al. (2014) and Lillicrap et al. (2015)
developed an offpolicy algorithm DDPG, leading to promising results on continuous control environments. That said, current offpolicy methods including DDPG often improve data efficiency at the cost of optimization stability. The behaviour of DDPG is known to be highly dependent on hyperparameter selection and initialization
(Metz et al., 2017); even when using optimal hyperparameters, individual training runs can display highly varying outcomes.On the other hand, in an attempt to improve the stability and convergence speed of policybased RL methods, Kakade (2002) developed a natural policy gradient algorithm based on Amari (1998), which subsequently led to the development of trust region policy optimization (TRPO) (Schulman et al., 2015). TRPO has shown strong empirical performance on difficult continuous control tasks often outperforming valuebased methods like DDPG. However, a major drawback is that such methods are not able to exploit offpolicy data and thus require a large amount of onpolicy interaction with the environment, making them impractical for solving challenging realworld problems.
Efforts at combining the stability of trust region policybased methods with the sample efficiency of valuebased methods have focused on using offpolicy data to better train a value estimate, which can be used as a control variate for variance reduction
(Gu et al., 2017a, b).In this paper, we investigate an alternative approach to improving the sample efficiency of trust region policybased RL methods. We exploit the key fact that, under entropy regularization, the optimal policy and value function satisfy a set of pathwise consistency properties along any sampled path (Nachum et al., 2017), which allows both on and offpolicy data to be incorporated in an actorcritic algorithm, PCL. The original PCL algorithm optimized an entropy regularized maximum reward objective and was evaluated on relatively simple tasks. Here we extend the ideas of PCL to achieve strong results on standard, challenging continuous control benchmarks. The main observation is that by alternatively augmenting the maximum reward objective with a relative entropy regularizer, the optimal policy and values still satisfy a certain set of pathwise consistencies along any sampled trajectory. The resulting objective is equivalent to maximizing expected reward subject to a penaltybased constraint on divergence from a reference (i.e., previous) policy.
We exploit this observation to propose a new offpolicy trust region algorithm, TrustPCL, that is able to exploit offpolicy data to train policy and value estimates. Moreover, we present a simple method for determining the coefficient on the relative entropy regularizer to remain agnostic to reward scale, hence ameliorating the task of hyperparameter tuning. We find that the incorporation of a relative entropy regularizer is crucial for good and stable performance. We evaluate TrustPCL against TRPO, and observe that TrustPCL is able to solve difficult continuous control tasks, while improving the performance of TRPO both in terms of the final reward achieved as well as sampleefficiency.
2 Related Work
Trust Region Methods.
Gradient descent is the predominant optimization method for neural networks. A gradient descent step is equivalent to solving a trust region constrained optimization,
(1) 
which yields the locally optimal update such that ; hence by considering a Euclidean ball, gradient descent assumes the parameters lie in a Euclidean space.
However, in machine learning, particularly in the context of multilayer neural network training, Euclidean geometry is not necessarily the best way to characterize proximity in parameter space. It is often more effective to define an appropriate Riemannian metric that respects the loss surface
(Amari, 2012), which allows much steeper descent directions to be identified within a local neighborhood (e.g., Amari (1998); Martens & Grosse (2015)). Whenever the loss is defined in terms of a Bregman divergence between an (unknown) optimal parameter and model parameter , i.e., , it is natural to use the same divergence to form the trust region:(2) 
The natural gradient (Amari, 1998) is a generalization of gradient descent where the Fisher information matrix is used to define the local geometry of the parameter space around . If a parameter update is constrained by , a descent direction of is obtained. This geometry is especially effective for optimizing the loglikelihood of a conditional probabilistic model, where the objective is in fact the KL divergence . The local optimization is,
(3) 
Thus, natural gradient approximates the trust region by , which is accurate up to a second order Taylor approximation. Previous work (Kakade, 2002; Bagnell & Schneider, 2003; Peters & Schaal, 2008; Schulman et al., 2015) has applied natural gradient to policy optimization, locally improving expected reward subject to variants of . Recently, TRPO (Schulman et al., 2015, 2016) has achieved stateoftheart results in continuous control by adding several approximations to the natural gradient to make nonlinear policy optimization feasible.
Another approach to trust region optimization is given by proximal gradient methods (Parikh et al., 2014). The class of proximal gradient methods most similar to our work are those that replace the hard constraint in (2) with a penalty added to the objective. These techniques have recently become popular in RL (Wang et al., 2016; Heess et al., 2017; Schulman et al., 2017b), although in terms of final reward performance on continuous control benchmarks, TRPO is still considered to be the stateoftheart.
Norouzi et al. (2016) make the observation that entropy regularized expected reward may be expressed as a reversed KL divergence , which suggests that an alternative to the constraint in (3) should be used when such regularization is present:
(4) 
Unfortunately, this update requires computing the Fisher matrix at the endpoint of the update. The use of in previous work can be considered to be an approximation when entropy regularization is present, but it is not ideal, particularly if is large. In this paper, by contrast, we demonstrate that the optimal under the reverse KL constraint can indeed be characterized. Defining the constraint in this way appears to be more natural and effective than that of TRPO.
Softmax Consistency. To comply with the information geometry over policy parameters, previous work has used the relative entropy (i.e., KL divergence) to regularize policy optimization; resulting in a softmax relationship between the optimal policy and state values (Peters et al., 2010; Azar et al., 2012, 2011; Fox et al., 2016; Rawlik et al., 2013) under singlestep rollouts. Our work is unique in that we leverage consistencies over multistep rollouts.
The existence of multistep softmax consistencies has been noted by prior work—first by Nachum et al. (2017) in the presence of entropy regularization. The existence of the same consistencies with relative entropy has been noted by Schulman et al. (2017a). Our work presents multistep consistency relations for a hybrid relative entropy plus entropy regularized expected reward objective, interpreting relative entropy regularization as a trust region constraint. This work is also distinct from prior work in that the coefficient of relative entropy can be automatically determined, which we have found to be especially crucial in cases where the reward distribution changes dramatically during training.
Most previous work on softmax consistency (e.g., Fox et al. (2016); Azar et al. (2012); Nachum et al. (2017)) have only been evaluated on relatively simple tasks, including gridworld and discrete algorithmic environments. Rawlik et al. (2013) conducted evaluations on simple variants of the CartPole and Pendulum continuous control tasks. More recently, Haarnoja et al. (2017) showed that soft Qlearning (a singlestep special case of PCL) can succeed on more challenging environments, such as a variant of the Swimmer task we consider below. By contrast, this paper presents a successful application of the softmax consistency concept to difficult and standard continuouscontrol benchmarks, resulting in performance that is competitive with and in some cases beats the stateoftheart.
3 Notation & Background
We model an agent’s behavior by a policy distribution over a set of actions (possibly discrete or continuous). At iteration , the agent encounters a state and performs an action sampled from . The environment then returns a scalar reward and transitions to the next state . When formulating expectations over actions, rewards, and state transitions we will often omit the sampling distributions, , , and , respectively.
Maximizing Expected Reward. The standard objective in RL is to maximize expected future discounted reward. We formulate this objective on a perstate basis recursively as
(5) 
The overall, stateagnostic objective is the expected perstate objective when states are sampled from interactions with the environment:
(6) 
Most policybased algorithms, including REINFORCE (Williams & Peng, 1991) and actorcritic (Konda & Tsitsiklis, 2000), aim to optimize given a parameterized policy.
Path Consistency Learning (PCL). Inspired by Williams & Peng (1991), Nachum et al. (2017) augment the objective in (5) with a discounted entropy regularizer to derive an objective,
(7) 
where is a userspecified temperature parameter that controls the degree of entropy regularization, and the discounted entropy is recursively defined as
(8) 
Note that the objective can then be reexpressed recursively as,
(9) 
Nachum et al. (2017) show that the optimal policy for and mutually satisfy a softmax temporal consistency constraint along any sequence of states starting at and a corresponding sequence of actions :
(10) 
This observation led to the development of the PCL algorithm, which attempts to minimize squared error between the LHS and RHS of (10) to simultaneously optimize parameterized and . Importantly, PCL is applicable to both onpolicy and offpolicy trajectories.
Trust Region Policy Optimization (TRPO). As noted, standard policybased algorithms for maximizing can be unstable and require small learning rates for training. To alleviate this issue, Schulman et al. (2015) proposed to perform an iterative trust region optimization to maximize . At each step, a prior policy is used to sample a large batch of trajectories, then is subsequently optimized to maximize while remaining within a constraint defined by the average perstate KLdivergence with . That is, at each iteration TRPO solves the constrained optimization problem,
(11) 
The prior policy is then replaced with the new policy , and the process is repeated.
4 Method
To enable more stable training and better exploit the natural information geometry of the parameter space, we propose to augment the entropy regularized expected reward objective in (7) with a discounted relative entropy trust region around a prior policy ,
(12) 
where the discounted relative entropy is recursively defined as
(13) 
This objective attempts to maximize entropy regularized expected reward while maintaining natural proximity to the previous policy. Although previous work has separately proposed to use relative entropy and entropy regularization, we find that the two components serve different purposes, each of which is beneficial: entropy regularization helps improve exploration, while the relative entropy improves stability and allows for a faster learning rate. This combination is a key novelty.
Using the method of Lagrange multipliers, we cast the constrained optimization problem in (13) into maximization of the following objective,
(14) 
Again, the environmentwide objective is the expected perstate objective when states are sampled from interactions with the environment,
(15) 
4.1 Path Consistency with Relative Entropy
A key technical observation is that the objective has a similar decomposition structure to , and one can cast as an entropy regularized expected reward objective with a set of transformed rewards, i.e.,
(16) 
where is an expected reward objective on a transformed reward distribution function . Thus, in what follows, we derive a corresponding form of the multistep path consistency in (10).
Let denote the optimal policy, defined as . As in PCL (Nachum et al., 2017), this optimal policy may be expressed as
(17) 
where are the softmax state values defined recursively as
(18) 
We may rearrange (17) to yield
(19)  
(20) 
This is a singlestep temporal consistency which may be extended to multiple steps by further expanding on the RHS using the same identity. Thus, in general we have the following softmax temporal consistency constraint along any sequence of states defined by a starting state and a sequence of actions :
(21) 
4.2 TrustPCL
We propose to train a parameterized policy and value estimate to satisfy the multistep consistencies in (21). Thus, we define a consistency error for a sequence of states, actions, and rewards sampled from the environment as
(22)  
We aim to minimize the squared consistency error on every subtrajectory of length . That is, the loss for a given batch of episodes (or subepisodes) is
(23) 
We perform gradient descent on and to minimize this loss. In practice, we have found that it is beneficial to learn the parameter at least as fast as , and accordingly, given a minibatch of episodes we perform a single gradient update on and possibly multiple gradient updates on (see Appendix for details).
In principle, the minibatch may be taken from either onpolicy or offpolicy trajectories. In our implementation, we utilized a replay buffer prioritized by recency. As episodes (or subepisodes) are sampled from the environment they are placed in a replay buffer and a priority is given to a trajectory equivalent to the current training step. Then, to sample a batch for training, episodes are sampled from the replay buffer proportional to exponentiated priority for some hyperparameter .
For the prior policy
, we use a lagged geometric mean of the parameters. At each training step, we update
. Thus on average our training scheme attempts to maximize entropy regularized expected reward while penalizing divergence from a policy roughly training steps in the past.4.3 Automatic Tuning of The Lagrange Multiplier
The use of a relative entropy regularizer as a penalty rather than a constraint introduces several difficulties. The hyperparameter must necessarily adapt to the distribution of rewards. Thus, must be tuned not only to each environment but also during training on a single environment, since the observed reward distribution changes as the agent’s behavior policy improves. Using a constraint form of the regularizer is more desirable, and others have advocated its use in practice (Schulman et al., 2015) specifically to robustly allow larger updates during training.
To this end, we propose to redirect the hyperparameter tuning from to . Specifically, we present a method which, given a desired hard constraint on the relative entropy defined by , approximates the equivalent penalty coefficient . This is a key novelty of our work and is distinct from previous attempts at automatically tuning a regularizing coefficient, which iteratively increase and decrease the coefficient based on observed training behavior (Schulman et al., 2017b; Heess et al., 2017).
We restrict our analysis to the undiscounted setting with entropy regularizer . Additionally, we assume deterministic, finitehorizon environment dynamics. An additional assumption we make is that the expected KLdivergence over states is wellapproximated by the KLdivergence starting from the unique initial state . Although in our experiments these restrictive assumptions are not met, we still found our method to perform well for adapting during training.
In this setting the optimal policy of (14) is proportional to exponentiated scaled reward. Specifically, for a full episode , we have
(24) 
where and . The normalization factor of is
(25) 
We would like to approximate the trajectorywide KLdivergence between and . We may express the KLdivergence analytically:
(26)  
(27)  
(28)  
(29) 
Since all expectations are with respect to , this quantity is tractable to approximate given episodes sampled from
Therefore, in TrustPCL, given a set of episodes sampled from the prior policy and a desired maximum divergence , we can perform a simple line search to find a suitable which yields as close as possible to .
The preceding analysis provided a method to determine given a desired maximum divergence . However, there is still a question of whether should change during training. Indeed, as episodes may possibly increase in length, naturally increases when compared to the average perstate , and vice versa for decreasing length. Thus, in practice, given an and a set of sampled episodes , we approximate the best which yields a maximum divergence of . This makes it so that corresponds more to a constraint on the lengthaveraged KLdivergence.
To avoid incurring a prohibitively large number of interactions with the environment for each parameter update, in practice we use the last 100 episodes as the set of sampled episodes . While this is not exactly the same as sampling episodes from , it is not too far off since is a lagged version of the online policy . Moreover, we observed this protocol to work well in practice. A more sophisticated and accurate protocol may be derived by weighting the episodes according to the importance weights corresponding to their true sampling distribution.
5 Experiments
We evaluate TrustPCL against TRPO on a number of benchmark tasks. We choose TRPO as a baseline since it is a standard algorithm known to achieve stateoftheart performance on the continuous control tasks we consider (see e.g., leaderboard results on the OpenAI Gym website (Brockman et al., 2016)). We find that TrustPCL can match or improve upon TRPO’s performance in terms of both average reward and sample efficiency.
5.1 Setup
We chose a number of control tasks available from OpenAI Gym (Brockman et al., 2016). The first task, Acrobot, is a discretecontrol task, while the remaining tasks (HalfCheetah, Swimmer, Hopper, Walker2d, and Ant) are wellknown continuouscontrol tasks utilizing the MuJoCo environment (Todorov et al., 2012).
For TRPO we trained using batches of steps ( for Acrobot), which is the approximate batch size used by other implementations (Duan et al., 2016; Schulman, 2017). Thus, at each training iteration, TRPO samples steps using the policy and then takes a single step within a KLball to yield a new .
TrustPCL is offpolicy, so to evaluate its performance we alternate between collecting experience and training on batches of experience sampled from the replay buffer. Specifically, we alternate between collecting steps from the environment and performing a single gradient step based on a batch of size subepisodes of length from the replay buffer, with a recency weight of on the sampling distribution of the replay buffer. To maintain stability we use and we modified the loss from squared loss to Huber loss on the consistency error. Since our policy is parameterized by a unimodal Gaussian, it is impossible for it to satisfy all path consistencies, and so we found this crucial for stability.
For each of the variants and for each environment, we performed a hyperparameter search to find the best hyperparameters. The plots presented here show the reward achieved during training on the best hyperparameters averaged over the best seeds of randomly seeded training runs. Note that this reward is based on greedy actions (rather than random sampling).
Experiments were performed using Tensorflow
(Abadi et al., 2016). Although each training step of TrustPCL (a simple gradient step) is considerably faster than TRPO, we found that this does not have an overall effect on the run time of our implementation, due to a combination of the fact that each environment step is used in multiple training steps of TrustPCL and that a majority of the run time is spent interacting with the environment. A detailed description of our implementation and hyperparameter search is available in the Appendix.5.2 Results
Acrobot  HalfCheetah  Swimmer 
Hopper  Walker2d  Ant 
The results of TrustPCL against a TRPO baseline. Each plot shows average greedy reward with single standard deviation error intervals capped at the min and max across
best of randomly seeded training runs after choosing best hyperparameters. The xaxis shows millions of environment steps. We observe that TrustPCL is consistently able to match and, in many cases, beat TRPO’s performance both in terms of reward and sample efficiency.Hopper  Walker2d 

Hopper  Walker2d 

Domain  TRPOGAE  TRPO (rllab)  TRPO (ours)  TrustPCL  IPG 

HalfCheetah  4871.36  2889  4343.6  7057.1  4767 
Swimmer  137.25  –  288.1  297.0  – 
Hopper  3765.78  –  3516.7  3804.9  – 
Walker2d  6028.73  1487  2838.4  5027.2  3047 
Ant  2918.25  1520  4347.5  6104.2  4415 
We present the reward over training of TrustPCL and TRPO in Figure 1. We find that TrustPCL can match or beat the performance of TRPO across all environments in terms of both final reward and sample efficiency. These results are especially significant on the harder tasks (Walker2d and Ant). We additionally present our results compared to other published results in Table 1. We find that even when comparing across different implementations, TrustPCL can match or beat the stateoftheart.
5.2.1 Hyperparameter Analysis
The most important hyperparameter in our method is , which determines the size of the trust region and thus has a critical role in the stability of the algorithm. To showcase this effect, we present the reward during training for several different values of in Figure 2. As increases, instability increases as well, eventually having an adverse effect on the agent’s ability to achieve optimal reward. Note that standard PCL (Nachum et al., 2017) corresponds to (that is, ). Therefore, standard PCL would fail in these environments, and the use of trust region is crucial.
The main advantage of TrustPCL over existing trust region methods for continuous control is its ability to learn in an offpolicy manner. The degree to which TrustPCL is offpolicy is determined by a combination of the hyparparameters , , and . To evaluate the importance of training offpolicy, we evaluate TrustPCL with a hyperparameter setting that is more onpolicy. We set , , and . In this setting, we also use large batches of episodes of length (a total of environment steps per batch). Figure 3 shows the results of TrustPCL with our original parameters and this new setting. We note a dramatic advantage in sample efficiency when using offpolicy training. Although TrustPCL (onpolicy) can achieve stateoftheart reward performance, it requires an exorbitant amount of experience. On the other hand, TrustPCL (offpolicy) can be competitive in terms of reward while providing a significant improvement in sample efficiency.
One last hyperparameter is , determining the degree of exploration. Anecdotally, we found to not be of high importance for the tasks we evaluated. Indeed many of our best results use . Including had a marginal effect, at best. The reason for this is likely due to the tasks themselves. Indeed, other works which focus on exploration in continuous control have found the need to propose explorationadvanageous variants of these standard benchmarks (Haarnoja et al., 2017; Houthooft et al., 2016).
6 Conclusion
We have presented TrustPCL, an offpolicy algorithm employing a relativeentropy penalty to impose a trust region on a maximum reward objective. We found that TrustPCL can perform well on a set of standard control tasks, improving upon TRPO both in terms of average reward and sample efficiency. Our best results on TrustPCL are able to maintain the stability and solution quality of TRPO while approaching the sampleefficiency of valuebased methods (see e.g., Metz et al. (2017)). This gives hope that the goal of achieving both stability and sampleefficiency without tradingoff one for the other is attainable in a single unifying RL algorithm.
7 Acknowledgment
We thank Matthew Johnson, Luke Metz, Shane Gu, and the Google Brain team for insightful comments and discussions.
References
 Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for largescale machine learning. arXiv:1605.08695, 2016.
 Amari (1998) ShunIchi Amari. Natural gradient works efficiently in learning. Neural Comput., 10, 1998.
 Amari (2012) ShunIchi Amari. Differentialgeometrical methods in statistics, volume 28. Springer Science & Business Media, 2012.
 Azar et al. (2011) Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert J Kappen. Dynamic policy programming with function approximation. AISTATS, 2011.
 Azar et al. (2012) Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert J Kappen. Dynamic policy programming. JMLR, 13, 2012.
 Bagnell & Schneider (2003) J Andrew Bagnell and Jeff Schneider. Covariant policy search. 2003.
 Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv:1606.01540, 2016.
 Duan et al. (2016) Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. 2016.
 Fox et al. (2016) Roy Fox, Ari Pakman, and Naftali Tishby. Glearning: Taming the noise in reinforcement learning via soft updates. Uncertainty in Artifical Intelligence, 2016. URL http://arxiv.org/abs/1512.08562.
 Gu et al. (2017a) Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Qprop: Sampleefficient policy gradient with an offpolicy critic. ICLR, 2017a.
 Gu et al. (2017b) Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging onpolicy and offpolicy gradient estimation for deep reinforcement learning. arXiv preprint arXiv:1706.00387, 2017b.
 Haarnoja et al. (2017) Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energybased policies. arXiv preprint arXiv:1702.08165, 2017.
 Heess et al. (2017) Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017.
 Houthooft et al. (2016) Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109–1117, 2016.
 Kakade (2002) Sham M Kakade. A natural policy gradient. In NIPS, 2002.
 Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
 Konda & Tsitsiklis (2000) Vijay R Konda and John N Tsitsiklis. Actorcritic algorithms, 2000.
 Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
 Martens & Grosse (2015) James Martens and Roger Grosse. Optimizing neural networks with kroneckerfactored approximate curvature. In ICML, 2015.
 Metz et al. (2017) Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep RL. CoRR, abs/1705.05035, 2017. URL http://arxiv.org/abs/1705.05035.
 Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. arXiv:1312.5602, 2013.
 Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, 2016.
 Nachum et al. (2017) Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. CoRR, abs/1702.08892, 2017. URL http://arxiv.org/abs/1702.08892.
 Norouzi et al. (2016) Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. NIPS, 2016.
 Parikh et al. (2014) Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and Trends® in Optimization, 1(3):127–239, 2014.
 Peters & Schaal (2008) Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21, 2008.
 Peters et al. (2010) Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In AAAI, 2010.

Rawlik et al. (2013)
Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar.
On stochastic optimal control and reinforcement learning by
approximate inference.
In
TwentyThird International Joint Conference on Artificial Intelligence
, 2013.  Schulman (2017) John Schulman. Modular rl. http://github.com/joschu/modular_rl, 2017. Accessed: 20170601.
 Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
 Schulman et al. (2016) John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. ICLR, 2016.
 Schulman et al. (2017a) John Schulman, Pieter Abbeel, and Xi Chen. Equivalence between policy gradients and soft qlearning. arXiv preprint arXiv:1704.06440, 2017a.
 Schulman et al. (2017b) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b.
 Silver et al. (2014) David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning (ICML14), pp. 387–395, 2014.
 Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for modelbased control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026–5033. IEEE, 2012.
 Van Hasselt et al. (2016) Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qlearning. AAAI, 2016.
 Wang et al. (2016) Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actorcritic with experience replay. arXiv preprint arXiv:1611.01224, 2016.
 Watkins (1989) Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge England, 1989.
 Williams & Peng (1991) Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 1991.
Appendix A Implementation Benefits of TrustPCL
We have already highlighted the ability of TrustPCL to use offpolicy data to stably train both a parameterized policy and value estimate, which sets it apart from previous methods. We have also noted the ease with which exploration can be incorporated through the entropy regularizer. We elaborate on several additional benefits of TrustPCL.
Compared to TRPO, TrustPCL is much easier to implement. Standard TRPO implementations perform secondorder gradient calculations on the KLdivergence to construct a Fisher information matrix (more specifically a vector product with the inverse Fisher information matrix). This yields a vector direction for which a line search is subsequently employed to find the optimal step. Compare this to TrustPCL which employs simple gradient descent. This makes implementation much more straightforward and easily realizable within standard deep learning frameworks.
Even if one replaces the constraint on the average KLdivergence of TRPO with a simple regularization penalty (as in proximal policy gradient methods (Schulman et al., 2017b; Wang et al., 2016)), optimizing the resulting objective requires computing the gradient of the KLdivergence. In TrustPCL, there is no such necessity. The perstate KLdivergence need not have an analytically computable gradient. In fact, the KLdivergence need not have a closed form at all. The only requirement of TrustPCL is that the logdensity be analytically computable. This opens up the possible policy parameterizations to a much wider class of functions. While continuous control has traditionally used policies parameterized by unimodal Gaussians, with TrustPCL the policy can be replaced with something much more expressive—for example, mixtures of Gaussians or autoregressive policies as in Metz et al. (2017).
We have yet to fully explore these additional benefits in this work, but we hope that future investigations can exploit the flexibility and ease of implementation of TrustPCL to further the progress of RL in continuous control environments.
Appendix B Experimental Setup
We describe in detail the experimental setup regarding implementation and hyperparameter search.
b.1 Environments
In Acrobot, episodes were cutoff at step . For the remaining environments, episodes were cutoff at step .
Acrobot, HalfCheetah, and Swimmer are all nonterminating environments. Thus, for these environments, each episode had equal length and each batch contained the same number of episodes. Hopper, Walker2d, and Ant are environments that can terminate the agent. Thus, for these environments, the batch size throughout training remained constant in terms of steps but not in terms of episodes.
There exists an additional common MuJoCo task called Humanoid. We found that neither our implementation of TRPO nor TrustPCL could make more than negligible headway on this task, and so omit it from the results. We are aware that TRPO with the addition of GAE and enough finetuning can be made to achieve good results on Humanoid (Schulman et al., 2016). We decided to not pursue a GAE implementation to keep a fair comparison between variants. TrustPCL can also be made to incorporate an analogue to GAE (by maintaining consistencies at varying time scales), but we leave this to future work.
b.2 Implementation Details
We use fullyconnected feedforward neural networks to represent both policy and value.
The policy is represented by a neural network with two hidden layers of dimension with activations. At time step , the network is given the observation . It produces a vector , which is combined with a learnable (but agnostic) parameter to parametrize a unimodal Gaussian with mean and standard deviation . The next action is sampled randomly from this Gaussian.
The value network is represented by a neural network with two hidden layers of dimension with activations. At time step the network is given the observation and the componentwise squared observation . It produces a single scalar value.
b.2.1 TRPO Learning
At each training iteration, both the policy and value parameters are updated. The policy is trained by performing a trust region step according to the procedure described in Schulman et al. (2015).
The value parameters at each step are solved using an LBFGS optimizer. To avoid instability, the value parameters are solved to fit a mixture of the empirical values and the expected values. That is, we determine to minimize , where again is the previous value parameterization. We use . This method for training is according to that used in Schulman (2017).
b.2.2 TrustPCL Learning
At each training iteration, both the policy and value parameters are updated. The specific updates are slightly different between TrustPCL (onpolicy) and TrustPCL (offpolicy).
For TrustPCL (onpolicy), the policy is trained by taking a single gradient step using the Adam optimizer (Kingma & Ba, 2015) with learning rate . The value network update is inspired by that used in TRPO we perform 5 gradients steps with learning rate , calculated with regards to a mix between the empirical values and the expected values according to the previous . We use .
For TrustPCL (offpolicy), both the policy and value parameters are updated in a single step using the Adam optimizer with learning rate . For this variant, we also utilize a target value network (lagged at the same rate as the target policy network) to replace the value estimate at the final state for each path. We do not mix between empirical and expected values.
b.3 Hyperparameter Search
We found the most crucial hyperparameters for effective learning in both TRPO and TrustPCL to be (the constraint defining the size of the trust region) and (the rollout determining how to evaluate the empirical value of a state). For TRPO we performed a grid search over . For TrustPCL we performed a grid search over . For TrustPCL we also experimented with the value of , either keeping it at a constant 0 (thus, no exploration) or decaying it from to by a smoothed exponential rate of every 2,500 training iterations.
We fix the discount to for all environments.
Appendix C Pseudocode
A simplified pseudocode for TrustPCL is presented in Algorithm 1.
Comments
There are no comments yet.