) allows a reinforcement learning agent to represent, learn and plan with temporally extended actions. These temporally extended actions consist of a set of internal policies, termination conditions and sometimes initiation sets that allow controlling the number of choices available to an agent. Given a set of options, the agent will learn a policy over options, which is typically viewed as executing in a call-and-return fashion: once this policy chooses an option, the option will execute until it terminates, then the policy over options will make a new choice. Learning options is beneficial as it leads to specialization in the state space, and therefore to potentially reduced complexity in terms of the internal policies of the options. The option-critic architecture (Bacon et al. (2017)) provides an agent with an end-to-end algorithm to learn options in order to maximize the expected discounted return, by relying on ideas akin to actor-critic methods. In this work, we exploit the option-critic architecture by combining it to a recent algorithm, Proximal Policy Optimization (PPO) (Schulman et al. (2017)), which is very well suited for continuous control tasks and has shown better sample complexity in empirical comparisons. We present results of our approach on a set of environments from the Mujoco framework; our results are consistent with published evaluations which show that learning options provides increased performance, better interpretability and faster learning.
is a set of probability distributions over actions conditioned on states: . For a given policy, the value function defines the expected return obtained by following . satisfies the Bellman equations : .
The policy gradient theorem (Sutton et al. (1999a)) provides the gradient of a parametrized stochastic policy with respect to the expected discounted return from an initial state distribution . For simplicity, we write the policy as , making its parametrization () implicit.
where is a weighting of states along the trajectories generated by and passing through . Using the log-likelihood trick (Williams (1992)),
2.1 Trust region methods and Proximal Policy Optimization (PPO)
Trust region methods, and in particular the TRPO algorithm (Schulman et al. (2015a)), are second-order methods that maximize a surrogate objective subject to a constraint. TRPO has proven useful for continuous control, but it can be computationally expensive and doesn’t allow for parameter sharing.
Proximal Policy Optimiation (PPO) achieves the same level of reliability and performance as TRPO while being a first-order method. To do so, it uses an objective with clipped probability ratios, preventing an excessive shift in the probability distribution between updates. This clipping also allows for multiple epochs of minibatch updates on a single sampled trajectory. The clipped surrogate objective is:
where is the importance sampling ratio. The authors use the Generalized Advantage Estimation (Schulman et al. (2015b)) to calculate the advantage function .
The option-critic architecture (Bacon et al. (2017)) is a gradient-based approach for learning intra-option policies as well termination conditions, assuming that all options are available at every state. Moreover, the parameters of the intra-option policies () and the termination function () are assumed to be independent. The intra-option policy gradient is as follows:
where a baseline (i.e. the above state-option value function parametrized by ) is generally added. However, if the options are learned to optimize returns, in the long run, they will tend to disappear since any MDP can be solved optimally using primitive actions. To avoid this problem, Harb et al. (2018) use the bounded rationality framework (Simon (1969)) and introduce a deliberation cost (), interpreted as a margin of how much better an option should be than the current option in order to replace it. The termination gradient then takes the following form:
where is the termination advantage function and stems directly from the derivation of the gradient.
We introduce the Proximal Policy Option-Critic (PPOC) algorithm which, just like PPO, works in two stages. In the first stage, the agent collects trajectories of different options and computes the advantage functions using Monte-Carlo returns. We then proceed to the optimization stage where, for K optimizer iterations, we choose M tuples and apply the gradients. We also chose to use a stochastic policy over options, parameterized by an independent vector(as opposed to -greedy) which we learned under the same policy gradient approach.
We performed experiments on locomotion tasks available on OpenAI’s Gym (Brockman et al. (2016)) using the Mujoco simulator (Todorov et al. (2012)). We aim to assess the following: (1) whether the use of options can increase the speed of learning as well as the final performance, (2) the interpretability of the resulting options.
In our experiments, we used as input the vectors defining joint angles, joint velocities, and coordinates of the center of mass. We used two separate networks with 64 hidden units per layer, each containing two layers.111 The code, as well as the values for the hyperparameters, are available here:
The code, as well as the values for the hyperparameters, are available here:https://github.com/mklissa/PPOC For all the layers we used tanh non-linearity, except for the output which was linear for of value functions and intra-option policies, sigmoid for the termination probability and softmax for the policy over options. The first network was used to output the policy over options and the intra-option policies , while the second network was used to output the value functions and the termination probabilities
. The log-standard deviations were parameterized by a vector independent of the input state. We used the exact same hyper-parameters as mentioned inSchulman et al. (2017), except for the optimizer mini-batch size which was divided by the number of options. We proceeded so in order to avoid training more samples per iteration with the options framework, thus enabling a fair comparison between options and primitive actions. In the case of options, we also divide the reward by 10 to reduce the scale of the value functions, and therefore the termination probability gradient, making it more stable. We didn’t proceed to any hyper-parameters search to improve the results. Our experiments exclusively investigate the merits of using two options and compare the results to the case of primitive actions (no options).
In addition to the classic Mujoco environments, we ran agents in an environment called HopperIceBlock-v0.222HopperIceBlock-v0 is based on Henderson et al. (2017) and is avaialable here: https://github.com/mklissa/gym-extensions This environment contained a more explicit compositionality than the original Mujoco environments available on OpenAI’s Gym, which simply require to learn a gait that maximizes speed in a direction. We used Hopper-v1 as a starting point and added some obstacles in the agent’s path: solid slippery blocks. The agent had to learn to pass them either by jumping completely over them or by sliding on their surface.
The results are summarized in Fig.1. As expected, using options with a deliberation cost yields better results and faster learning on most environments. It is interesting to note that the increase in performance is not directly proportional to the value of . This is due to the different scales of the average returns across environments, as well as during the course of learning. In the current formulation of the deliberation cost, its value is a hyperparameter that has to be set. It would be useful to explore the possibility of working with a learned value instead of a constant. This is left as future work.
The results that stand out the most are the one on the customized environment. More importantly, the success threshold for the environment is around 1200 points, under that level the agent actually doesn’t learn to pass the iceblock and continue its gait. So, the agent using options is the only one solving this environment. This also led us to investigate how the options are used in this environment as opposed to the classic Mujoco environments.333Videos from the environments are available at https://www.youtube.com/watch?v=XI_txkRnKjU In HopperIceBlock-v0, the interpretability of the options is obvious and greatly helps the performance: one option is used to hop when there is no iceblock nearby, but then when passing over the iceblock, both options are used to complete the specific task. In the case of the classic Mujoco environments, one option is used to gain momentum at the start of the episode and is never used thereafter. Even if the agents using options outperform the agents with primitive actions on classic environments, we can only truly see the benefits of a hierarchical framework when used in the appropriate environment.
Our experiments demonstrate that it is possible to learn options in an end-to-end manner using deep networks on continuous actions environments, and to the best of our knowledge this is the first work to do so. Our results also suggest that the increase in performance is not directly linked to the deliberation cost, which is problematic as it leaves us with the task of finding the right value. For the options framework to be truly end-to-end it would be necessary to learn a value of . More importantly, we have seen that the increase in performance is related to the compositionality of the environment. In the classic Mujoco environments, using options is not as beneficial as using them in a customized environment with a more obvious division in the state-space. This leads to the following question: when should we be using options? This question also points to a fundamental problem in the current options framework: it is necessary for us to manually specify the number of available options. How should one decide on this number? As stated in Bacon et al. (2017), one way to answer this question would be to reintroduce the notion of initiation sets in the option-critic architecture.
- Bacon et al.  Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pages 1726–1734, 2017.
- Brockman et al.  Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.
- Harb et al.  Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option : Learning options with a deliberation cost. In AAAI, pages 1726–1734, 2018.
- Henderson et al.  P. Henderson, W.-D. Chang, F. Shkurti, J. Hansen, D. Meger, and G. Dudek. Benchmark environments for multitask learning in continuous domains. ICML Lifelong Learning: A Reinforcement Learning Approach Workshop, 2017.
- Precup  Doina Precup. Temporal abstraction in reinforcement learning. PhD thesis, University of Massachusetts Amhersts, 2000.
- Schulman et al. [2015a] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015.
- Schulman et al. [2015b] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. CoRR, abs/1506.02438, 2015.
- Schulman et al.  John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
- Simon  Herbert A. Simon. The Sciences of the Artificial. MIT Press, Cambridge, Massachusetts, first edition, 1969.
- Sutton et al. [1999a] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pages 1057–1063, 1999.
- Sutton et al. [1999b] Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2):181–211, 1999.
- Todorov et al.  Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012.
- Williams  Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, May 1992.