In the past few years, deep reinforcement learning (DRL) has shown remarkable progress in challenging application domains, such as Atari GamesMnih et al. (2015), Go game Silver et al. (2017), poker Brown & Sandholm (2018), StarCraft II Vinyals et al. (2019), and Dota 2 Openai et al. (2019)
. The combination of RL and high-capacity function approximators, such as neural networks, holds the promise of solving complex tasks in continuous control. However, millions of steps of data collection are needed to train effective behaviors. This training process might be simplified with a comprehensive understanding of tasks. A sophisticated agent should have the ability to identify distinct temporally-extended sub-tasks in a long-horizon task. How to efficiently discover such temporal abstractions has been widely studied in reinforcement learning (RL)McGovern & Barto (2001); Barto & Mahadevan (2003); Konidaris & Barto (2009); Da Silva et al. (2012); Kulkarni et al. (2016); Li et al. (2019). In this paper, we focus on the option framework Sutton et al. (1999), a distinct temporal abstraction method that can automatically discover courses of action with different intervals Riemer et al. (2018). This distinct hierarchical structure has achieved notable success recently Bacon et al. (2017); Fox et al. (2017).
However, there are remain challenges hampering widespread adoption of the option framework. One important such aspect is exploration. The option framework suffers from a degradation problem caused by ineffective exploration: there might be just one option selected to complete the entire task, which is tantamount to traditional end-to-end learning. Previous research tends to use on-policy learning to concurrently train option selection policy and intra-option policies Riemer et al. (2018); Bacon et al. (2017); Fox et al. (2017); Zhang & Whiteson (2019). However, only actually invoked options can be updated in on-policy learning. Intra-option policies sampled more frequently will be trained better to get more chance to be selected. This biased sampling makes the degradation problem worse. Another widely-known challenge is instability caused by simultaneous updates of high-level and low-level policies. Learning of intra-option policies will be unstable if the option selection policy frequently switches options to solve one sub-task. Previous work adapts option selection policy to updates of intra-option policies Zhang & Whiteson (2019); Osa et al. (2019). However, this short-sighted learning might exacerbate instability.
To address these challenges, we present an off-policy soft option actor-critic (SOAC) approach that maximizes discounted rewards with entropy terms. This maximum entropy formulation provides sufficient exploration and robustness while acquiring diverse behaviors Haarnoja et al. (2017, 2018). The entropy bonus encourages the option selection policy to consider each intra-option policy in a balanced way. In addition, we introduce an information-theoretical intrinsic reward to enhance identifiability of intra-option policies. We utilize this intrinsic reward with another intrinsic reward related to anti-interference to define the objective for learning the optimal option selection policy. Meanwhile, we utilize external rewards to define the objective for learning the action selection policy. We theoretically derive that optimizing our maximum entropy model is equivalent to fitting optimal trajectories. Our algorithm can alternate between policy evaluation and policy improvement to learn optimal policies. Moreover, in our approach, the soft optimality of policies allows that behavior policies can be different from target policies Levine (2018); Schulman et al. (2017a). With this flexibility, the option selection policy can be trained to select options with considering all historical behavior of each intra-option policy to reduce instability. As shown in Figure 1, our algorithm learns a deep hierarchy of options.
Experimental results indicate that our hierarchical approach significantly improves the performance of SAC Haarnoja et al. (2018) and outperforms state-of-the-art hierarchical RL algorithms Zhang & Whiteson (2019); Osa et al. (2019) on the benchmark Mujoco tasks (Section 5). In addition, we observe an obvious distinction between options, which indicates that a well trained option selection policy is sophisticated enough to invoke a diverse set of options in different situations. We also show that, the option selection policy can be transferred and accelerate learning in a new environment, even if the target task is dramatically different from the original one.
2 Related Work
Considerable prior work has explored how to extend the option framework Sutton et al. (1999) to deep reinforcement learning (DRL). Compared with the end-to-end learning progress, learning the option framework from a single task brings more complex networks and more computational complexity. How to quickly learn an effective hierarchical structure is still an open question. Bacon et al. Bacon et al. (2017) train the whole option framework with policy gradient method. To leverage recent advances in gradient-based policies, the option framework has combined with PPO Zhang & Whiteson (2019); Schulman et al. (2017b), TD3 Osa et al. (2019); Fujimoto et al. (2018) and multitask Igl et al. (2019). To improve sample efficiency, all available intra-option policies can be trained simultaneously with a marginal distribution evaluating the probability of all options being selected Smith et al. (2018). In addition, important sampling (IS) has been used to propose off-policy algorithms Harutyunyan et al. (2018); Guo et al. (2017) to reuse past experience. These research does show some interesting ways forward. However, they are not efficient enough compared with current baseline model-free DRL algorithms such as SAC Haarnoja et al. (2018).
Probability inference models provide a way to analyze the probability of optimal trajectories Levine (2018); Kappen et al. (2012); Schulman et al. (2017a). Recently, these models have been adapted to numerous environments with DRL. Eysenbach et al. Eysenbach et al. (2018) utilize diversity only to update parameters. Haarnoja et al. Haarnoja et al. (2018) propose the Soft Actor-Critic algorithm which is a state-of-the-art algorithm in single agent DRL. Huang et al. Huang et al. (2019) optimize Partially Observable MDPs (POMDPs) with sequential variational soft Q-learning. Previous work based on soft optimality has shown both sample-efficient learning and stability. Meanwhile, the information bottleneck, related to mutual information (MI) and the Kullback-Leibler (KL) divergence, is widely used to control the spread of information Alemi et al. (2016); Galashov et al. (2019); Goyal et al. (2019a); Wang et al. (2019). It can be used to judge division of state-action space Osa et al. (2019) or used to distinguish different skills Sharma et al. (2019). We propose our approach on these basis.
3.1 The Option Framework
Traditional Markov Decision Process (MDP) considers a tuple. is the state space, is the action space, is the transition probability, is the relevant reward function, and is a discount factor. The option framework extends the original MDP problem to a SMDP problem. It consists three components: a policy choosing options , a termination condition , and an initiation set Sutton et al. (1999). In this paper, we use to denote the option space. At each time step , agents will decide whether to terminate the previous intra-option policy labeled as with the termination probability . If the previous intra-option policy is terminated, another intra-option policy will be sampled from . The whole probability of transitioning options written as below is called high-level option selection policy in this paper. Meanwhile each action is sampled from corresponding to the current option and state.
3.2 Probability Inference Models
Different from the general form of reinforcement learning problems, DRL based on probability inference models attempts to directly optimize the probability of optimal trajectories. An additional variable is introduced to denote whether the current time step is optimal. This variable provides a mathematical formalization to analyze whether current policies are optimal. The log form of the probability of the optimal trajectories can be theoretically proved having an evidence low bound related to dense rewards and entropy Levine (2018).
where is the actor policy, and is the entropy regular term.
In this section, we propose a maximum entropy problem and simplify it as fitting optimal trajectories with probability inference models. We propose an algorithm to estimate optimal policies iteratively.
4.1 Problem Formulation
Although previous research based on the option framework usually considers directly maximizing the reward function. We are interested in optimizing a maximum entropy model to solve the ineffective exploration challenge. In addition, we introduce mutual information as an intrinsic reward to enhance identifiability of each intra-option policy. Meanwhile, disturbance in a state-action pair should not lead to a substantial change in option selection Li et al. (2019); Puri et al. (2019). So we add another intrinsic reward based on TV distance to encourage option selection policy to consider connectivity in state-action space while allocating options. In , , and are gaussian noise, and represents parameters of our model which can be neural networks. The whole maximum entropy problem is:
where we label high-level option selection policy as and label low-level intra-option policys as , is entropy,
is a hyperparameter representing importance of external rewards,and are weights of intrinsic rewards.
To simplify the above problem, we introduce probability inference models. An additional variable are introduced to describe whether the current condition is optimal. indicates time step is optimal, and indicates time step is not optimal. In the rest of this paper, we use to represent for concise functions. With this additional variable, we define a conditional probability model representing the probability of a trajectory with optimal policies:
where means for all steps from to . The probability of whether a state-action pair is optimal is defined as below, which is based on boltzmann distribution of energy Levine (2018).
Inspired by Equation 5, we utilize a similar exponential form to define the optimal option selection.
With option selection policy selecting options and intra-option policy selecting actions, the probability of sampling a trajectory is:
The original maximum entropy optimization problem shown in Equation3 can be simplified as shrinking the Kullback-Leibler (KL) divergence between and .
Proof. See supplementary materials.
4.2 Optimal Policies with Probability Inference Models
In this sub-section, we derive optimal policies with probability inference models. First, we introduce three backward messages: , and . These messages denote the probability of whether a trajectory starting from corresponding condition is optimal. With these backward messages, we can derive optimal option selection probability and optimal action selection probability as below.
Inspired by Levine Levine (2018), we use the log form of three backward messages to define value functions. We define , , and . With these value functions, optimal high level policy and optimal low level policy are derived as below.
where controls exploration degree. If
approaches infinity, optimal policies will obey uniform distribution. In contrast, ifapproaches zero, optimal policies will be greedy. To estimate , and , we derive relationships between them.
The relationship between and is:
Proof. See supplementary materials.
The relationship between and is:
Proof. See supplementary materials.
The relationship between and is:
Proof. See supplementary materials.
With these relationships between value functions, we can iteratively train them to estimate optimal policies. In the next sub-section, we will explain our algorithm in detail.
In this subsection, we will literally show our training process. We use function approximators and stochastic gradient descent to estimate and train U-value functionsand , Q-value functions and , option selection policy , and intra-option policys . For more stable training, we utilize double neural networks Fujimoto et al. (2018); Van Hasselt et al. (2016) and target neural networks Van Hasselt et al. (2016); Mnih et al. (2015) while estimating U-value functions and Q-value functions. Q-value functions are trained by minimizing the Bellman residual shown as below, where we use the relationship between and shown in Equation 13to replace .
The Bellman residual of U-value functions are:
It is difficult to directly calculate optimal high level policy and optimal low level policy from Equation 11 and Equation 12. We use KL divergence to estimate policies. Option selection policy can be optimized by minimizing . Our option space is discrete. So we calculcate the expectation directly Christodoulou (2019).
where and are the list of and , , decides whether to terminate previous options, chooses new options, and . Here both and are trained by minimizing .
Intra-option policy is also optimized by minimizing . We use the reparameterization trick to allow gradients to pass through the expectations operator. At each time step , is sampled from , where
In this section, we design experiments to answer following questions: (1) Can the additional option framework accelerate training? (2) Whether state-action space related to each option has strong coherence? (3) What is the impact of a well trained option selection policy in an opposite task? We adapt several benchmarking robot control tasks in Mujoco domains to answer the above questions.
5.1 Results and Comparisons
We compare our algorithm with three other algorithms: Soft Actor-Critic (SAC)Haarnoja et al. (2018), Double Actor-Critic (DAC) Zhang & Whiteson (2019) and adInfoHRL Osa et al. (2019). SAC is a current baseline off-policy DRL algorithm, which is also based on maximum entropy and probability inference models. We use it here to test whether our option framework can accelerate learning. Meanwhile, to the best of our knowledge, DAC and adInfoHRL are current best on-policy and off-policy algorithms with a similar hierarchical structure introducing a hidden and latent variable to abstractly present state-action space. All corresponding hyperparameters are literally listed in supplement materials.
demonstrates the average return of test rollout during training for SOAC (our algorithm), SAC, DAC and adInfoHRL on four Mujoco tasks. We train four different instances of each algorithm with random seeds from zero to three with each performing ten evaluation rollouts every 5000 environment steps and choose the best three instances. The solid curves represent the mean value smoothed by the Moving Average method and the shaded region represents the minimum and maximum returns over related trials. We notice that our algorithm dramatically outperforms DAC and adInfoHRL, both in terms of learning speed and stability. For example, on Hopper-v2, DAC and adInfoHRL suffer from unstable learning, but our algorithm quickly stabilizes at the highest score. Meanwhile, on Ant-v2, addInfoRL fails to make any progress, but our algorithm dramatically outperforms other algorithms. Compared with SAC, our algorithm performs comparably on HalfCheetah-v2 and Walker2d-v2 and outperforms on Hopper-v2 and Ant-v2. These results indicate that our algorithm can accelerate learning by softly dividing state-action space based on the option framework with sufficient exploration. We address part of the reason as the multimodal treatment of our actor’s policy. To deal with continuous action space, an actor’s policy is usually defined as a normal distribution. However, this might not meet the actual optimal policy. The entire policy of our actor has a multimodal distribution similar to Gaussian Mixture Model (GMM) and give our agents a stronger ability to make decisions. In addition, part of neural networks related to different intra-option policies are shared to accelerate trainingZheng12 et al. (2018)
. This provides the same feature extraction strategy for all intra-option policies.
5.2 Visualization of State-Action Space with Different Options
Our algorithm performs well in Mujoco domains with stable learning curves. To verify whether our option selection policy is reasonable, we utilize the t-sne method Maaten & Hinton (2008) to illustrate state-action space corresponding to each option in Figure 3. We notice distinct clusters for different options in each Mujoco task. This indicates that our option selection policy is well trained to assign options for different situations. In addition, we notice multi-cluster related to one option, which is similiar with Osa et al. (2019); Oord et al. (2018); Goyal et al. (2019b). This is because option selection policy might assign different sub-tasks for one option to solve limited by the fixed number of options. How to determine the most suitable number of options is still an open question, although most previous research tends to set the number to four Bacon et al. (2017); Smith et al. (2018); Osa et al. (2019); Zhang & Whiteson (2019). An exploration of variable number of options is a future direction.
5.3 Transfer of Option Selction Policy
We are wondering whether the option selection policy learned from a certain task can discover a general division method of the environment. Even though our algorithm is not designed for transfer learning, we find out that our well trained option selection policy can accelerate training in a diametrically different task with an opposite reward function. As shown in Figure 4, transferring high-level option selection policy will accelerate learning compared with transferring nothing in most Mujoco domains. Meanwhile, the transferred option selection policy makes the training more stable. Especially on Hopper-v2, a hopper suffers from falling down while attempting to jump backwards. With transferred option selection policy, agents have more opportunities to learn to jump backwards rather than staying in place. These results indicate that our well trained option selection policy can generally divide the environment and assign sub-tasks with probability models, which will provide benefits for transfer learning.
In this paper, we propose soft option actor-critic (SOAC), an off-policy maximum entropy DRL algorithm with the option framework. With probability inference models, we theoretically derive optimal policies based on soft optimality and simplify our optimization problem as fitting optimal trajectories. We empirically demonstrate that our algorithm matches or exceeds prior on-policy and off-policy methods in a range of Mujoco benchmark tasks and while still providing benefits for transfer learning. The state-action space associated with each option shows strong connectivity. These results indicate that our option selection policy is sophisticated to assign options for different situation. Our algorithm has shown the potential to boost sample efficiency with operative exploration to address current well-known challenges restricting the applicability of the option framework.
7 Broader Impact
Deep reinforcement learning (DRL) has achieved remarkable progress in recent years. It has exceeded the human level performance in many challenging environments such as Atari Games Mnih et al. (2015, 2013), game of go Silver et al. (2017), poker Brown & Sandholm (2018), and StarCraft II Vinyals et al. (2019)
. However, classical end-to-end learning progress still suffers from high dimension in state and action space, which might influence the convergence rate and cause unbearable training time. In this paper, we attempt to train an option framework, which can extract sub-tasks with arbitrary interval from a long-horizon task to simplify the original MDP problem. We combine the option framework with probability inference models and information-theoretical intrinsic rewards and propose a novel and stable off-policy algorithm to address the well known challenges mentioned in the introduction section. As we all know, creation starts from the ability to discover and summarize problems. With the option framework, agents can learn diverse skills from sub-tasks proposed by themselves while solving the entire task. In general, the option framework encourages agents to explore the environment and ask questions. This might be a key point in the artificialization of artificial intelligence. Learning the option framework will definitely bring more computational complexity. Nevertheless, our approach has shown that learning this hierarchical structure can accelerate training in Mujoco domains. Our approach can be regarded as a step for the option framework to be widespreadly adopted.
- Alemi et al. (2016) Alemi, Alexander A, Fischer, Ian, Dillon, Joshua V, & Murphy, Kevin. 2016. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410.
- Bacon et al. (2017) Bacon, Pierre-Luc, Harb, Jean, & Precup, Doina. 2017. The option-critic architecture. In: Thirty-First AAAI Conference on Artificial Intelligence.
- Barto & Mahadevan (2003) Barto, Andrew G, & Mahadevan, Sridhar. 2003. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1-2), 41–77.
- Brown & Sandholm (2018) Brown, Noam, & Sandholm, Tuomas. 2018. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374), 418–424.
- Christodoulou (2019) Christodoulou, Petros. 2019. Soft Actor-Critic for Discrete Action Settings. arXiv preprint arXiv:1910.07207.
- Da Silva et al. (2012) Da Silva, Bruno, Konidaris, George, & Barto, Andrew. 2012. Learning parameterized skills. arXiv preprint arXiv:1206.6398.
- Eysenbach et al. (2018) Eysenbach, Benjamin, Gupta, Abhishek, Ibarz, Julian, & Levine, Sergey. 2018. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070.
- Fox et al. (2017) Fox, Roy, Krishnan, Sanjay, Stoica, Ion, & Goldberg, Ken. 2017. Multi-level discovery of deep options. arXiv preprint arXiv:1703.08294.
- Fujimoto et al. (2018) Fujimoto, Scott, Van Hoof, Herke, & Meger, David. 2018. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477.
- Galashov et al. (2019) Galashov, Alexandre, Jayakumar, Siddhant M, Hasenclever, Leonard, Tirumala, Dhruva, Schwarz, Jonathan, Desjardins, Guillaume, Czarnecki, Wojciech M, Teh, Yee Whye, Pascanu, Razvan, & Heess, Nicolas. 2019. Information asymmetry in KL-regularized RL. arXiv preprint arXiv:1905.01240.
- Goyal et al. (2019a) Goyal, Anirudh, Islam, Riashat, Strouse, Daniel, Ahmed, Zafarali, Botvinick, Matthew, Larochelle, Hugo, Bengio, Yoshua, & Levine, Sergey. 2019a. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint arXiv:1901.10902.
- Goyal et al. (2019b) Goyal, Anirudh, Sodhani, Shagun, Binas, Jonathan, Peng, Xue Bin, Levine, Sergey, & Bengio, Yoshua. 2019b. Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives. arXiv preprint arXiv:1906.10667.
- Guo et al. (2017) Guo, Zhaohan, Thomas, Philip S, & Brunskill, Emma. 2017. Using options and covariance testing for long horizon off-policy policy evaluation. Pages 2492–2501 of: Advances in Neural Information Processing Systems.
- Haarnoja et al. (2017) Haarnoja, Tuomas, Tang, Haoran, Abbeel, Pieter, & Levine, Sergey. 2017. Reinforcement Learning with Deep Energy-Based Policies. arXiv: Learning.
- Haarnoja et al. (2018) Haarnoja, Tuomas, Zhou, Aurick, Hartikainen, Kristian, Tucker, George, Ha, Sehoon, Tan, Jie, Kumar, Vikash, Zhu, Henry, Gupta, Abhishek, Abbeel, Pieter, et al. 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905.
- Harutyunyan et al. (2018) Harutyunyan, Anna, Vrancx, Peter, Bacon, Pierre-Luc, Precup, Doina, & Nowe, Ann. 2018. Learning with options that terminate off-policy. In: Thirty-Second AAAI Conference on Artificial Intelligence.
- Hessel et al. (2019) Hessel, Matteo, Soyer, Hubert, Espeholt, Lasse, Czarnecki, Wojciech, Schmitt, Simon, & van Hasselt, Hado. 2019. Multi-task deep reinforcement learning with popart. Pages 3796–3803 of: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33.
- Huang et al. (2019) Huang, Shiyu, Su, Hang, Zhu, Jun, & Chen, Ting. 2019. SVQN: Sequential Variational Soft Q-Learning Networks. In: International Conference on Learning Representations.
- Igl et al. (2019) Igl, Maximilian, Gambardella, Andrew, He, Jinke, Nardelli, Nantas, Siddharth, N, Böhmer, Wendelin, & Whiteson, Shimon. 2019. Multitask Soft Option Learning. arXiv preprint arXiv:1904.01033.
- Kappen et al. (2012) Kappen, Hilbert J, Gómez, Vicenç, & Opper, Manfred. 2012. Optimal control as a graphical model inference problem. Machine learning, 87(2), 159–182.
- Kingma & Ba (2014) Kingma, Diederik P, & Ba, Jimmy. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Konidaris & Barto (2009) Konidaris, George, & Barto, Andrew G. 2009. Skill discovery in continuous reinforcement learning domains using skill chaining. Pages 1015–1023 of: Advances in neural information processing systems.
- Kulkarni et al. (2016) Kulkarni, Tejas D, Narasimhan, Karthik, Saeedi, Ardavan, & Tenenbaum, Josh. 2016. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Pages 3675–3683 of: Advances in neural information processing systems.
- Levine (2018) Levine, Sergey. 2018. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909.
- Li et al. (2019) Li, Siyuan, Wang, Rui, Tang, Minxue, & Zhang, Chongjie. 2019. Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards. Pages 1407–1417 of: Advances in Neural Information Processing Systems.
- Maaten & Hinton (2008) Maaten, Laurens van der, & Hinton, Geoffrey. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov), 2579–2605.
- McGovern & Barto (2001) McGovern, Amy, & Barto, Andrew G. 2001. Automatic discovery of subgoals in reinforcement learning using diverse density.
- Mnih et al. (2013) Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan, & Riedmiller, Martin. 2013. Playing Atari with Deep Reinforcement Learning. Computer Science.
- Mnih et al. (2015) Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540), 529.
- Oord et al. (2018) Oord, Aaron van den, Li, Yazhe, & Vinyals, Oriol. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
- Openai et al. (2019) Openai, Berner, Christopher, Brockman, Greg, Chan, Brooke, Cheung, Vicki, Debiak, Przemyslaw, Dennison, Christy, Farhi, David, Fischer, Quirin, Hashme, Shariq, et al. 2019. Dota 2 with Large Scale Deep Reinforcement Learning. arXiv: Learning.
- Osa et al. (2019) Osa, Takayuki, Tangkaratt, Voot, & Sugiyama, Masashi. 2019. Hierarchical reinforcement learning via advantage-weighted information maximization. arXiv preprint arXiv:1901.01365.
- Puri et al. (2019) Puri, Nikaash, Verma, Sukriti, Gupta, Piyush, Kayastha, Dhruv, Deshmukh, Shripad, Krishnamurthy, Balaji, & Singh, Sameer. 2019. Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution. In: International Conference on Learning Representations.
- Riemer et al. (2018) Riemer, Matthew, Liu, Miao, & Tesauro, Gerald. 2018. Learning abstract options. Pages 10424–10434 of: Advances in Neural Information Processing Systems.
- Schulman et al. (2017a) Schulman, John, Chen, Xi, & Abbeel, Pieter. 2017a. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440.
- Schulman et al. (2017b) Schulman, John, Wolski, Filip, Dhariwal, Prafulla, Radford, Alec, & Klimov, Oleg. 2017b. Proximal Policy Optimization Algorithms.
- Sharma et al. (2019) Sharma, Archit, Gu, Shixiang, Levine, Sergey, Kumar, Vikash, & Hausman, Karol. 2019. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657.
- Silver et al. (2017) Silver, David, Schrittwieser, Julian, Simonyan, Karen, Antonoglou, Ioannis, Huang, Aja, Guez, Arthur, Hubert, Thomas, Baker, Lucas, Lai, Matthew, Bolton, Adrian, et al. 2017. Mastering the game of go without human knowledge. nature, 550(7676), 354–359.
- Smith et al. (2018) Smith, Matthew, Hoof, Herke, & Pineau, Joelle. 2018. An inference-based policy gradient method for learning options. Pages 4703–4712 of: International Conference on Machine Learning.
- Sutton et al. (1999) Sutton, Richard S, Precup, Doina, & Singh, Satinder. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2), 181–211.
- Van Hasselt et al. (2016) Van Hasselt, Hado, Guez, Arthur, & Silver, David. 2016. Deep reinforcement learning with double q-learning. In: Thirtieth AAAI conference on artificial intelligence.
- Vinyals et al. (2019) Vinyals, Oriol, Babuschkin, Igor, Czarnecki, Wojciech M, Mathieu, Michaël, Dudzik, Andrew, Chung, Junyoung, Choi, David H, Powell, Richard, Ewalds, Timo, Georgiev, Petko, et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 350–354.
- Wang et al. (2019) Wang, Tonghan, Wang, Jianhao, Zheng, Chongyi, & Zhang, Chongjie. 2019. Learning Nearly Decomposable Value Functions Via Communication Minimization. arXiv preprint arXiv:1910.05366.
- Zhang & Whiteson (2019) Zhang, Shangtong, & Whiteson, Shimon. 2019. DAC: The double actor-critic architecture for learning options. Pages 2010–2020 of: Advances in Neural Information Processing Systems.
- Zheng12 et al. (2018) Zheng12, Zhuobin, Yuan, Chun, Lin12, Zhihui, & Cheng12, Yangyang. 2018. Self-adaptive double bootstrapped DDPG.
Appendix A Theory Details
a.1 Graphical Models
The whole trajectory is shown in Figure. 1. Its corresponding distribution is:
where , is a terminal condition function, and is an option choosing policy.
a.2 Derivation of the Optimization Problem
is a constant representing the multiplication of some prior probabilities.
Our optimization process can be defined as continuously shrinking the KL divergence from the optimal strategy, which can be written as
where is a constant which can be ignored while optimizing policies to maximize or minimize . Our optimization problem can be further simplified to:
a.3 Relationship among Backward Messages
The relationship among , and is:
where is the prior option choosing policy and can be assumed as a uniform distribution over the set of option.
where is the prior action choosing policy and can be assumed as a uniform distribution over the set of action.
a.4 Proof of Lemma 1
We assume the prior option choosing policy is equally probable in all possible values. To simplify our formulation, we assume the value of is one no matter what is. This might cause the estimated V function to be a multiple of the actual V function. Our optimal option choosing policy and optimal action choosing policy have the softmax form. So this multiple form error will not lead to changes in the optimal policies. In addition, we believe a sophisticated alpha can offset the deviation. Based on assumptions above, the V function can be written as: