Deep reinforcement learning (RL) has advanced many control domains, including dexterous object manipulation (Akkaya et al., 2019; Andrychowicz et al., 2018; Nagabandi et al., 2019), agile locomotion (Tan et al., 2018) and navigation (Faust et al., 2018; Held et al., 2018). Despite these successes, several key challenges remain. Stuart Russell phrases one of these challenges eloquently in Russell (2019): “Intelligent behavior over long time scales requires the ability to plan and manage activity hierarchically, at multiple levels of abstraction” and “the main missing piece of the puzzle is a method for constructing the hierarchy of abstract actions in the first place”. If achieved, this capability would be “The most important step needed to reach human-level AI” Russell (2019). This challenge is particularly daunting when temporally extended tasks are combined with sparse binary rewards. In this case the agent does not receive any feedback from the environment while having to decide on a complex course of action, and receives a non-zero reward only after having fully solved the task.
Low sample efficiency is another challenge. In the absence of demonstrations, model-free RL agents require many interactions with the environment to converge to a satisfactory policy (Akkaya et al., 2019). We argue in this paper that both challenges can be addressed by learning models and compositional neural programs. In particular, planning with a learned internal model of the world reduces the amount of necessary interactions with the environment. By imagining likely future scenarios, an agent can avoid making mistakes in the real environment and instead find a sound plan before acting on the environment.
Many real-world tasks are naturally decomposed into hierarchical structures. We hypothesize that learning a variety of skills which can be reused and composed to learn more complex skills is key to tackling long-horizon sparse reward tasks in a sample efficient manner. Such compositionality, formalised by hierarchical RL (HRL), enables agents to explore in a temporally correlated manner, improving sample efficiency by reusing previously trained lower level skills. Unfortunately, prior studies in HRL typically assume that the hierarchy is given, or learn very simple forms of hierarchy in a model-free manner.
We propose a novel method, AlphaNPI-X, to learn programmatic policies which can perform hierarchical planning at multiple levels of abstraction in sparse reward continuous control problems. We first train low-level atomic policies that can be recomposed and re-purposed, represented by a single
goal-conditioned neural network. We leverage off-policy reinforcement learning with hindsight experience replay(Andrychowicz et al., 2017) to train these efficiently. Next, we learn a transition model over the effects of these atomic policies, to imagine likely future scenarios, removing the need to interact with the real environment. Lastly, we learn recursive compositional programs, which combine low-level atomic policies at multiple levels of hierarchy, by planning over the learnt transition models, alleviating the need to interact with the environment. This is made possible by extending the AlphaNPI algorithm Pierrot et al. (2019) which applies AlphaZero-style planning (Silver et al., 2017) in a recursive manner to learn recombinable libraries of symbolic programs.
We show that our agent can learn to successfully combine skills hierarchically to solve challenging robotic manipulation tasks through look-ahead planning, even in the absence of any further interactions with the environment and where powerful model-free baselines struggle to get off the ground.111Videos of agent behaviour are available at: https://sites.google.com/view/alphanpix
2 Related Work
Our work is motivated by the central concern of constructing a hierarchy of abstract actions to deal with multiple challenging continuous control problems with sparse rewards. In addition, we want these actions to be reused and recomposed across different levels of hierarchy through planning with a focus on sample efficiency. This brings several research areas together, namely multitask learning, hierarchical reinforcement learning (HRL) and model-based reinforcement learning (MBRL).
As we have shown, learning continuous control from sparse binary rewards is difficult because it requires the agent to find long sequences of continuous actions from very few information. Other works tackling similar block stacking problems Lanier et al. (2019); Li et al. (2019) have either used a very precisely tuned curriculum, auxiliary tasks or reward engineering to succeed. In contrast, we show that by relying on planning even in absence of interactions with the environment we can successfully learn from raw reward signals.
Multitask learning and the resulting transfer learning challenge have been extensively studied across a large variety of domains, ranging from supervised problems, to goal-reaching in robotics or multi-agent settings(Taylor and Stone, 2009; Zhang and Yang, 2017). A common requirement is to modify an agent’s behaviour depending on the currently inferred task. Universal value functions (Schaul et al., 2015) have been identified as a general factorized representation which is amenable to goal-conditioning agents and can be trained effectively (Barreto et al., 2019), when one has access to appropriately varied tasks to train from. Hindsight Experience Replay (her) (Andrychowicz et al., 2017)
helps improve the training efficiency by allowing past experience to be relabelled retroactively and has shown great promises in variety of domains. However, using a single task conditioning vector has its limitations when addressing long horizon problems. Other works explore different ways to break long-horizon tasks into sequences of sub-goals(Nair and Finn, 2019; Bharadhwaj et al., 2020), or leverage her and curiosity signals to tackle similar robotic manipulation tasks (Lanier et al., 2019). In our work, we leverage her to train low-level goal-conditioned policies, but additionally learn to combine them sequentially using a program-guided meta-controller.
The problem of learning to sequence sub-behaviours together has been well-studied in the context of HRL (Sutton et al., 1999; Bacon et al., 2016) and has seen consistent progress throughout the years (Levy et al., 2017, 2018; Vezhnevets et al., 2017; Nachum et al., 2018). Learning both which sub-behaviours could be useful and how to combine them still remains a hard problem, which is often addressed by pre-learning sub-behaviours using a variety of signals (e.g. rewards (Barreto et al., 2019), demonstrations (Gupta et al., 2019), state-space coverage (Eysenbach et al., 2018; Pong et al., 2019; Lee et al., 2019; Islam et al., 2019), empowerment (Gregor et al., 2016), among many others). Our work assumes we know that some sub-behaviours can be generically useful (like stacking or moving blocks (Riedmiller et al., 2018)). We learn them using goal-conditioned policies as explained above, but we expand how these can be combined by allowing sub-behaviours to call other sub-behaviours, enabling complex hierarchies of behaviours to be discovered. We share this motivation with (Levy et al., 2019) which explores learning policies with multiple levels of hierarchy.
Our work also relates to frameworks which decompose the value function of an MDP into combination of smaller MDPs, such as MAXQ (Dietterich, 2000) or the factored MDP formulation Guestrin et al. (2003), and more recent work which leverage natural language to achieve behaviour compositionality and generalisation (Jiang et al., 2019; Andreas et al., 2017). As our meta-controller uses mcts with a learned model, this also relates to advances in MBRL (Nagabandi et al., 2018; Nasiriany et al., 2019; Schrittwieser et al., 2019; Sekar et al., 2020). Our work extends this to the use of hierarchical programmatic skills. This closely connects to model-based HRL research area which has seen only limited attention so far (Silver and Ciosek, 2012; Nasiriany et al., 2019; Illanes et al., 2020).
. NPI learns a library of program embeddings that can be recombined using a core recurrent neural network that learns to interpret arbitrary programs. AlphaNPI augments NPI with AlphaZero(Silver et al., 2016, 2017) style mcts search and RL applied to symbolic tasks such as Tower of Hanoi and sorting. We extend AlphaNPI to challenging continuous control domains and learn a transition model over programs instead of using the environment. Our work shares similar motivation with (Xu et al., 2017) where NPI is applied to solve different robotic tasks, but we do not require execution traces or pre-trained low-level policies.
3 Problem Definition
In this paper, we aim to learn libraries of skills to solve a variety of tasks in continuous action domains with sparse rewards. Consider the task shown in Figure 1 where the agent’s goal is to take the environment from its initial state where four blocks are randomly placed to the desired final state where blocks are in their corresponding coloured zones and stacked on top of each other. We formalize skills and their combinations as programs. An example programmatic trace for solving this task is shown where the sequence of programs are called to take the environment from the initial state to the final rewarding state. We specify two distinct types of programs: Atomic programs (shown in green) are low-level goal-conditioned policies which take actions in the environment for a fixed number of steps . Non-atomic programs (shown in blue) are a combination of atomic and/or other non-atomic programs, allowing multiple possible levels of hierarchies in behaviour.
|Stack||Two blocks id||Stack a block on another block.|
|Move_To_Zone||Block id & colour||Move a block to a colour zone.|
|Stack_All_Blocks||No arguments||Stack all blocks together in any order.|
|Stack_All_To_Zone||colour||Stack blocks of the same colour in a zone.|
|Move_All_To_Zone||colour||Move blocks of the same colour to a zone.|
|Clean_Table||No arguments||Move all blocks to their colour zone.|
|Clean_And_Stack||No arguments||Stack blocks of the same colour in zones.|
We base our experiments on a set of robotic tasks with continuous action space. Due to the lack of any long-horizon hierarchical multi-task benchmarks, we extended the OpenAI Gym Fetch environment (Brockman et al., 2016) with tasks exhibiting such requirements. We consider a target set of tasks represented by a hierarchical library of programs, see Table 1. These tasks involve controlling a robotic arm with 7-DOF to manipulate 4 coloured blocks in the environment. Tasks vary from simple block stacking to arranging all blocks into different areas depending on their colour. Initial block positions on the table and arm positions are randomized in all tasks. We consider 20 atomic programs that correspond to operating on one block at a time as well as 7 non-atomic programs that require interacting with 2 to 4 blocks. We give the full specifications of these tasks in Sup. Table 3.
We consider a continuous action space , a continuous state space , an initial state distribution and a transition function . The state vector contains the positions, rotations, linear and angular velocities of the gripper and all blocks. More precisely, a state has the form where is the position of block at time step and contains additional information about the gripper and velocities.
More formally, we aim to learn a set of programs . A program is defined by its pre-condition which assesses whether the program can start and its post-condition which corresponds to the reward function here. Each program is associated to an MDP which can start only in states such that the pre-condition is satisfied, and where is a reward function that outputs when the post-condition is satisfied and 0 otherwise.
Atomic programs are represented by a goal-conditioned neural network with continuous action space. Non-atomic programs use a modified action space: we replace the original continuous action space by a discrete action space Stop where actions call programs and the Stop action enables the current program to terminate and return the execution to its calling program. Atomic programs don’t have a Stop action, they terminate after time steps.
AlphaNPI-X learns to solve multiple tasks by composing programs at different levels of hierarchy. Given a one-hot encoding of a program and states from the environment, the meta-controller calls either an atomic program, a non-atomic program or theStop action. In the next sections, we will describe how learning and inference is performed.
4.1 Training AlphaNPI-X
Learning in our system operates in three stages: first we learn the atomic programs, then we learn a transition model over their effect and finally we train the meta-controller to combine them. We provide detailed explanation of how these modules are learned below.
Learning Atomic Programs
Our atomic programs consist of two components: a) a goal-setter that, given the atomic program encoding and the current environment state, generates a goal vector representing the desired position of the blocks in the environment and b) a goal-conditioned policy that gets a “goal” as a conditioning vector, produces continuous actions and terminates after T time steps.
To execute an atomic program from an initial state satisfying the pre-condition , the goal-setter, , computes the goal : . We then roll the goal-conditioned policy for time steps to achieve the goal. In this work, the goal setter module is provided which translates the atomic program specification into the corresponding goal vector indicating the desired position of the blocks which is also used to compute rewards.
We parametrize our shared goal-conditioned policy using Universal Value Function Approximators (uvfas) (Schaul et al., 2015). uvfa
s estimate a value function that does not only generalise over states but also goals. To accelerate training of this goal-conditioneduvfa, we leverage the "final" goal relabeling strategy introduced in her (Andrychowicz et al., 2017). Past episodes of experience are relabelled retroactively with goals that are different from the goal aimed for during data collection and instead correspond to the goal achieved in the final state of the episode. The mappings from state vector to goals is simply done via extracting the blocks positions directly from the state vector . To deal with continuous control, we arbitrarily use ddpg (Lillicrap et al., 2015) to train the goal-conditioned policy.
More formally, we define a goal space which is a subspace of the state space . The goal-conditioned policy takes as inputs a state as well as a goal . We define the function that extracts the goal from a state . This policy is trained with the reward function defined as where . The goal setter as takes an initial state and an atomic program index and returns a goal such that . We can thus express any atomic program policy as
We first train the policy with HER to achieve goals sampled uniformly in from any state sampled from
. However, the distribution of initial states encountered by each atomic program may be very different when executing programs sequentially (as will happen when non-atomic programs are introduced). Thus, after the initial training, we continue training the policy, but with probabilitywe do not reset the environment between episodes, to approximate the real distribution of initial states which happens when atomic programs are chained together. Hence, the initial state is either sampled randomly or kept as the last state observed in the previous episode. We later show our empirical results and analysis regarding both training phases.
Learning Self-Behavioural Model
After learning a set of atomic programs we learn a transition model over their effects: , parameterized with a neural network. As Figure 2-B shows, this module takes as input an initial state and an atomic program index and the output is the prediction of the environment state obtained when rolling the policy associated to this program for time steps from the initial state. We use a fully connected MLP and train it by minimizing the mean-squared error to the ground-truth final states. This enables our model to make jumpy predictions over the effect of executing an atomic program during search, hence removing the need for any interactions with the environment during planning. Learning this model enables us to imagine the state of the environment following an atomic program execution and hence to avoid any further calls to atomic programs that would each have to perform many actions in the environment.
Learning the Meta-Controller
In order to compose atomic programs together into hierarchical stacks of non-atomic programs, we use a meta-controller inspired by AlphaNPI (Pierrot et al., 2019). The meta-controller interprets and selects the next program to execute using neural-network guided Monte Carlo Tree Search (MCTS) (Silver et al., 2017; Pierrot et al., 2019), conditioned on the current program index and states from the environment (see Figure 2 for an overview of a node expansion).
We train the meta-controller using the recursive mcts strategy introduced in AlphaNPI (Pierrot et al., 2019): during search, if the selected action is non-atomic, we recursively build a new Monte Carlo tree for that program, using the same state . See Supp. Section B for a detailed description of the search process and pseudo-code.
In AlphaNPI (Pierrot et al., 2019), similar to AlphaZero, during the tree search, future scenarios were evaluated by leveraging the ground-truth environment, without any temporal abstraction. Instead in this work, we do not use the environment directly during planning, but replace it by our learnt transition model over the effects of the atomic programs, the self-behavioural model described in Section 4.1, resulting in a far more sample efficient algorithm. Besides, AlphaNPI used a hand-coded curriculum where the agent gradually learned from easy to hard programs. Instead here, we randomly sample at each iteration a program to learn from, removing the need for any supervision and hand-crafting of a curriculum. Inspired by recent work (Colas et al., 2018), we also implemented an automated curriculum strategy based on learning progress, however we found that it does not outperform random sampling (see more detailed explanations and results in Supp. Section B).
4.2 Inference with AlphaNPI-X
To infer with AlphaNPI-X, we compare three different inference strategies: 1) Rely only on the policy network, without planning. 2) Plan a whole trajectory using the learned transition model (i.e. self-behavioural model) and execute it fully (i.e. open-loop planning). 3) Use a receding horizon for planning inspired by Model Predictive Control (MPC) (Garcia et al., 1989). During execution, observed states can diverge from the predictions made by the self-behavioural model due to distribution shift, especially for long planning horizons, which deteriorates performance. To counter that, we do not commit to a plan for the full episode, but instead re-plan after executing any atomic program (non-atomic programs do not trigger re-planning). The comparison between these inference methods can be found in Table 2. We also provide a detailed example in Supp. Section C.
5 Experiments and Results
We first train the atomic policies using ddpg with her relabelling described in Section 4.1. Initial block positions on the table as well as gripper position are randomized. Additional details regarding the experimental setup is provided in Supp. Section D
. In the first phase of training, we train the agent for 100 epochs during uniform goal sampling. In the second phase we continue training for 150 epochs where we change the sampling distribution to allow no resets between episodes to mimic the desired setting where skills are sequentially executed.
We evaluate the agent performance on goals corresponding to single atomic programs as well as two multi-step sequential atomic program executions, MultiStep-Moving which requires 4 consecutive calls to move_to_zone and MultiStep-Stacking which requires 3 consecutive calls to stack (see Figure 4). We observe that while the agent obtains decent performance on executing atomic programs in the first phase, the second phase is indeed crucial to ensure success of the sequential execution of the programs. We also observe that the reset probability affects the performance and provides the better trade-off between the asymptotic training performance and the agent ability to execute programs sequentially. More details are included in Supp. Section E.
We then train the self-behavioural model to predict the effect of atomic programs, using three datasets respectively made of 10k, 50k and 100k episodes played with the goal-conditioned policy. As during the second phase of training, we did not reset the environment between two episodes with a probability . We trained the self-behavioural model for 500 epochs on each data set. Figure 4 shows the performance aggregated across different variations of stack and move_to_zone behaviours.
To train the meta-controller, we randomly sample from the set of non-atomic programs during training. In Figure-5, we report the performance on all non-atomic tasks during the evolution of training. We also investigated the use of an automated curriculum via learning progress but observed that it does not outperform random sampling (please see Supp. Section B.3 for details). After training for epochs, we evaluate the AlphaNPI-X agent on each non-atomic program in the library shown in Table 2. Our results indicate that removing planning significantly reduces performance, especially for stacking 4 blocks where removing planning results in complete failure. We observe that re-planning during inference significantly helps improve the agent’s performance overall.
DDPG + HER
We compare our method to two baselines to illustrate the difficulty for standard RL methods to solve tasks with sparse reward signals. First, we implemented a multitask ddpg (m-ddpg) that takes as input the environment state and a one hot encoding of a non-atomic program. At each iteration, m-ddpg selects randomly a program index and plays one episode in the environment. Second, we implemented a m-ddpg + her agent which leverages a goal setter for non-atomic tasks described in Section 4.1. This non atomic programs goal setter is not available to AlphaNPI-X. This agent has more information than its m-ddpg counterpart. Instead of receiving a program index, it receives a goal vector representing the desired end state of the blocks. As in her, goals are relabelled during training. We observe that m-ddpg is unable to learn any non-atomic program. It is also the case for m-ddpg + her despite having access to the additional goal representations. This shows that standard exploration mechanisms in model-free agents such as in ddpg, where Gaussian noise is added to the actions, is very unlikely to lead to rewarding sequences and hence learning is hindered.
In this paper, we proposed AlphaNPI-X, a novel method for constructing a hierarchy of abstract actions in a rich object manipulation domain with sparse rewards and long horizons. Several ingredients proved critical in the design of AlphaNPI-X. First, by learning a self-behavioural model, we can leverage the power of recursive AlphaZero-style look-ahead planning across multiple levels of hierarchy, without ever interacting with the real environment. Second, planning with a receding horizon at inference time resulted in more robust performance. We also observed that our approach does not require a carefully designed curriculum commonly used in NPI so random task sampling can be simply used instead. Experimental results demonstrated that AlphaNPI-X, using an abstract imagination-based reasoning, can simultaneously solve multiple complex tasks involving dexterous object manipulation beyond the reach of model-free methods.
A limitation of our work is that similar to AlphaNPI, we pre-specified a hierarchical library of the programs to be learned. While it enables human interpretability, this requirement is still quite strong. Thus, a natural next step would be to extend our algorithm to discover these programmatic abstractions during training so that our agent can specify and learn new skills hierarchically in a fully unsupervised manner.
Our paper presents a sample efficient technique for learning in sparse-reward temporally extended settings. It does not require any human demonstrations and can learn purely from sparse reward signals. Moreover, after learning low-level skills, it can learn to combine them to solve challenging new tasks without any requirement to interact with the environment. We believe this has a positive impact in making reinforcement learning techniques more accessible and applicable in settings where interacting with the environment is costly or even dangerous such as robotics. Due to the compositional nature of our method, the interpretability of the agent’s policy is improved as high-level programs explicitly indicate the intention of the agent over multiple steps of behaviour. This participates in the effort of building more interpretable and explainable reinforcement learning agents in general. Furthermore, by releasing our code and environments we believe that we help efforts in reproducible science and allow the wider community to build upon and extend our work in the future.
- Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113. Cited by: §1, §1.
Modular multitask reinforcement learning with policy sketches.
Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 166–175. Cited by: §2.
- Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177. Cited by: §1.
- Hindsight experience replay. CoRR abs/1707.01495. External Links: Cited by: Appendix C, §E.4, §1, §2, §4.1.
- The option-critic architecture. CoRR abs/1609.05140. External Links: Cited by: §2.
- The option keyboard: combining skills in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 13031–13041. Cited by: §2, §2.
- Dynamics-aware latent space reachability for exploration in temporally-extended tasks. arXiv preprint arXiv:2005.10934. Cited by: §2.
- OpenAI gym. CoRR abs/1606.01540. External Links: Cited by: §D.1, §3.
- CURIOUS: intrinsically motivated multi-task, multi-goal reinforcement learning. CoRR abs/1810.06284. External Links: Cited by: §B.3, §E.4, §4.1.
Hierarchical reinforcement learning with the maxq value function decomposition.
Journal of artificial intelligence research13, pp. 227–303. Cited by: §2.
- Diversity is all you need: learning skills without a reward function. arXiv preprint arXiv:1802.06070. Cited by: §2.
- PRM-rl: long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5113–5120. Cited by: §1.
- Model predictive control: theory and practice—a survey. Automatica 25 (3), pp. 335–348. Cited by: §4.2.
- Variational intrinsic control. arXiv preprint arXiv:1611.07507. Cited by: §2.
- Efficient solution algorithms for factored mdps. Journal of Artificial Intelligence Research 19, pp. 399–468. Cited by: §2.
- Relay policy learning: solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956. Cited by: §2.
- Automatic goal generation for reinforcement learning agents. arXiv preprint arXiv:1705.06366. Cited by: §1.
- Symbolic plans as high-level instructions for reinforcement learning. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 30, pp. 540–550. Cited by: §2.
- Marginalized state distribution entropy regularization in policy optimization. arXiv preprint arXiv:1912.05128. Cited by: §2.
- Language as an abstraction for hierarchical deep reinforcement learning. CoRR abs/1906.07343. External Links: Cited by: §2.
- Curiosity-driven multi-criteria hindsight experience replay. arXiv preprint arXiv:1906.03710. Cited by: §E.4, §2, §2.
- Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274. Cited by: §2.
- Hierarchical actor-critic. arXiv preprint arXiv:1712.00948. Cited by: §2.
- Hierarchical reinforcement learning with hindsight. arXiv preprint arXiv:1805.08180. Cited by: §2.
- Hierarchical reinforcement learning with hindsight. In International Conference on Learning Representations, Cited by: §2.
- Towards practical multi-object manipulation using relational reinforcement learning. arXiv preprint arXiv:1912.11032. Cited by: §2.
- Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. Cited by: §4.1.
- Data-efficient hierarchical reinforcement learning. CoRR abs/1805.08296. External Links: Cited by: §2.
- Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7559–7566. Cited by: §2.
- Deep dynamics models for learning dexterous manipulation. arXiv preprint arXiv:1909.11652. Cited by: §1.
Hierarchical foresight: self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint arXiv:1909.05829. Cited by: §2.
- Planning with goal-conditioned policies. In Advances in Neural Information Processing Systems, pp. 14814–14825. Cited by: §2.
- Learning compositional neural programs with recursive tree search and planning. In Advances in Neural Information Processing Systems 32, pp. 14646–14656. Cited by: Figure 6, §1, §2, §4.1, §4.1, §4.1.
- Skew-fit: state-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698. Cited by: §2.
- Neural programmer-interpreters. arXiv preprint arXiv:1511.06279. Cited by: §2.
- Learning by playing - solving sparse reward tasks from scratch. CoRR abs/1802.10567. External Links: Cited by: §2.
- Human compatible: artificial intelligence and the problem of control. Penguin Publishing Group. External Links: Cited by: §1.
- Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning, F. Bach and D. Blei (Eds.), Proceedings of Machine Learning Research, Vol. 37, Lille, France, pp. 1312–1320. Cited by: §2, §4.1.
- Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265. Cited by: §2.
- Planning to explore via self-supervised world models. arXiv preprint arXiv:2005.05960. Cited by: §2.
- Compositional planning using optimal option models. arXiv preprint arXiv:1206.6473. Cited by: §2.
- Mastering the game of go with deep neural networks and tree search. nature 529 (7587), pp. 484. Cited by: §2.
- Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. Cited by: §1, §2, §4.1.
- Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artificial intelligence 112 (1-2), pp. 181–211. Cited by: §2.
- Sim-to-real: learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10332. Cited by: §1.
- Transfer learning for reinforcement learning domains: a survey. Journal of Machine Learning Research 10 (Jul), pp. 1633–1685. Cited by: §2.
- Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.01161. Cited by: §2.
- Neural Task Programming: Learning to Generalize Across Hierarchical Tasks. arXiv e-prints, pp. arXiv:1710.01813. External Links: Cited by: §2.
- A survey on multi-task learning. arXiv preprint arXiv:1707.08114. Cited by: §2.
Appendix A Programs library
Our program library includes 20 atomic programs and 7 non-atomic ones. The full list of these programs can be viewed in Table 3. In the AlphaNPI paper, the program library contained only five atomic programs and 3 non-atomic programs. Thus, the branching factor in the tree search in AlphaNPI-X is on average much greater than in the AlphaNPI paper. Furthermore, in this work, these atomic programs are learned and therefore might not always execute as expected while in AlphaNPI the five atomic programs are hard-coded in the environment and thus execute successfully anytime they are called.
While in this study we removed the need for a hierarchy of programs during learning, we still defined programs levels for two purposes: (i) to control the dependencies between programs, programs of lower levels cannot call programs of higher levels; (ii) to facilitate the tree search by relying on the level balancing term introduced in the original AlphaNPI P-UCT criterion. In this context, we defined Clean_Table and Clean_And_Stack as level 2 programs, Stack_All_Blocks, Move_All_To_Zone and Move_All_To_Zone as level 1 programs. The atomic programs are defined as level 0 programs. Interestingly, the natural order learned during training (when sampling random program indices to learn from) matches this hierarchy, see Figure 11.
|Stack_0_1||Stack block number 0 on block number 1.|
|Stack_0_2||Stack block number 0 on block number 2.|
|Stack_0_3||Stack block number 0 on block number 3.|
|Stack_1_0||Stack block number 1 on block number 0.|
|Stack_1_2||Stack block number 1 on block number 2.|
|Stack_1_3||Stack block number 1 on block number 3.|
|Stack_2_0||Stack block number 2 on block number 0.|
|Stack_2_1||Stack block number 2 on block number 1.|
|Stack_2_3||Stack block number 2 on block number 3.|
|Stack_3_0||Stack block number 3 on block number 0.|
|Stack_3_1||Stack block number 3 on block number 1.|
|Stack_3_2||Stack block number 3 on block number 2.|
|Move_To_Zone_0_ORANGE||Move block number 0 to orange zone.|
|Move_To_Zone_1_ORANGE||Move block number 1 to orange zone.|
|Move_To_Zone_2_ORANGE||Move block number 2 to orange zone.|
|Move_To_Zone_3_ORANGE||Move block number 3 to orange zone.|
|Move_To_Zone_0_BLUE||Move block number 0 to blue zone.|
|Move_To_Zone_1_BLUE||Move block number 1 to blue zone.|
|Move_To_Zone_2_BLUE||Move block number 2 to blue zone.|
|Move_To_Zone_3_BLUE||Move block number 3 to blue zone.|
|Stack_All_To_Zone_ORANGE||Stack orange blocks in the orange zone.|
|Stack_All_To_Zone_BLUE||Stack blue blocks in the blue zone.|
|Move_All_To_Zone_ORANGE||Move orange blocks to the orange zone.|
|Move_All_To_Zone_BLUE||Move blue blocks to the orange zone.|
|Stack_All_Blocks||Stack all blocks together in any order.|
|Clean_Table||Move all blocks to their colour zone.|
|Clean_And_Stack||Stack blocks of the same colour in zones.|
Appendix B Details of the AlphaNPI-X method
b.1 Table of symbols
Here we provide the list of symbols used in our method section:
|Action space||is continuous|
|State space||is continuous|
|Goal space||is a sub-space of|
|Initial state distribution|
|Reward obtained at time|
|Program||If then is atomic|
|Program index selected at time|
|her mapping function||extract a goal from a state vector|
|Goal conditioned policy|
Here we provide some further details on the AlphaNPI method, which we use and extend for our meta-controller. The AlphaNPI agent uses a recursion augmented Monte Carlo Tree Search (MCTS) algorithm to learn libraries of hierarchical programs from sparse reward signals. The tree search is guided by an actor-critic network inspired from the Neural Programmers-Interpreter (NPI) architecture. A stack is used to handle programs as in a standard program execution: when a non-atomic program calls another program, the current NPI network’s hidden state is saved in a stack and the next program execution starts. When it terminates, the execution goes back to the previous program and the network gets back its previous hidden state. The mechanism is also applied to MCTS itself: when the current search decides to execute another program, the current tree is saved along with the network’s hidden state in the stack and a new search starts to execute the desired program.
The neural network takes as input an environment state and a program index and returns a vector of probabilities over the next programs to call, , as well as a prediction for the value . The network is composed of five modules. An encoder encodes the environment state into a vector and a program embedding matrix contains learnable embedding for each non-atomic programs. The -th row of the matrix contains the embedding of the program referred by the index such that . An LSTM core takes the encoded state and the program embedding as input and returns its hidden state . Finally, a policy head and a value head take the hidden state as input and return respectively and , see Figure 6.
The guided MCTS process is used to generate data to train the AlphaNPI neural network. A search returns a sequence of transitions where corresponds to the tree policy at time and is the final episode reward. The tree policy is computed from the tree nodes visit counts. These transitions are stored in a replay buffer. At training time, batches of trajectories are sampled and the network is trained to minimize the loss
Note that in AlphaNPI-X, in contrast with the original AlphaNPI work, proper back-propagation through time (BPTT) over batches of trajectories is performed instead of gradient descent over batches of transitions, resulting in a more stable updates.
AlphaNPI can explore and exploit. In exploration mode, many simulations per node are performed, Dirichlet noise is added to the priors to help exploration and actions are chosen by sampling the tree policy vectors. In exploitation mode, significantly less simulations are used, no noise is added and actions are chosen by taking the argmax of the tree policy vectors. The tree is used in exploration mode to generate data for training and in exploitation mode at inference time.
b.3 Improvements over AlphaNPI
Distributed training and BPTT
We improved AlphaNPI training speed and stability in distributing the algorithm. We use 10 actors in parallel. Each actor has a copy of the neural networks weights. It uses it to guide its tree searches to collect experiences. In each epoch, the experience collected by all the actors is sent to a centralized replay buffer. A learner, that has a copy of the network weights, samples batches in the buffer, computes the losses and performs gradient descent to update the network weights. When the learner is done, it sends the new weights to all the actors. We use the MPI (Message Passing Interface) paradigm through its python package MPI4py to implement the processes in parallel. Each actor uses 1 CPU.
Another improvement over the standard AlphaNPI is leveraging back-propagation through time (BPTT). The AlphaNPI agent uses an LSTM. During tree search, the LSTM states are stored inside the tree nodes. However, in the original AlphaNPI, transitions are stored independently and gradient descents are performed on batches of uncorrelated transitions. While it worked on the examples presented in the original paper, we found that implementing BPTT improves training stability.
At each training iteration, the agent selects the programs to train on. In the original AlphaNPI paper, the agents select programs following a hard-coded curriculum based on program complexity. In this work, we select programs randomly instead and thus remove the need for extra supervision. We also implemented an automated curriculum learning paradigm based on learning progress signal Colas et al.  and observed that it does not improve over random sampling. We compare both strategies in Figure 5. We detail below the learning progress based curriculum we used as a comparison.
The curriculum based on the program levels from the original AlphaNPI might be replaced by a curriculum based on learning progress. The agent focuses on programs for which its learning progress is maximum. This strategy requires less hyper-parameters and enables the agent to discover the same hierarchy implicitly. The learning progress for a program is defined as the derivative over time of the agent performance on this program. At the beginning of each training iteration , the agent attempts every non atomic program times in exploitation mode. We note the average performance, computed as mean reward over the episodes, on program at iteration . Rewards are still assumed to be binary: 0 or 1. We compute the learning progress for program as
where the absolute derivative over time of the agent performance is approximated as the first order. Finally, we compute the probability of choosing to train on the program at iteration as
is an hyperparameter andis the total number of non atomic programs. The term is used to balance exploration and exploitation, it ensures that the agent tries all programs with a non-zero probability.
Goal policy training
Self-behavioural model training
Appendix C Example execution of the programs
Let us imagine we want to run the atomic program Stack_1_2. The state space is and the goal space is . A state contains all positions, angles, velocities and angular velocities of the blocks and of the robot articulations at time . A goal is the concatenation of desired final positions for each block. Let us consider atomic program Stack_1_2 has index 0. In this case, we compute the goal corresponding to this program as where is the block radius. This goal corresponds to positions where block 1 is stacked on block 2 and where other blocks have not been moved. The pre-condition of this program verifies in state that no other block is already on top of block so that stacking is possible. If the pre-condition is verified, the goal policy is rolled for time steps to execute this goal. When the goal policy terminates, the program post-condition is called on the final state and verifies for all blocks that their final position is in a sphere of radius around their expected final position described by goal . The value we use for is the standard value used in her [Andrychowicz et al., 2017].
We train the goal-conditioned policy so that it can reach goals that corresponds to the atomic programs for any initial position, as long as the initial position verifies the atomic program pre-condition. Once trained, we learn its self behavioural model . This model takes an environment state and an atomic program index and predicts the environment state obtained when the goal policy has been rolled for timesteps conditioned on goal and initial environment state. Thus, this model performs jumpy predictions, it does not capture instantaneous effects but rather global effects such as block has been moved, block has remained at its initial position.
When the model has been trained, we use it to learn non-atomic programs with our extended AlphaNPI. Let us imagine that we want to execute the Clean_And_Stack program. This program is expected to stack, in any order, both orange blocks in the orange zone and both blue blocks in the blue zone. Its pre-condition is always true since it can be called from any initial position. Its post-condition looks at a state and returns if it finds an orange block with its center of gravity in the orange zone and with an orange block stacked in top of it. It verifies the same for the blue blocks. The post-condition returns otherwise. The stacking test is performed as explained above looking at a sphere of radius around an ideal position.
A possible execution trace for this program is:
In this trace, the non-atomic programs are shown in blue and the atomic programs are shown in green. Non-atomic program execution happens as in the original AlphaNPI. The execution of an atomic program happens as described before: when the atomic program is called, the current environment state is transformed into a goal vector that is realised by the goal-conditioned policy. In the trace above, the goal policy has been called times with different goal vectors.
Appendix D Experimental setting
d.1 Experiment setup
We base our experiments on a set of robotic tasks with continuous action space. Due to the lack of any long-horizon hierarchical multi-task benchmarks, we extended the OpenAI Gym Fetch environment Brockman et al.  with tasks exhibiting such requirements. We reused the core of the Pick And Place environment and added two colored zones as well as four colored blocks that are indexed. We did not modify the environment physics. The action space is , it corresponds to continuous command to the arm gripper. In this setting, the observation space is the same as the state space and contains all blocks and robot joint positions together with their linear and angular velocities. It is of dimension , with blocks ordered in the observations removing the need to provide their colour or index. More precisely, a state has the form where is the position of block at time step and contains additional information about the gripper and velocities. When executing atomic programs, we assume that we always aim to move the block which is at the first position in the observation vector. Then, to move any block , we apply a circular permutation to the observation vector such that block arrives at the first position in the observation.
We consider two different initial state distributions in the environment. During goal-conditioned policy training, the block at the first position in the observation vector has a probability to start in the gripper, a probability to start under the gripper and a probability to start elsewhere uniformly on the table. All other blocks may start anywhere on the table. Once the first block is positioned, we place the other blocks in a random order. When a block is placed, it has a probability to be placed on top of another block and to be placed on the table. In this case, its position is sampled uniformly on the table until there is no collision. During training, we anneal these probabilities. By the end of training all blocks are always initialised at random positions on the table.
d.2 Computational resources
We ran all experiments on a 30 CPU cores. We used 28 CPUs to train the goal-conditioned policy and 10 CPU cores to train the meta-controller. We trained the goal-conditioned policy for 2 days (counting both phases) and the meta-controller for also 2 days.
We provide all hyper-parameters used in our experiments for our method together with baselines.
|DDPG actor||hidden layers size||256/256|
|DDPG critic||hidden layers size||256/256|
|batch size||batch size (number of transitions)||256|
|actor learning rate|
|critic learning rate|
|replay_k||ratio to resample transitions in HER||4|
|number of actors used in parallel||28|
|number of cycles per epoch||40|
|number of updates per cycle||40|
|buffer size||replay buffer size (number of transitions)|
|number of epochs for phase 1||100|
|number of epochs for phase 2||150|
|Model layers||hidden layers size||512/512|
|Dataset size||number of episodes collected to create dataset||50000|
|number of epochs||500|
|Observation Module||hidden layer size||128|
|program embedding dimension||256|
|LSTM hidden state dimension||128|
|observation encoding dimension||128|
|discount factor to penalize long traces reward||0.97|
|number of simulations in the tree in exploration mode||100|
|number of simulations in the tree in exploitation mode||5|
|number of gradient descent per episode played||2|
|number of episodes at each iteration||20|
|number of episodes for validation||10|
|number of iterations||700|
|coefficient to encourage choice of higher level programs||3.0|
|number of actors used in parallel||10|
|coefficient to balance exploration/exploitation in mcts||0.5|
|batch size||batch size (number of trajectories)||16|
|maximum size of buffer memory (nb of full trajectories)||100|
|probability to draw positive reward experience in buffer||0.5|
|tree policies temperature coefficient||1.3|
|AlphaZero Dirichlet noise fraction||0.25|
|AlphaZero Dirichlet distribution parameter||0.03|
Appendix E Additional Results
e.1 Results of Goal Policy Training
We train the goal-conditioned policy in two consecutive phases. In the first phase we sample the initial states and goal-vectors randomly. During the second phase, we change the initial and goal state distributions to ensure that our goal-conditioned policy still performs well when called sequentially by the meta-controller. For initial states, with probability we do not reset the initial state of the environment between episodes. This means that the initial state of some episodes are the final state achieved in the previous episode. To simplify this process in a distributed setting, we store the final states of all episodes in a buffer and in each episode, with probability , we sample a state from this buffer to be our initial state in this episode. Additionally, instead of random goal-vector sampling we randomly select an atomic program , then given the initial state we compute the goal vector using the goal-setter. We study the impact of the reset probability as well as the usefulness of the second phase in Figure 9.
e.2 Results of self-behavioural model training
We represent the self-behavioural model with a 2 layer MLP that takes as input the initial environment state and the atomic program and predicts the environment state after steps. Learning this model enables us to imagine the state of the environment following an atomic program execution and hence to avoid any further calls to atomic programs that would each have to perform many actions in the environment.
To train the model, we play episodes with goal-conditioned policy in the environment on uniformly sampled atomic programs and record initial and final environment states in a dataset. Then, we train the model to minimize its prediction error, computed as a mean square error over this dataset. We study the impact of in Figure 10.
e.3 Curriculum learning comparisons
We compare training the meta-controller with random program sampling compared to an automated curriculum using a learning progress signal detailed in section B.3. We observe in Figure 11 that both random and automated curriculum gives rise to a natural hierarchy for programs. We observe that the MoveAllToZone and StackAllToZone are learned first. The traces generated by AlphaNPI-X after training confirms that these programs are used to execute the more complex programs CleanAndStack and CleanTable. Given the results shown in Figure 11 we cannot conclude that the learning progress based curriculum outperforms random sampling.
e.4 Sample efficiency
We compare our method’s sample efficiency with other works on the Fetch environment. For training the goal-conditioned policy, 28 ddpg actors sample 100 episodes per epoch in parallel. Thus, 2800 episodes are sampled per epoch. In total, we train the agent for 250 epochs resulting in episodes. To train the self-behavioural model, we show that sampling episodes is enough to perform well. Thus, in total we only use episodes to master all tasks. Note that training the meta-controller does not require any interaction with the environment. In comparison, in the her original paper [Andrychowicz et al., 2017], each task alone requires around episodes to be mastered. In curious [Colas et al., 2018], training the agent requires episodes. Finally, in [Lanier et al., 2019] where the agent is trained only to perform the task Stack_All_Blocks, around episodes have been sampled.