Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning

by   Richard Li, et al.
berkeley college

Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements. Many practical tasks require manipulation of multiple objects, and the complexity of such tasks increases with the number of objects. Learning from a curriculum of increasingly complex tasks appears to be a natural solution, but unfortunately, does not work for many scenarios. We hypothesize that the inability of the state-of-the-art algorithms to effectively utilize a task curriculum stems from the absence of inductive biases for transferring knowledge from simpler to complex tasks. We show that graph-based relational architectures overcome this limitation and enable learning of complex tasks when provided with a simple curriculum of tasks with increasing numbers of objects. We demonstrate the utility of our framework on a simulated block stacking task. Starting from scratch, our agent learns to stack six blocks into a tower. Despite using step-wise sparse rewards, our method is orders of magnitude more data-efficient and outperforms the existing state-of-the-art method that utilizes human demonstrations. Furthermore, the learned policy exhibits zero-shot generalization, successfully stacking blocks into taller towers and previously unseen configurations such as pyramids, without any further training.


page 1

page 5

page 6

page 9


Curiosity-Driven Multi-Criteria Hindsight Experience Replay

Dealing with sparse rewards is a longstanding challenge in reinforcement...

Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations

Learning robot manipulation policies through reinforcement learning (RL)...

MaMiC: Macro and Micro Curriculum for Robotic Reinforcement Learning

Shaping in humans and animals has been shown to be a powerful tool for l...

Compositional Multi-Object Reinforcement Learning with Linear Relation Networks

Although reinforcement learning has seen remarkable progress over the la...

Adaptive Curriculum Generation from Demonstrations for Sim-to-Real Visuomotor Control

We propose Adaptive Curriculum Generation from Demonstrations (ACGD) for...

Curriculum Learning with Hindsight Experience Replay for Sequential Object Manipulation Tasks

Learning complex tasks from scratch is challenging and often impossible ...

Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation

It has been a long-standing dream to design artificial agents that explo...

Code Repositories

I Introduction

The main idea in reinforcement learning is to incentivize actions that maximize rewards. Unlike video games, where rewards are readily available, for manipulation tasks, a reward function must be manually constructed. For example, to pick and place a block, the rewards might be inversely proportional to the manipulator’s distance from the block and the block’s distance from the target location. Such rewards that frequently provide information about the task are known as dense rewards. Using dense rewards, an agent can get stuck in a local minimum and never complete the desired task. It is well known that intuitively reasonable reward functions can often result in unexpected or undesirable behaviors [24]. This makes reward design very challenging.

An alternative is to provide the agent with sparse rewards, which may either be provided after the agent completes the overall task (i.e., terminal reward) or extremely intermittently when the agent completes critical steps (i.e., step-wise rewards). Sparse rewards are more straightforward to define than dense rewards. However, because many tasks require execution of a long sequence of actions, sparse rewards drastically complicate the challenges of exploration and credit-assignment. Training with sparse rewards, therefore, either completely fails or requires massive amounts of data.

Practical reinforcement learning systems have sidestepped the challenge of learning from sparse rewards by either using (a) human demonstrations, (b) sim2real transfer, (c) careful environmental instrumentation to simplify the task or (d) meticulous reward shaping. Using these ideas, RL has been applied to wide variety of robotic tasks such as stacking blocks in a tower [12, 48, 37], opening doors [22], flipping pan-cakes [32], hitting a ball, orienting a cube [2] and other dexterous manipulation tasks [45].

Developing data-efficient algorithms that can learn from sparse rewards will alleviate the need for demonstrations and painful reward design. It will consequently open up many application areas, where RL cannot be applied today. Past works have improved learning efficiency of RL algorithms using better optimization methods [26, 23, 51], combining model-based and model-free learning [34], hierarchical learning [53], and design of better exploration methods [50, 36, 39, 41, 52]. A few recent works used the compositional task structure to improve the data efficiency of RL algorithms [63, 61, 42].

In the related field of supervised deep learning, transfer of knowledge by pre-training on a source task followed by finetuning on a target task 

[14, 1, 46] has been very successful in reducing the data requirements. However, in the context of RL, learning from multiple tasks and transferring this knowledge to reduce data requirements for a new task remains an open challenge [65, 9, 29, 38]

. One potential reason for lack of transfer is that learning from a new task exacerbates the already existing problem of credit assignment. The inability to assign credit, in turn, increases the variance in the gradients and consequently results in learning failure 

[20]. One solution is to pace the agent’s learning, where it only gets a new task when it has mastered previous tasks (i.e., curriculum learning [7]).

It turns out that in RL settings, curriculum learning is also not straightforward. To better understand why it is the case, consider solving the problem of stacking multiple blocks into a tower using a curriculum of stacking an increasing number of blocks. Suppose the agent has mastered the skill of stacking two blocks. The introduction of the third block preserves the task structure but changes the distribution of the agent’s input. In the absence of appropriate inductive biases to deal with changes in inputs, the agent resorts to treating the new data distribution as a new learning problem and is unable to leverage its knowledge from past tasks efficiently.

One well-known method to tackle changing data distribution is training with data-augmentation. In RL settings, this idea has translated into domain randomization [55]

. In the running example, training with randomization involves sampling a random number of blocks from a uniform distribution in every episode. However, because, in most episodes, the agent would be tasked to stack multiple blocks, learning in such a situation remains very challenging. This consideration suggests that the major hindrance in learning from a curriculum may not be in the design of the curriculum, but the inability of learning systems to transfer knowledge across the different tasks in the curriculum.

In this work, we show that training a policy represented by a attention based graph neural network (GNN) overcomes the challenges associated with curriculum learning in multi-object manipulation tasks. Our agent learns to stack six or more blocks from scratch (see Figure 

LABEL:fig:fig1). We use a simple curriculum strategy, which increases the number of blocks when the agent masters a target task with a fewer number of blocks. The attention-based GNN complements the curriculum by providing the appropriate inductive bias to transfer knowledge between tasks with a different number of objects. To the best of our knowledge, ours is the first work to solve the problem of stacking six or more blocks using RL and without requiring any expert demonstrations. Our method is orders of magnitude more efficient than the previous state-of-the-art method relying on human-provided demonstrations [37].

Furthermore, our system can build towers that are taller than the training time. It also succeeds at placing blocks in different configurations such as pyramids without any additional training (i.e., zero-shot generalization). While we present results on the task of stacking blocks in various arrangements, the approach developed in this work does not make any task-specific assumption and is therefore applicable to a wide range of tasks involving manipulation of multiple objects.

Ii Related Work

Our work is broadly related to techniques for scaling reinforcement learning algorithms to more complex robotic manipulation settings, as well as the use of relational and curricular inductive biases in machine learning.

Relational Inductive Bias:  The use of relational inductive biases has a long history in reinforcement learning [30, 17, 58], and more broadly in logic and machine learning [47]. Recently, there has been great interest in the use of Graph Neural Networks (GNNs) for representing graph data structures, which are especially suitable for object-oriented environments [8, 11, 31, 27, 59, 5]. In the context of RL, a key motivation for relational representation is to support a varying number of objects as inputs and to explicitly model relationships between objects. In the past, GNNs have been studied in context of learning and transferring policies for locomotion across agents with variable morphologies [61, 42].

Closest to our work is past research combining GNNs with policy learning for manipulation tasks. However these works either rely on tens or hundreds of thousands of expert demonstrations [16, 28] or exclusively show results on video games[63]. Furthermore, while these works have considered GNNs to improve efficiency of solving a single task, we combine GNNs with learning from a curriculum of increasingly complex tasks to solve long-horizon manipulation problems that cannot be solved directly using current methods.

Curriculum Learning:  Curriculum learning addresses the effect of data sampling strategies on learning, under the presumption that proper sampling of tasks can allow for more sample efficient learning and avoidance of local minima [18]

. In particular, prior work has shown that ordering tasks by heuristic measures of difficulty can be effective

[6, 64]. A line of work has studied automatic discovery of curricula based on learning progress [21], adversarial self-play [52, 25], or backtracking [19]. So far, these methods have not yielded curricula capable of automatically discovering tasks of the complexity we consider. In this paper, our contribution is not in proposing a new algorithm or heuristic for choosing the task curricula, but to demonstrate the graph-based representations can make use of a curriculum for learning complex tasks.

Block Stacking:  Prior work on block stacking either heavily relied on human demonstrations [37, 15], or required significant reward engineering [48, 44], and/or carefully designed curriculum [48] of reaching, picking and placing blocks. Such design of curriculum and reward functions are hard problems with no known principled solutions. The work of  [12] stacked blocks using a low-cost robot. However, they assumed the blocks were already picked and used a dense reward function. Other lines of work [33, 57] achieved impressive results on stacking objects, but relied on extensive human-defined knowledge of detecting keypoints or assuming access to physics simulation. In contrast, we present a simple but effective method for stacking blocks using RL that makes minimal assumptions about task structure or the environment.

Hierarchical Reinforcement Learning (HRL)  aims to address the scaling and generalization problem in RL by decomposing problems into smaller subproblems. Examples of HRL frameworks include the “options" framework [53], feudal learning [60, 10] and the MaxQ framework [13]. A key unsolved challenge is joint end-to-end learning of multiple levels of control, while avoiding degenerate solutions that lack hierarchical abstraction. Most successful instantiations of hierarchical RL make use of domain knowledge to construct a hierarchy [4]. To our knowledge, no HRL algorithms have been successful at stacking tasks of the complexity we consider [40].

Iii Experimental Setup

Figure LABEL:fig:fig1 shows our simulated robotic environment consisting of a 7-DoF Fetch robot arm equipped with a two-fingered parallel jaw gripper based on OpenAI’s  [43]. MuJoCo physics engine [56] was used for simulations. The robot is tasked to manipulate 1-9 blocks kept on a table. Each block is a cube with sides of 5cm. The robot’s action space is 4D, consisting of relative change in 3D position of its end-effector and a scalar value representing the distance between two fingers of the gripper.

Observations:  The agent observes gripper features , including gripper velocity and position, and features representing N blocks. The block features are denoted by , where and is the feature representation of the

block. Each block is represented by a 15-D vector consisting of 3D position

, 3D orientation expressed as Euler angles, 3D position relative to the gripper, 3D cartesian velocity and 3D angular velocity. The goal is expressed as set of 3D block positions, . The overall input to the agent is therefore . At the start of every episode, the initial block positions are randomly initialized on the table and the goal positions are sampled using a pre-determined distribution. The maximum length of every episode is steps, where is the number of blocks.

Reward:  We use a step-wise sparse reward function where the robot is only rewarded when it places the block within a distance of from its desired goal location. The overall reward for placing blocks is given by: . We noticed that with this reward function, the robot learns to hold the top two blocks in its gripper instead of placing them and moving its hand away. To discourage this behavior, we added an additional term in the reward function to encourage the robot to move its hand away from the tower. This additional penalty was only provided when the hand was at a distance greater than from a “fully-stacked" tower. The overall reward is therefore given by, . Following [43], we set = 5cm, the size of each block.

Iv Preliminaries

Iv-a Reinforcement Learning

A typical RL agent acts within an environment E

, modeled by a discrete-time Markov Decision Process (MDP) described by state space

, action space , transition function , reward function , and discounting factor . The aim of the agent is to maximize the expected cumulative reward along states caused by a sequence of actions , by learning a suitable policy , i.e. .

A relatively efficient class of policy search algorithms is off-policy reinforcement learning. Q-learning [62] is a well known choice for off-policy learning, wherein the aim is to model the Q-function, i.e. . In principle, the optimal Q-function is found by solving the Bellman equation [54]. In practice, we approximate the Q-function with a function approximator (i.e. a neural network) parameterized by by minimizing the Bellman error , where is an optimization constant that represents the weights of a slowly-updated "target" network.

Iv-B Goal-Conditioned RL

While the above formulation is appropriate for a single goal, for solving multiple tasks, it is necessary to provide a task description as input [49, 1, 3]. Goal conditioned policies are expressed as , where represents the goal state. The learning problem is expressed as:


where goal is sampled from a goal distribution .

Iv-C Graph Neural Networks (GNN)

The central computation in a GNN is message passing between 1-hop vertices of a graph, performed by a graph-to-graph module. This module takes as input a variable-size vertex set and outputs an updated set , where is the number of vertices in the input graph. denote feature vectors of the node before and after a round of message passing. In each message passing round, each vertex sends a message to every other vertex. In attention-based GNNs, the incoming messages are weighted by a scalar coefficient (computed by attention) according to their relevance to the receiving vertex. The new feature representation of the vertex is the weighted sum of incoming messages. Message passing is typically performed multiple times. After message passing, the entire graph is represented as a fixed-sized embedding by pooling features across all vertices.

Mathematically, let the feature representation of the vertex at timestep be . In every message passing round, each vertex generates a query , key and a message using independently-parameterized functions , , and . Each vertex in the graph receives a message from all the vertices and computes it’s feature representation, , where are the attention weights and are computed as follows: .

V Method

We present a simple, but effective method for solving long-horizon, sparse reward tasks using reinforcement learning. Our core contribution is to equip the RL agent with inductive biases of relational reasoning in order to enable learning from a curriculum of tasks of increasing complexity. We use Soft-Actor Critic (SAC; [23]

) as our base learning algorithm because it is more robust to choice of hyperparameters and random seeds as compared to alternative off-policy learners such as DDPG 

[35]. To use the same policy for multiple tasks, we modified SAC to be goal-conditioned [49, 1, 3]. For better sample efficiency, we also incorporated the idea of goal re-labelling via hindsight experience replay (HER; [3]). Details of SAC and HER can be found in the respective papers and are not directly relevant to our work. While we use SAC + HER for policy learning, our contributions are not specific to these algorithms and are applicable to any policy learning method.

We represent both the actor and critic in SAC using the graph neural network architecture described in Section IV-C. The various components of the GNN (

) use 64D linear layers. We use separate weights for each round of message passing and terminate the message passing after 3 rounds. We use a residual connection and layer normalization between the output of message passing round

and the input of message passing round to ease optimization. We call this agent architecture ReNN. We compare the performance of ReNN against the baseline system that constructs the actor and critic using four layers of 256D fully connected layers (referred to as MLP in rest of the paper).

Fig. 1:

Comparing task performance measured as the mean number of blocks stacked per timestep during training. We report the mean and standard deviation across multiple workers and/or seeds. The performance of relational (

ReNN) and the usual multi-layer (MLP) architectures are reported when they are subjected to different training curricula described in Section 1. Both ReNN and MLP fail to stack blocks, when the robot was directly trained for stacking 6 blocks (, ). Only ReNN trained with sequential curriculum () succeeds at stacking six blocks.

Training Curriculum:  We trained the robot to stack multiple blocks using three different curricula of tasks:

  • Direct: The robot was directly tasked to learn a policy to stack six blocks starting from scratch.

  • Uniform: At every episode, the number of blocks was uniformly sampled between 1 and 6.

  • Sequential: The robot was tasked to first pick and place a single block at goal positions that were uniformly and randomly chosen to be on the table or in the air. The robot then had to pick and place 2 blocks, where goal position of one block was sampled on the table and the goal position for the second block was sampled using the process described above. Thereafter, the robot was tasked with stacking blocks in a single tower configuration starting with 2 blocks. After the robot perfected stacking (N-1) blocks, it was given N blocks to stack. N was sequentially increased from 3 to 6. The transition points in this curriculum were manually chosen based on the success rates on stacking.

V-a Testing Details

Fig. 2: Quantitative evaluation of zero-shot generalization results of policies trained to stack blocks in a single tower. These policies were evaluated, without any further training, on (a) single (but taller) tower (shown in shades of red); (b) multiple towers; (c) pyramid configurations. Details of these testing setups can be found in Section V-A. The results show that robot is capable of zero-shot generalization to many of these tasks.

We evaluated the generalization of the policy trained for stacking a single tower by evaluating its performance on the following tests (see Appendix for visuals):

  • Single Tower: A single point was uniformly sampled on the table to serve as the base of a block tower. The goal positions of the blocks corresponded to translation along the z-axis from the base.

  • Multiple Towers: Few points were sampled on the table to serve as the base location of multiple towers. Each block was randomly assigned to a tower to produce towers of approximately equal height.

  • Pyramid: A uniformly sampled point on the table served as a corner point for pyramid configuration. Figure A.1 shows different Multiple Towers and Pyramid goal configurations for varying number of blocks.

We report performance of ReNN- Sequential (referred to as ReNN in later text) across three seeds. For other methods we report performance on a single seed. Success rate is reported as accuracy of completing a task averaged over 100 episodes. An episode is counted as successful when each block is within its goal position at the final time step.

Vi Results

Task Single Tower 4 Single Tower 5 Single Tower 6
Nair‘17 [37] 91% (850M) 50% (1000M) 32% (2300M)
Ours 93%4% (23M) 84%6% (27M) 75%4% (30M)
TABLE I: Comparing the performance of our method against the previous state-of-the art [37] that makes use of human demonstrations on the block stacking task. Each entry, , denotes accuracy of after number of environment steps. Our method is both more sample efficient and outperforms prior work.

Figure 1 shows that ReNN trained with the sequential curriculum (green line; section V) succeeds at stacking six blocks into a tower. Standard MLP architectures or ReNN trained to directly stack 6 blocks without the curriculum fail. Our experiments revealed that training with uniform curriculum was also insufficient. These results show that both ReNN and the sequential training are critical for success. To the best of our knowledge, ours is the first paper to show that is is possible to train a RL agent to stack six or more blocks in a tower after starting from scratch, without requiring expert demonstrations.

We report quantitative performance of our method and baselines in Table I. Our method achieves a success rate of 75% at stacking 6 blocks in 30 million timesteps. In comparison, the existing state-of-art method [37], that makes use of human demonstrations and resets, achieves only a success rate of 32% after over 2.3 billion timesteps. While the base learning algorithm used by [37] is DDPG + HER, in comparison to SAC + HER used by us, the orders of magnitude difference in performance cannot be attributed to the choice of using SAC instead of DDPG. We attempted to replicate results of [37] using SAC. However, we were unsuccessful at training SAC with behavior cloning due to the challenge in weighing the entropy term in SAC against the behavior cloning loss.

Careful analysis of Figure 1 reveals that there are several dips in performance as the training progresses. Many of the significant dips correspond to increase in task complexity to stack N+1 blocks, after stacking N

blocks. In most cases, the dip in performance is overcome after little additional experience. The only notable exception is the performance dip at 9M steps that corresponds to transitioning from 1 to 2 blocks. This was the first time the agent observed multiple objects. Additionally, We found that SAC converged faster, albeit with higher variance when it’s exploration was augmented to take a random action with probability of 0.1.

Vi-a Zero-shot Generalization

Fig. 3: Analysis of internal representation used by our system to solve the task of stacking six blocks into a single tower. The first row shows the key step of this task. The second row, shows for each of these key frames, a matrix, whose entry represents the influence of feature representation of block on the block after three message passing rounds. The labelled index along the y-axis and x-axis correspond to the object ID. Object IDs 1 - 6 correspond to green, yellow, blue, pink, red, and black blocks in order. The attention map reveals that our agent pays attention to the sub-set of blocks relevant to solving the sub-problem (stacking one block) at hand. We speculate that such an attention map suggests that the agent has internally learnt to decompose the complex block stacking task into simpler sub-problems.

It is desirable to learn policies that are not only adept at the task they were trained on, but can be re-purposed for new and related tasks. If our ReNN architecture indeed provides a good inductive bias, then it should be possible solve different block configuration tasks with high-accuracy. To test this, we evaluated the performance of the learned policy, without any fine-tuning on previously unseen block configurations (i.e. zero-shot generalization) described in Section V-A. The results of this analysis are summarized in Figure 2.

Single Tower Evaluation:  Figure 2 shows that a policy learned to stack blocks generalizes to stacking blocks without any training. The performance on stacking blocks, where drops significantly. One possible explanation is that it becomes progressively harder to stabilize larger number of blocks in a tower and the robot needs to substantially refine its strategy to stack more blocks. An analysis of failure modes is presented in Appendix B.

Multiple Towers Evaluation:  The previous experiments tested generalization to a larger number of blocks, but on the same task. To test if the learned policy generalizes to new tasks, we evaluated the performance on stacking multiple towers instead of a single tower. Results in Figure 2(b) show that the agent trained for stacking a single tower of blocks can successfully stack multiple towers blocks. The performance again drops for . However, generalization to is better on the multiple towers task as compared to the single tower task. This suggest that while ReNN can generalize to a larger number of blocks than seen during training time, stacking a taller single tower without additional training is hard due to the difficulty of stabilizing a taller stack of blocks.

Pyramid Evaluation:  To stress test our system further, we evaluated its performance on placing blocks in a pyramid configuration (see Figure A.1). Note that the robot never saw pyramids during training. Stacking blocks in pyramid is different than a tower, because now blocks may need to be balanced on two supporting blocks instead of only being stacked vertically. Figure 2(c) shows that our system is able to generalize and manipulate larger number of blocks than seen in training into pyramid configurations. Interestingly, the agent trained on Single Tower 4 performs better on the difficult Pyramid 5 and Pyramid 6 tasks than the agent trained on Single Tower 6. One possible explanation is that the agent trained on taller tower overfits to stacking blocks vertically, and is less able to stack blocks at an horizontal offset, which is useful for the pyramid task.

Emergent Strategies:  The accompanying videos ( show that our agent automatically learns to push other blocks to grasp a particular block, grasps two blocks at a time and places them one by one to save time and other complex behaviors. These strategies emerge automatically as a consequence of optimizing a sparse reward function.

To the best of our knowledge, ours is the first work that reports such zero-shot generalization on the block stacking task using RL. At the same time, we acknowledge, there is substantial room for improving the zero-shot results and the stacking performance. Some future directions are described in Section VII.

Vi-B Analyzing the learned representations

In order to gain insights into why ReNN leads to faster convergence and better generalization, we visualized the attention patterns as the robot stacked six blocks (see Figure 3). The first row shows the key steps in tower stacking. Each column in second row is a matrix (). Each entry in the matrix, represents the normalized relevance score of the block on the features of the block (see defined in Section  IV-C) computed by the final layer. It can be seen that maximum attention is to paid to the block that is to be placed and the attention on existing blocks in the stack decreases from the top-most to bottom-most block. Such attention pattern suggests that our system has learned to focus on the blocks most relevant to current block being placed. This is interesting because it suggests that ReNN has learned to decompose a complex problem into simpler sub-problems. We hypothesize that such decomposition is the reason why our system can learn from a task curricula and exhibits zero-shot generalization.

Vii Discussion

We have presented a framework for learning long-horizon, sparse reward tasks using deep reinforcement learning, relational graph architecture and curriculum learning. While we are orders of magnitude more sample efficient than the the existing state-of-the-art, our method would still require a few dozen robots (corresponding to our 35 workers) and several days (assuming each action takes .25 seconds) of real world training to achieve a comparable environment step complexity. And while block stacking is representative of long-horizon, multi-object manipulation tasks, it is important to scale our method to tasks involving more complicated object geometries and more granular manipulation.

In the current work, the curriculum is manually designed and based on the principle that smaller sets of objects are easier to learn to manipulate than larger sets of objects. However, more complicated and effective curricula could exist along axes of variation beyond just the object cardinality, and discovering these curricula automatically is an interesting direction for future research. One point of concern with relational architectures is that the computation time is quadratic in the number of entities. Developing computationally efficient methods is therefore important to scale these methods to environments with much larger numbers of objects. Finally, while we have presented results from state observation, in the future we would like to scale our system to work from visual and other sensory observations.

Viii Acknowledgements

We acknowledge support from US Department of Defense, DARPA’s Machine Common Sense Grant and the BAIR and BDD industrial consortia. We thank Amazon Web Services (AWS) for their generous support in the form of cloud credits. We’d like to thank Vitchyr Pong, Kristian Hartikainen, Ashvin Nair and other members of the BAIR lab and the Improbable AI lab for helpful discussions during this project.


  • [1] P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine (2016) Learning to poke by poking: experiential learning of intuitive physics. NIPS. Cited by: §I, §IV-B, §V.
  • [2] M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. (2018) Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177. Cited by: §I.
  • [3] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba (2017) Hindsight experience replay. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), pp. 5048–5058. External Links: Link Cited by: §IV-B, §V.
  • [4] P. Bacon, J. Harb, and D. Precup (2016) The option-critic architecture. CoRR abs/1609.05140. External Links: Link, 1609.05140 Cited by: §II.
  • [5] P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende, and K. kavukcuoglu (2016) Interaction networks for learning about objects, relations and physics. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, USA, pp. 4509–4517. External Links: ISBN 978-1-5108-3881-9, Link Cited by: §II.
  • [6] Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, New York, NY, USA, pp. 41–48. External Links: ISBN 978-1-60558-516-1, Link, Document Cited by: §II.
  • [7] Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48. Cited by: §I.
  • [8] J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun (2014) Spectral networks and locally connected networks on graphs. In International Conference on Learning Representations (ICLR2014), CBLS, April 2014, (English (US)). Cited by: §II.
  • [9] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman (2018) Quantifying generalization in reinforcement learning. arXiv preprint arXiv:1812.02341. Cited by: §I.
  • [10] P. Dayan and G. E. Hinton (1993) Feudal reinforcement learning. In Advances in neural information processing systems, pp. 271–278. Cited by: §II.
  • [11] M. Defferrard, X. Bresson, and P. Vandergheynst (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.), pp. 3844–3852. External Links: Link Cited by: §II.
  • [12] M. P. Deisenroth, C. E. Rasmussen, and D. Fox (2011) Learning to control a low-cost manipulator using data-efficient reinforcement learning. Robotics Science and Systems, pp. 57–64. Cited by: §I, §II.
  • [13] T. G. Dietterich (1998) The maxq method for hierarchical reinforcement learning. In In Proceedings of the Fifteenth International Conference on Machine Learning, pp. 118–126. Cited by: §II.
  • [14] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647–655. Cited by: §I.
  • [15] Y. Duan, M. Andrychowicz, B. Stadie, O. J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba (2017)

    One-shot imitation learning

    In Advances in neural information processing systems, pp. 1087–1098. Cited by: §II.
  • [16] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba (2017) One-shot imitation learning. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), pp. 1087–1098. External Links: Link Cited by: §II.
  • [17] S. Džeroski, L. De Raedt, and K. Driessens (2001-04) Relational reinforcement learning. Machine Learning 43 (1), pp. 7–52. External Links: ISSN 1573-0565, Document Cited by: §II.
  • [18] J. L. Elman (1993) Learning and development in neural networks: the importance of starting small. Cognition 48 (1), pp. 71–99. External Links: ISSN 0010-0277, Document Cited by: §II.
  • [19] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel (2017) Reverse curriculum generation for reinforcement learning. arXiv preprint arXiv:1707.05300. Cited by: §II.
  • [20] D. Ghosh, A. Singh, A. Rajeswaran, V. Kumar, and S. Levine (2017) Divide-and-conquer reinforcement learning. arXiv preprint arXiv:1711.09874. Cited by: §I.
  • [21] A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu (2017) Automated curriculum learning for neural networks. CoRR abs/1704.03003. External Links: Link, 1704.03003 Cited by: §II.
  • [22] S. Gu, E. Holly, T. Lillicrap, and S. Levine (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3389–3396. Cited by: §I.
  • [23] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR abs/1801.01290. External Links: Link, 1801.01290 Cited by: §I, §V.
  • [24] D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan (2017) Inverse reward design. In Advances in neural information processing systems, pp. 6765–6774. Cited by: §I.
  • [25] D. Held, X. Geng, C. Florensa, and P. Abbeel (2017) Automatic goal generation for reinforcement learning agents. CoRR abs/1705.06366. External Links: Link, 1705.06366 Cited by: §II.
  • [26] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver (2018) Rainbow: combining improvements in deep reinforcement learning. In

    Thirty-Second AAAI Conference on Artificial Intelligence

    Cited by: §I.
  • [27] A. Jain, A. R. Zamir, S. Savarese, and A. Saxena (2015) Structural-rnn: deep learning on spatio-temporal graphs. CoRR abs/1511.05298. External Links: Link, 1511.05298 Cited by: §II.
  • [28] M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu (2019) Reasoning about physical interactions with object-oriented prediction and planning. In International Conference on Learning Representations, Cited by: §II.
  • [29] N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. Togelius, and S. Risi (2018) Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv:1806.10729. Cited by: §I.
  • [30] L. P. Kaelbling, T. Oates, N. Hernandez, and S. Finney (2001) Learning in worlds with objects. In Working Notes of the AAAI Stanford Spring Symposium on Learning Grounded Representations, pp. 31–36. Cited by: §II.
  • [31] T. N. Kipf and M. Welling (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907. Cited by: §II.
  • [32] P. Kormushev, S. Calinon, and D. G. Caldwell (2010) Robot motor skill coordination with em-based reinforcement learning. In 2010 IEEE/RSJ international conference on intelligent robots and systems, pp. 3232–3237. Cited by: §I.
  • [33] O. Kroemer, S. Leischnig, S. Luettgen, and J. Peters (2018) A kernel-based approach to learning contact distributions for robot manipulation tasks. Autonomous Robots 42 (3), pp. 581–600. Cited by: §II.
  • [34] S. Levine, C. Finn, T. Darrell, and P. Abbeel (2016) End-to-end training of deep visuomotor policies. JMLR. Cited by: §I.
  • [35] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra (2016) Continuous control with deep reinforcement learning. ICLR. Cited by: §V.
  • [36] M. Lopes, T. Lang, M. Toussaint, and P. Oudeyer (2012)

    Exploration in model-based reinforcement learning by empirically estimating learning progress

    In NIPS, Cited by: §I.
  • [37] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel (2017) Overcoming exploration in reinforcement learning with demonstrations. CoRR abs/1709.10089. External Links: Link, 1709.10089 Cited by: §I, §I, §II, TABLE I, §VI.
  • [38] A. Nichol, J. Achiam, and J. Schulman (2018) On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999. Cited by: §I.
  • [39] P. Oudeyer and F. Kaplan (2009) What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics. Cited by: §I.
  • [40] A. Pashevich, D. Hafner, J. Davidson, R. Sukthankar, and C. Schmid (2018) Modulated policy hierarchies. CoRR abs/1812.00025. External Links: Link, 1812.00025 Cited by: §II.
  • [41] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell (2017) Curiosity-driven exploration by self-supervised prediction. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops

    pp. 16–17. Cited by: §I.
  • [42] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. A. Efros (2019) Learning to control self- assembling morphologies: a study of generalization via modularity. In arXiv preprint arXiv:1902.05546, Cited by: §I, §II.
  • [43] M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, et al. (2018) Multi-goal reinforcement learning: challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464. Cited by: §III, §III.
  • [44] I. Popov, N. Heess, T. P. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, T. Erez, Y. Tassa, and M. Riedmiller (2018) Data-efficient deep reinforcement learning for dexterous manipulation. External Links: Link Cited by: §II.
  • [45] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine (2017) Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087. Cited by: §I.
  • [46] S. Ren, K. He, R. Girshick, and J. Sun (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91–99. Cited by: §I.
  • [47] S. Russell and P. Norvig (2009) Artificial intelligence: a modern approach. 3rd edition, Prentice Hall Press. External Links: ISBN 978-0-13-604259-4 Cited by: §II.
  • [48] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell (2016) Progressive neural networks. arXiv preprint arXiv:1606.04671. Cited by: §I, §II.
  • [49] T. Schaul, D. Horgan, K. Gregor, and D. Silver (2015) Universal value function approximators. In International Conference on Machine Learning, pp. 1312–1320. Cited by: §IV-B, §V.
  • [50] J. Schmidhuber (2010) Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development. Cited by: §I.
  • [51] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §I.
  • [52] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus (2018) Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, External Links: Link Cited by: §I, §II.
  • [53] R.S. Sutton, D. Precup, and S. Singh (1999) Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112, pp. 181–211. Cited by: §I, §II.
  • [54] R. S. Sutton and A. G. Barto (2018) Reinforcement learning: an introduction. Second edition, The MIT Press. External Links: Link Cited by: §IV-A.
  • [55] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23–30. Cited by: §I.
  • [56] E. Todorov, T. Erez, and Y. Tassa (2012) MuJoCo: a physics engine for model-based control.. In IROS, pp. 5026–5033. External Links: ISBN 978-1-4673-1737-5, Link Cited by: §III.
  • [57] M. Toussaint (2015) Logic-geometric programming: an optimization-based approach to combined task and motion planning. In Twenty-Fourth International Joint Conference on Artificial Intelligence, Cited by: §II.
  • [58] M. Van Otterlo (2002) Relational representations in reinforcement learning: review and open problems. In Proceedings of the ICML, Vol. 2. Cited by: §II.
  • [59] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. CoRR abs/1706.03762. External Links: Link, 1706.03762 Cited by: §II.
  • [60] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu (2017) FeUdal networks for hierarchical reinforcement learning. CoRR abs/1703.01161. External Links: Link, 1703.01161 Cited by: §II.
  • [61] T. Wang, R. Liao, J. Ba, and S. Fidler (2018) NerveNet: learning structured policy with graph neural networks. In International Conference on Learning Representations, External Links: Link Cited by: §I, §II.
  • [62] C. J. Watkins and P. Dayan (1992) Q-learning. Machine learning 8 (3-4), pp. 279–292. Cited by: §IV-A.
  • [63] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, et al. (2019) Deep reinforcement learning with relational inductive biases. International Conference on Learning Representations. Cited by: §I, §II.
  • [64] W. Zaremba and I. Sutskever (2014) Learning to execute. CoRR abs/1410.4615. External Links: Link, 1410.4615 Cited by: §II.
  • [65] C. Zhang, O. Vinyals, R. Munos, and S. Bengio (2018) A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893. Cited by: §I.