1 Introduction
One of the aspirations of artificial intelligence is a cognitive agent that can adaptively and dynamically form plans to achieve its goal. Traditionally, this role has been filled by modelbased RL approaches, which first learn an explicit model of the environment’s system dynamics or rules, and then apply a planning algorithm (such as tree search) to the learned model. Modelbased approaches are potentially powerful but have been challenging to scale with learned models in complex and highdimensional environments
(Talvitie, 2014; Asadi et al., 2018), though there has been recent progress in that direction (Buesing et al., 2018; Ebert et al., 2018).More recently, a variety of approaches have been proposed that learn to plan implicitly, solely by modelfree training. These modelfree planning agents utilize a special neural architecture that mirrors the structure of a particular planning algorithm. For example the neural network may be designed to represent search trees (Farquhar et al., 2017; Oh et al., 2017; Guez et al., 2018), forward simulations (Racanière et al., 2017; Silver et al., 2016), or dynamic programming (Tamar et al., 2016). The main idea is that, given the appropriate inductive bias for planning, the function approximator can learn to leverage these structures to learn its own planning algorithm. This kind of algorithmic function approximation may be more flexible than an explicit modelbased approach, allowing the agent to customize the nature of planning to the specific environment.
In this paper we explore the hypothesis that planning may occur implicitly, even when the function approximator has no special inductive bias toward planning. Previous work (Pang & Werbos, 1998; Wang et al., 2018) has supported the idea that modelbased behavior can be learned with general recurrent architectures, with planning computation amortized over multiple discrete steps (Schmidhuber, 1990)
, but comprehensive demonstrations of its effectiveness are still missing. Inspired by the successes of deep learning and the universality of neural representations, our main idea is simply to furnish a neural network with a high capacity and flexible representation, rather than mirror any particular planning structure. Given such flexibility, the network can in principle learn its own algorithm for approximate planning. Specifically, we utilize a family of neural networks based on a widely used function approximation architecture: the stacked convolutional LSTMs (ConvLSTM by
Xingjian et al. (2015)).It is perhaps surprising that a purely modelfree reinforcement learning approach can be so successful in domains that would appear to necessitate explicit planning. This raises a natural question: what is planning? Can a modelfree RL agent be considered to be planning, without any explicit model of the environment, and without any explicit simulation of that model?
Indeed, in many definitions (Sutton et al., 1998), planning requires some explicit deliberation using a model, typically by considering possible future situations using a forward model to choose an appropriate sequence of actions. These definitions emphasize the nature of the mechanism (the explicit lookahead), rather than the effect it produces (the foresight). However, what would one say about a deep network that has been trained from examples in a challenging domain to emulate such a planning process with nearperfect fidelity? Should a definition of planning rule out the resulting agent as effectively planning?
Instead of tying ourselves to a definition that depends on the inner workings of an agent, in this paper we take a behaviourist approach to measuring planning as a property of the agent’s interactions. In particular, we consider three key properties that an agent equipped with planning should exhibit.
First, an effective planning algorithm should be able to generalize with relative ease to different situations. The intuition here is that a simple function approximator will struggle to predict accurately across a combinatorial space of possibilities (for example the value of all chess positions), but a planning algorithm can perform a local search to dynamically compute predictions (for example by tree search). We measure this property using procedural environments (such as random gridworlds, Sokoban (Racanière et al., 2017), Boxworld (Zambaldi et al., 2018)) with a massively combinatorial space of possible layouts. We find that our modelfree planning agent achieves stateoftheart performance, and significantly outperforms more specialized modelfree planning architectures. We also investigate extrapolation to a harder class of problems beyond those in the training set, and again find that our architecture performs effectively – especially with larger network sizes.
Second, a planning agent should be able to learn efficiently from relatively small amounts of data. Modelbased RL is frequently motivated by the intuition that a model (for example the rules of chess) can often be learned more efficiently than direct predictions (for example the value of all chess positions). We measure this property by training our modelfree planner on small datasets, and find that our modelfree planning agent still performs well and generalizes effectively to a heldout test set.
Third, an effective planning algorithm should be able to make good use of additional thinking time. Put simply, the more the algorithm thinks, the better its performance should be. This property is likely to be especially important in domains with irreversible consequences to wrong decisions (e.g. death or deadends). We measure this property in Sokoban by adding additional thinking time at the start of an episode, before the agent commits to a strategy, and find that our modelfree planning agent solves considerably more problems.
Together, our results suggest that a modelfree agent, without specific planninginspired network structure, can learn to exhibit many of the behavioural characteristics of planning. The architecture presented in this paper serves to illustrate this point, and shows the surprising power of one simple approach. We hope our findings broaden the search for more general architectures that can tackle an even wider range of planning domains.
2 Methods
We first motivate and describe the main network architecture we use in this paper. Then we briefly explain our training setup. More details can be found in Appendix C.
2.1 Model architectures
We desire models that can represent and learn powerful but unspecified planning procedures. Rather than encode strong inductive biases toward particular planning algorithms, we choose highcapacity neural network architectures that are capable of representing a very rich class of functions. As in many works in deep RL, we make use of convolutional neural networks (known to exploit the spatial structure inherent in visual domains) and LSTMs (known to be effective in sequential problems). Aside from these weak but common inductive biases, we keep our architecture as general and flexible as possible, and trust in standard modelfree reinforcement learning algorithms to discover the functionality of planning.
2.1.1 Basic architecture
The basic element of the architecture is a ConvLSTM (Xingjian et al., 2015) – a neural network similar to an LSTM but with a 3D hidden state and convolutional operations. A recurrent network stacks together ConvLSTM modules. For a stack depth of , the state contains all the cell states and outputs of each module : . The module weights
are not shared along the stack. Given a previous state and an input tensor
, the next state is computed as . The network is then repeated times within each timestep (i.e., multiple internal ticks per real timestep). If is the state at the end of the previous timestep, we obtain the new state given the input as:(1) 
The elements of all preserve the spatial dimensions of the input . The final output of the recurrent network for a single timestep is , the hidden state of the deepest ConvLSTM module after N ticks, obtained from . We describe the ConvLSTM itself and alternative choices for memory modules in Appendix C.
The rest of the network is rather generic. An encoder network composed of convolutional layers processes the input observation into a tensor — given as input to the recurrent module . The encoded input is also combined with through a skipconnection to produce the final network output. The network output is then flattend and an action distribution and a statevalue are computed via a fullyconnected MLP. The diagram in Fig 1 illustrates the full network.
From here on, we refer to this architecture as Deep Repeated ConvLSTM (DRC) network architecture, and sometimes followed explicitly by the value of and (e.g., DRC(3, 2) has depth and repeats).
2.1.2 Additional details
Less essential design choices in the architectures are described here. Ablation studies show that these are not crucial, but do marginally improve performance (see Appendix F).
Encoded observation skipconnection The encoded observation is provided as an input to all ConvLSTM modules in the stack.
Topdown skip connection As described above, the flow of information in the network only goes up (and right through time). To allow for more general computation we add feedback connection from the last layer at one time step to the first layer of the next step.
Poolandinject To allow information to propagate faster in the spatial dimensions than the size of the convolutional kernel within the ConvLSTM stack, it is useful to provide a pooled version of the module’s last output
as an additional input on lateral connections. We use both max and mean pooling. Each pooling operation applies pooling spatially for each channel dimension, followed by a linear transform, and then tiles the result back into a 2D tensor. This is operation is related to the poolandinject method introduced by
Racanière et al. (2017) and to SqueezeandExcitation blocks (Hu et al., 2017).Padding The convolutional operator is translation invariant. To help it understand where the edge of the input image is, we append a feature map to the input of the convolutional operators that has ones on the boundary and zeros inside.
2.2 Training
We used a distributed framework to train an RL agent using the IMPALA Vtrace actorcritic algorithm (Espeholt et al., 2018). While we found this training regime to help for training heavier networks, we also ran experiments which demonstrate that the DRC architecture can be trained effectively with A3C (Mnih et al., 2016). More details on the setup can be found in Appendix D.2.
3 Planning Domains
Our domains are formally specified as RL problems, where agents must learn via reward feedback obtained by interacting with the environment (Sutton et al., 1998). We focus on combinatorial domains for which episodes are procedurally generated. In these domains each episode is instantiated in a pseudorandom configuration, so solving an episode typically requires some form of reasoning. Most of the environments are fullyobservable and have simple 2D visual features. The domains are illustrated and explained in Appendix A. In addition to the planning domains listed below, we also run control experiments on a set of Atari 2600 games (Bellemare et al., 2013).
Gridworld A simple navigation domain following (Tamar et al., 2016), consisting of a grid filled with obstacles. The agent, goal, and obstacles are randomly placed for each episode.
Sokoban A difficult puzzle domain requiring an agent to push a set of boxes onto goal locations (Botea et al., 2003; Racanière et al., 2017). Irreversible wrong moves can make the puzzle unsolvable. We describe how we generate a large number of levels (for the fixed problem size 10x10 with 4 boxes) at multiple difficulty levels in Appendix B, and then each split into a training and test set. We are releasing these levels in the standard Sokoban format^{1}^{1}1https://github.com/deepmind/boxobanlevels. Unless otherwise specified, we ran experiments with the easier unfiltered set of levels.
Boxworld Introduced in (Zambaldi et al., 2018), the aim is to reach a goal target by collecting coloured keys and opening colourmatched boxes until a target is reached. The agent can see the keys (i.e., their colours) locked within boxes; thus, it must carefully plan the sequence of boxes that should be opened so that it can collect the keys that will lead to the target. Keys can only be used once, so opening an incorrect box can lead the agent down a deadend path from which it cannot recover.
MiniPacman (Racanière et al., 2017). The player explores a maze that contains food while being chased by ghosts. The aim of the player is to collect all the rewarding food. There are also a few power pills which allow the player to attack ghosts (for a brief duration) and earn a large reward. See Appendix A.2 for more details.
4 Results
We first examine the performance of our model and other approaches across domains. Then we report results aimed at understanding how elements of our architecture contribute to observed performance. Finally, we study evidence of iterative computation in Section 4.2 and generalization in Section 4.3.
4.1 Comparisons
In general, across all domains listed in Section 3, the DRC architecture performed very well with only modest tuning of hyperparameters (see Appendix D). The DRC(3,3) variant was almost always the best in terms both of data efficiency (early learning) and asymptotic performance.
Gridworld: Many methods efficiently learn the Gridworld domain, especially for small grid sizes. We found that for larger grid sizes the DRC architecture learns more efficiently than a vanilla Convolutional Neural Network (CNN) architecture of similar weight and computational capacity. We also tested Value Iteration Networks (VIN) (Tamar et al., 2016), which are specially designed to deal with this kind of problem (i.e. local transitions in a fullyobservable 2D state space). We found that VIN, which has many fewer parameters and a wellmatched inductive bias, starts improving faster than other methods. It outperformed the CNN and even the DRC during earlystage training, but the DRC reached better final accuracy (see Table 2 and Figure 14(a) in the Appendix). Concurrent to our work, Lee et al. (2018) observed similar findings in various path planning settings when comparing VIN to an architecture with weaker inductive biases.
Model  % solved at steps  % solved at steps 

DRC(3, 3)  30  99 
VIN  80  97 
CNN  3  90 
Model  % solved at steps  % solved at steps 

DRC(3, 3)  80  99 
ResNet  14  96 
CNN  25  92 
I2A (unroll=15)  21  83 
1D LSTM(3,3)  5  74 
ATreeC  1  57 
VIN  56 
Sokoban: In Sokoban, we demonstrate stateoftheart results versus prior work which targeted similar boxpushing puzzle domains (ATreeC (Farquhar et al., 2017), I2A (Racanière et al., 2017)) and other generic networks (LSTM (Hochreiter & Schmidhuber, 1997), ResNet (He et al., 2016), CNNs). We also test VIN on Sokoban, adapting the original approach to our state space by adding an input encoder to the model and an attention module at the output to deal with the imperfect stateaction mappings. Table 4 compares the results for different architectures at the end of training. Only 1% of test levels remain unsolved by DRC(3,3) after 1e9 steps, with the secondbest architecture (a large ResNet) failing four times as often.
Boxworld: On this domain several methods obtain nearperfect final performance. Still, the DRC model learned faster than published methods, achieving 80% success after 2e8 steps. In comparison, the best ResNet achieved 50% by this point. The relational method of Zambaldi et al. (2018) can learn this task well but only solved 10% of levels after 2e8 steps.
Atari 2600 To test the capacity of the DRC model to deal with richer sensory data, we also examined its performance on five planningfocussed Atari games (Bellemare et al., 2013). We obtained stateoftheart scores on three of five games, and competitive scores on the other two (see Appendix E.2 and Figure 10 for details).
4.1.1 Influence of network architecture
We studied the influence of stacking and repeating the ConvLSTM modules in the DRC architecture, controlled by the parameters (stack length) and (number of repeats) as described in Section 2.1
. These degrees of freedom allow our networks to compute its output using shared, iterative, computation with
, as well as computations at different levels of representation and more capacity with . We found that the DRC(3,3) (i.e, ) worked robustly across all of the tested domain. We compared this to using the same number of modules stacked without repeats (DRC(9,1)) or only repeated without stacking (DRC(1,9)). In addition, we also look at the same smaller capacity versions and (which reduces to a standard ConvLSTM). Figure 2(a) shows the results on Sokoban for the different network configurations. In general, the versions with more capacity performed better. When tradingoff stacking and repeating (with total of 9 modules), we observed that only repeating without stacking was not as effective (this has the same number of parameters as the DRC(1,1) version), and only stacking was slower to train in the early phase but obtained a similar final performance. We also confirmed that DRC(3,3) performed better than DRC(1,1) in Boxworld, MiniPacman, and Gridworld (see Figure 15(a) in Appendix).On harder Sokoban levels (Mediumdifficulty dataset), we trained the DRC(3,3) and the larger capacity DRC(9,1) configurations and found that, even though DRC(9,1) was slower to learn at first, it ended up reaching a better score than DRC(3,3) (94% versus 91.5% after 1e9 steps). See Fig 9 in appendix. We tested the resulting DRC(9,1) agent on the hardest Sokoban setting (Harddifficulty), and found that it solved 80% of levels in less than 48 minutes. In comparison, running a powerful tree search algorithm, Levin Tree Search (Orseau et al., 2018), with a DRC(1,1) as policy prior solves 94%, but in 10 hours.
In principle, deep feedforward models should support iterative procedures within a single timestep and perhaps match the performance of our recurrent networks (Jastrzebski et al., 2017). In practice, deep ResNets performed poorly versus our recurrent models (see Figure 2(b)), and are in any case incapable of caching implicit iterative planning steps over time steps. Finally, we note that recurrence by itself was also not enough: replacing the ConvLSTM modules with flat 1D LSTMs performed poorly (see Figure 2(b)).
Across experiments and domains, our results suggests that both the network capacity and the iterative aspect of a model drives the agent’s performance.
4.2 Iterative computation
One desirable property for planning mechanisms is that their performance scales with additional computation without seeing new data. Although RNNs (and more recently ResNets) can in principle learn a function that can be iterated to obtain a result (Graves, 2016; Jastrzebski et al., 2017; Greff et al., 2016), it is not clear whether the networks trained in our RL domains learn to amortize computation over time in this way. To test this, we took trained networks in Sokoban (unfiltered) and tested post hoc their ability to improve their results with additional steps. We introduced ‘noop’ actions at the start of each episode – up to extra computation steps where the agent’s action is fixed to have no effect on the environment. We observed clear performance improvements on Medium difficulty levels of about 5% for DRC networks (see Figure 4). We did not find such improvements for the simpler fullyconnected LSTM architecture. This suggests that the networks have learned a scalable strategy for the task which is computed and refined through a series of identical steps, thereby exhibiting one of the essential properties of a planning algorithm.
4.3 Generalization
In combinatorial domains generalization is a central issue. Given limited exposure to configurations in an environment, how well can a model perform on unseen scenarios? In the supervised setting, large flexible networks are capable of overfitting. Thus, one concern when using highcapacity networks is that they may overfit to the task, for example by memorizing, rather than learning a strategy that can generalize to novel situations. Recent empirical work in SL (Supervised Learning) has shown that the generalization of large networks is not well understood
(Zhang et al., 2016; Arpit et al., 2017). Generalization in RL is even less well studied, though recent work (Zhang et al., 2018a, b; Cobbe et al., 2018) has begun to explore the effect of training data diversity.We explored two main axes in the space of generalization. We varied both the diversity of the environments as well as the size of our models. We trained the DRC architecture in various data regimes, by restricting the number of unique Sokoban levels — during the training, similar to SL, the training algorithm iterates on those limited levels many times. We either train on a Large (900k levels), Medium (10k) or Small (1k) set. For each dataset size, we compared a larger version of the network, DRC(3,3), to a smaller version DRC(1,1).^{2}^{2}2DRC(3,3) has around 300K more parameters, and it requires around 3 times more computation Results are shown in Figure 5.
In all cases, the larger DRC(3,3) network generalized better than its smaller counterpart, both in absolute terms and in terms of generalization gap. In particular, in the Medium regime, the generalization gap^{3}^{3}3We compute the generalization gap by subtracting the performance (ratio of levels solved) on the training set from performance on the test set. is 6.5% for DRC(3,3) versus 33.5% for DRC(1, 1). Figures 5(a), 5(b) shows the results when tested after training on both unfiltered and Medium test sets. We performed an analogous experiment in the Boxworld environment and observed remarkably similar results (see Figure 5(c) and Appendix Figure 13).
Looking across these domains and experiments there are two findings that are of particular note. First, unlike analog SL experiments, reducing the number of training levels does not necessarily improve performance on the train set. Networks trained on 1k levels perform worse in terms of fraction of level solved. We believe this is due to the exploration problem in lowdiversity regime: With more levels, the training agent faces a natural curriculum to help it progress toward harder levels. Another view of this is that larger networks can overfit the training levels, but only if they experience success on these levels at some point. While local minima for the loss in SL are not practically an issue, local minima in policy space can be problematic.
From a classic optimization perspective, a surprising finding is that the larger networks in our experiment (both Sokoban & Boxworld) suffer less from overfitting in the lowdata regime than their smaller counterparts (see Figure 6). However, this is in line with recent findings (Zhang et al., 2016)
in SL that the generalization of a model is driven by the architecture and nature of the data, rather than simply as a results of the network capacity and size of the dataset. Indeed, we also trained the same networks in a purely supervised fashion through imitation learning of an expert policy.
^{4}^{4}4Data was sampled onpolicy from the expert policy executed on levels from the training datasets. We observed a similar result when comparing the classification accuracy of the networks on the test set, with the DRC(3,3) better able to generalize — even though both networks had similar training errors on small datasets.Extrapolation
Another facet of generality in the strategy found by the DRC network is how it performs outside the training distribution. In Sokoban, we tested the DRC(3,3) and DRC(1,1) networks on levels with a larger number of boxes than those seen in the training set. Figure 13(a) shows that DRC was able to extrapolate with little loss in performance to up to 7 boxes (for a a fixed grid size). The performance degradation for DRC(3,3) on 7 boxes was 3.5% and 18.5% for DRC(1,1). In comparison, the results from Racanière et al. (2017) report a loss of 34% when extrapolating to 7 boxes in the same setup.
5 Discussion
We aspire to endow agents with the capacity to plan effectively in combinatorial domains where simple memorization of strategies is not feasible. An overarching question is regarding the nature of planning itself. Can the computations necessary for planning be learned solely using modelfree RL, and can this be achieved by a generalpurpose neural network with weak inductive biases? Or is it necessary to have dedicated planning machinery — either explicitly encoding existing planning algorithms, or implicitly mirroring their structure? In this paper, we studied a variety of different neural architectures trained using modelfree RL in procedural planning tasks with combinatorial and irreversible state spaces. Our results suggest that generalpurpose, highcapacity neural networks based on recurrent convolutional structure, are particularly efficient at learning to plan. This approach yielded stateoftheart results on several domains – outperforming all of the specialized planning architectures that we tested. Our generalization and scaling analyses, together with the procedural nature of the studied domains, suggests that these networks learn an algorithm for approximate planning that is tailored to the domain. The algorithmic function approximator appears to compute its plan dynamically, amortised over many steps, and hence additional thinking time can boost its performance.
Recent work in the context of supervised learning is pushing us to rethink how large neural network models generalize (Zhang et al., 2016; Arpit et al., 2017). Our results further demonstrate the mismatch between traditional views on generalisation and model size. The surprising efficacy of our planning agent, when trained on a small number of scenarios across a combinatorial state space, suggests that any new theory must take into account the algorithmic function approximation capabilities of the model rather than simplistic measures of its complexity. Ultimately, we desire even more generality and scalability from our agents, and it remains to be seen whether modelfree planning will be effective in reinforcement learning environments of realworld complexity.
References
 Arpit et al. (2017) Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. arXiv preprint arXiv:1706.05394, 2017.
 Asadi et al. (2018) Asadi, K., Misra, D., and Littman, M. L. Lipschitz continuity in modelbased reinforcement learning. arXiv preprint arXiv:1804.07193, 2018.
 Bellemare et al. (2013) Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
 Botea et al. (2003) Botea, A., Müller, M., and Schaeffer, J. Using abstraction for planning in sokoban. In Computers and Games, volume 2883, pp. 360, 2003.
 Buesing et al. (2018) Buesing, L., Weber, T., Racaniere, S., Eslami, S., Rezende, D., Reichert, D. P., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006, 2018.
 Cobbe et al. (2018) Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. Quantifying generalization in reinforcement learning. arXiv preprint arXiv:1812.02341, 2018.
 Ebert et al. (2018) Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A., and Levine, S. Visual foresight: Modelbased deep reinforcement learning for visionbased robotic control. arXiv preprint arXiv:1812.00568, 2018.
 Espeholt et al. (2018) Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deeprl with importance weighted actorlearner architectures. arXiv preprint arXiv:1802.01561, 2018.
 Farquhar et al. (2017) Farquhar, G., Rocktäschel, T., Igl, M., and Whiteson, S. Treeqn and atreec: Differentiable tree planning for deep reinforcement learning. In ICLR, 2017.
 Graves (2016) Graves, A. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016.
 Greff et al. (2016) Greff, K., Srivastava, R. K., and Schmidhuber, J. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016.
 Guez et al. (2018) Guez, A., Weber, T., Antonoglou, I., Simonyan, K., Vinyals, O., Wierstra, D., Munos, R., and Silver, D. Learning to search with mctsnets. arXiv preprint arXiv:1802.04697, 2018.

He et al. (2016)
He, K., Zhang, X., Ren, S., and Sun, J.
Deep residual learning for image recognition.
In
Proceedings of the IEEE conference on computer vision and pattern recognition
, pp. 770–778, 2016.  Hochreiter & Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J. Long shortterm memory. Neural computation, 9(8):1735–1780, 1997.
 Horgan et al. (2018) Horgan, D., Quan, J., Budden, D., BarthMaron, G., Hessel, M., Van Hasselt, H., and Silver, D. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018.
 Hu et al. (2017) Hu, J., Shen, L., and Sun, G. Squeezeandexcitation networks. arXiv preprint arXiv:1709.01507, 7, 2017.
 Jastrzebski et al. (2017) Jastrzebski, S., Arpit, D., Ballas, N., Verma, V., Che, T., and Bengio, Y. Residual connections encourage iterative inference. arXiv preprint arXiv:1710.04773, 2017.
 Kingma & Ba (2014) Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
 Lee et al. (2018) Lee, L., Parisotto, E., Chaplot, D. S., Xing, E., and Salakhutdinov, R. Gated path planning networks. arXiv preprint arXiv:1806.06408, 2018.
 Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Humanlevel control through deep reinforcement learning. Nature, 518(7540):529, 2015.

Mnih et al. (2016)
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T.,
Silver, D., and Kavukcuoglu, K.
Asynchronous methods for deep reinforcement learning.
In
International conference on machine learning
, pp. 1928–1937, 2016.  Oh et al. (2017) Oh, J., Singh, S., and Lee, H. Value prediction network. In Advances in Neural Information Processing Systems, pp. 6118–6128, 2017.
 Orseau et al. (2018) Orseau, L., Lelis, L., Lattimore, T., and Weber, T. Singleagent policy tree search with guarantees. In Advances in Neural Information Processing Systems, 2018.
 Pang & Werbos (1998) Pang, X. and Werbos, P. Neural network design for j function approximation in dynamic programming. arXiv preprint adaporg/9806001, 1998.
 Racanière et al. (2017) Racanière, S., Weber, T., Reichert, D., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., et al. Imaginationaugmented agents for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 5690–5701, 2017.

Schmidhuber (1990)
Schmidhuber, J.
Making the world differentiable: On using selfsupervised fully recurrent neural networks for dynamic reinforcement learning and planning in nonstationary environments.
1990.  Silver et al. (2016) Silver, D., van Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., DulacArnold, G., Reichert, D., Rabinowitz, N., Barreto, A., et al. The predictron: Endtoend learning and planning. arXiv preprint arXiv:1612.08810, 2016.
 Sutton et al. (1998) Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction. 1998.
 Talvitie (2014) Talvitie, E. Model regularization for stable sample rollouts. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, pp. 780–789. AUAI Press, 2014.
 Tamar et al. (2016) Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value iteration networks. In Advances in Neural Information Processing Systems, pp. 2154–2162, 2016.
 Wang et al. (2018) Wang, J. X., KurthNelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., and Botvinick, M. Prefrontal cortex as a metareinforcement learning system. Nature neuroscience, 21(6):860, 2018.
 Xingjian et al. (2015) Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., and Woo, W.c. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802–810, 2015.
 Zambaldi et al. (2018) Zambaldi, V., Raposo, D., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D., Lillicrap, T., Lockhart, E., et al. Relational deep reinforcement learning. arXiv preprint arXiv:1806.01830, 2018.
 Zhang et al. (2018a) Zhang, A., Ballas, N., and Pineau, J. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937, 2018a.
 Zhang et al. (2016) Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
 Zhang et al. (2018b) Zhang, C., Vinyals, O., Munos, R., and Bengio, S. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893, 2018b.
Appendix
The appendix is organized as follows: we first describe the five environments we used in this paper. We then provide details about the dataset generation process used for Sokoban dataset that we are releasing. Next, we describe the DRC architecture, various choices of memory type, and all the implementation details. We also list the parameters and experimental setup for our DRC all our models. We then present some extended experiments and extension of our generalization analysis, followed by ablation experiments for DRC and finally we compare the DRC against various baseline agents (VIN, ATreeC, and ResNet).
Appendix A Domains
a.1 Sokoban
We follow the reward structure in (Racanière et al., 2017), with a reward of 10 for completing a level, 1 for getting a box on a target, and 1 for removing a box from a target, in addition to a cost per timestep of 0.01. Each episode is capped at 120 frames. We use a spritebased representation of the state as input. Each sprite is an (8, 8, 3)shaped RGB image.
Dataset generation is described in Appendix B. Unless otherwise noted, we have used the levels from the unfiltered dataset in experiments.
a.2 MiniPacman
MiniPacman is a simplified version of the popular game PacMan. The first, deterministic version, was introduced in (Racanière et al., 2017)
. Here we use a stochastic version, where at every timestep, each ghost moves with a probability of
. This simulates, in a gridworld, the fact that ghosts move slower than the player. An implementation of this game is available under the name Pill Eater at https://github.com/vasiloglou/mltrainnips2017/tree/master/sebastien_racaniere, where different reward structures (or modes) are available.a.3 Boxworld
In all our experiments we use branch length of 3 and maximum solution length of 4. Each episode is capped at 120 frames. Unlike Sokoban, Boxworld levels are generated by the game engine. Unlike Sokoban we do not used spritebased images; the environment state is provided as a (14, 14, 3) tensor.
a.4 Atari
We use the standard Atari setup of (Mnih et al., 2015) with up to 30 noops at random, episodes capped at 30 mins, an action repeat of , and the full set of
actions. We made little effort to tune the hyperparameters for Atari; only the encoder size was increased. See Appendix
E.2 for details about the experiments, and Figure 10 for the learning curves.a.5 Gridworld
We generate 32x32shaped levels by sampling a number of obstacles between 12 and 24. Each obstacle is a square, with side in [2, 10]. These obstacles may overlap. The player and goal are placed on random different empty squares. We use rejection sampling to ensure the player can reach the goal. Episodes terminate when reaching a goal or when stepping on an obstacle, with a reward of 1 and 1 respectively. There is a small negative reward of 0.01 at each step otherwise.
Appendix B Dataset generation details
The unfiltered Sokoban levels are generated by the procedure described in (Racanière et al., 2017). Simple hashing is used to avoid repeats. To generate medium levels, we trained a DRC(1,1) policy whose LSTM state is reset at every time step. It achieves around 95% of level solved on the easy set. Then, we generate medium difficulty levels by rejection sampling: we sample levels using the procedure from (Racanière et al., 2017), and accept a level if the policy cannot solve it after 10 attempts. The levels are not subsampled from the easy levels dataset but from fresh generation instead, so overlap between easy and medium is minimal (though there are 120 levels in common in the easy and medium datasets). To generate hard levels, we train a DRC(3,3) policy on a mixture of easy and medium levels, until it reached a performance level of around 85%. We then select the first 3332 levels of a separate medium dataset which are not solved by that policy after 10 attempts (so there is no overlap between hard and medium sets). The datasets for all level types are available at https://github.com/deepmind/boxobanlevels.
Unfiltered  Medium  Hard 

99%  95%  80% 
Unfiltered train  Unfiltered test  Medium train  Medium test  hard  

Unfiltered train  900,000  0  119  10  0 
Unfiltered test  0  100,000  10  1  0 
Medium train  119  1  450,000  0  0 
Medium test  10  0  0  50,000  0 
Hard  0  0  0  0  3332 
Appendix C Model
We denoted as the computation at a full timestep of the DRC(D, N) architecture. Here are the parameters of stacked memory modules, and is the agent’s encoded observation.
Let be the state of the stacked memory modules at the ticks at timestep . Each . We then have the "repeat" recurrence below:
(2) 
We can now describe the computation within a single tick, i.e., outputs and at for a fixed . This is the "stack" recurrence:
(3) 
Note that the encoded observation is fed as an input not only at all ticks 1, …, N, but also at all depths 1, …, D of the memory stack. (The latter is described in Section 2.1.2 but not depicted in Figure 1 for clarity.)
Generally each memory module is a ConvLSTM parameterized by at location (d, n) of the DRC grid. But we describe alternative choices of memory module in Appendix C.2.
c.1 Additional details/improvements
c.1.1 Topdown skip connection
As described in the stack recurrence, we feed the hidden state of each memory module as an input to the next module in the stack. But this input is only available for modules at depth . Instead of setting , we use a topdown skip connection i.e. . We will explore the role of this skipconnection in Appendix F.
c.1.2 Poolandinject
The pool operation aggregates hidden states across their spatial dimensions to give vectors
. We project these through a linear layer each (weights ), and then tile them over space to obtain summary tensors . These have the same shapes as the original hidden states, and can be injected as additional inputs to the memory modules at the next tick.(4) 
Finally, is provided as an additional input at location (d,n) of the DRC, and equation C becomes:
c.2 Memory modules
c.2.1 ConvLSTM
For all :
For , we use the topdown skip connection in place of as described above.
Here denotes the convolution operator, and denotes pointwise multiplication. Note that parameterizes the computation of the forget gate, input gate, output gate, and new cell state. and should not be confused with the encoded input or the final output of the entire network.
c.2.2 GatedConvRNN
This is a simpler module with no cell state.
For all :
has half as many parameters as in the case of the ConvLSTM.
c.2.3 SimpleConvRNN
This is a classic RNN without gating.
has half as many parameters as the GatedConvRNN, and only a quarter as the ConvLSTM.
We will explore the difference between these memory types in Appendix F. Otherwise, we use the ConvLSTM for all experiments.
Appendix D HyperParameters
We tuned our hyperparameters for the DRC models on Sokoban and used the same settings for all other domains. We only changed the encoder network architectures to accommodate the specific observation format of each domain.
d.1 Network parameters
All activation functions are ReLU unless otherwise specified.
Observation size

Sokoban: (80, 80, 3)

Boxworld: (14, 14, 3)

MiniPacman: (15, 19, 3)

Atari: (210, 160, 3)
Encoder network is a multilayer CNN with following parameters for each domain:

Sokoban: number of channels: (32, 32), kernel size: (8, 4), strides: (4, 2)

BoxWorld: number of channels: (32, 32), kernel size: (3, 2), strides: (1, 1)

MiniPacman: number of channels: (32, 32), kernel size: (3, 3), strides: (1, 1)

Gridworld: number of channels: (64, 64, 32), kernel size: (3, 3, 2), strides: (1, 1, 2)

Atari: number of channels: (16, 32, 64, 128, 64, 32), kernel size: (8, 4, 4, 3, 3, 3), strides: (4, 2, 2, 1, 1, 1)
DRC(D, N): all ConvLSTM modules use one convolution with 128 output channels, a kernel size of 3 and stride of 1. All cell states and hidden states have 32 channels.
1D LSTM(D, N): all LSTM modules have hidden size 200.
Policy: single hidden layer MLP with 256 units.
Total number of parameters for all our models are shown in Figure 7.
Model  Number of parameters 

DRC(3,3)  2,023,174 
DRC(1,1)  1,746,054 
1D LSTM (3,3)  2,151,790 
CNN  2,089,222 
ResNet for Sokoban  52,584,982 
ResNet for Boxworld  3,814,197 
ResNet for MiniPacman  2,490,902 
I2A  7,781,269 
VIN  61,224 
ATreeC  3,269,063 
d.2 RL training setup
We use the Vtrace actorcritic algorithm described by (Espeholt et al., 2018)
, with 4 GPUs for each learner and 200 actors generating trajectories. We reduce variance and improve stability by using
returns targets () and a smaller discount factor (). This marginally reduces the maximum performance observed, but increases the stability and average performance across runs, allowing better comparisons. For all experiments, we use a BPTT (Backpropagation Through Time) unroll of length
and a batch size of . We use the Adam optimizer (Kingma & Ba, 2014). The learning rate is initialized to 4e4 and is annealed to 0 over 1.5e9 environment steps with polynomial annealing. The other Adam optimizer parameters are 1e4. The entropy and baseline loss weights are set to 0.01 and 0.5 respectively. We also apply anorm cost with a weight of 1e3 on the logits, and a
regularization cost with a weight of 1e5 to the linear layers that compute the baseline value and logits.Appendix E Extra experiments
e.1 Sokoban
Figure 9 shows learning curve when different DRC architectures have been trained on the medium set as opposed to unfiltered set in the main text.
e.2 Atari
We train DRC(3,3) and DRC(1,1) on five Atari 2600 games: Alien, Asteroid, Breakout, Ms. PacMan, and Up’n Down. We picked these for their planning focus. Figure 10 shows learning curves comparing DRC(3,3) against DRC(1,1) and ApeXDQN (Horgan et al., 2018) on the above games. We improve on ApeXDQN on three out of five games.
Our scores generally improve with more training. For example, if we let the DRC(3,3) run for longer, it reaches scores above 75000 in Ms. Pacman.
Appendix F Extra ablation studies
f.1 Shortcut
There is a residual or shortcut connection for the encoder module to the output of the ConvLSTM module. Here we show the effect of this connection, which is a minor improvemnt.
f.2 Memory type
The ConvLSTM is the topperforming memory module among those we tested. The Gated ConvRNN module comes out very close with half as many parameters. We surmise that the gap between these two types could be larger for other domains.
The Simple ConvRNN suffers from instability and high variance as expected, asserting that some gating is essential for the DRC’s performance.
f.3 Additional improvements
Without poolandinject, the network learns significantly slower at early stages but nevertheless converges to the same performance as the baseline. The vision shortcut seems to have little influence on the baseline model. This is likely because we already feed the encoded observation to every (d,n)location of the DRC grid (see Appendix C). Without the topdown skip connection, the model exhibits larger performance variance and also slightly lower final performance. On the whole, none of these are critical to the model’s performance.
Appendix G Generalization
For generalization experiments in Boxworld we generate levels online for each levelset size using seeds. Large approximately corresponds to 22k levels, medium to 5k, and small to 800. But at each iteration it’s guaranteed that same levels are generated. Figure 13 shows results of similar experiment as Figure 5 but for BoxWorld domain.
Figure 13(a) shows the extrapolation results on Sokoban when a model that is trained with 4 boxes is tested on levels with larger number of boxes. Performance degradation for DRC(3,3) is very minimal and the models is still able to perform with performance of above 90% even on the levels with 7 boxes, whereas the performance on for DRC(1,1) drops to under 80%.
Figure 13(b) shows the generalization gap for Sokoban. Generalization gap is computed by the difference of performance (ratio of the levels solved) between train set and test set. The gap increase substantially more for DRC(1,1) compared to DRC(3,3) as the training set size decreases.
Appendix H Baseline architectures and experiments
h.1 Value Iteration Networks
Our implementation of VIN closely follows (Tamar et al., 2016). This includes using knowledge of the player position in the mechanism for attention over the qvalues produced by the VIN module in the case of Gridworld.
For Sokoban, which provides pixel observations, we feed an encoded frame to the VIN module. The convolutional encoder is identical to that used by our DRC models (see Appendix D.1). We also couldn’t use the player position directly; instead we use a learned soft attention mask over the 2D state space, and sum over the attended values to obtain an attentionmodulated value per abstract action .
We introduce an additional choice of update rule in our experiments. The original algorithm sets at each value iteration. We also try the following alternative: (note that ’’ denotes concatenation along the feature dimension and ’’ denotes 2D convolution). Here, is a discounting parameter independent of the MDP’s discount factor. In both cases, and . Although these equations denote a single convolutional operation in the computation of , , and the attention map A, in practice we use two convolutional layers in each case with a relu activation in between.
We performed parameter sweeps over the choice of update rules (including ), learning rate initial value, value iteration depth, number of abstract actions, and intermediate reward channels. The annealing schedule for the learning rate was the same as described in Appendix D.2. In total we had 64 parameter combinations for Gridworld, and 102 for Sokoban. We show the average of the top five performing runs from each sweep in Figures 14(a) and 14(b).
h.2 ATreeC
We implement and use the actorcritic formulation of (Farquhar et al., 2017). We feed it similarly encoded observations from Sokoban as in our DRC and VIN setups. This is flattened and passed through a fully connected layer before the tree planning and backup steps. We do not use any auxiliary losses for reward or state grounding.
We swept over the learning rate initial value, tree depth (2 or 3), embedding size (128, 512), choice of value aggregation (max or softmax), and TDlambda (0.8, 0.9). We average the top five bestperforming runs from the sweep in Figure 14(b).
h.3 ResNet
We experiment with different ResNet architectures for each domain and choose one that performs best.
Sokoban Each layer was constructed by one CNN layer that could be potentially used for downsampling using the stride parameter, and two residual blocks on top which consisted of one CNN layer. We used 9 of the described layers with (32, 32, 64, 64, 64, 64, 64, 64, 64) channels, (8, 4, 4, 4, 4, 4, 4, 4, 4) kernel shapes and (4, 2, 1, 1, 1, 1, 1, 1, 1) strides for the downsampling CNN. And the output flattened and passed to 1 hidden layer MLP with 256 units, before computing the policy and baseline.
Boxworld We had a three layer CNN with (16, 32, 64) channels, kernel shape of 2 and stride of 1. And on top of that 8 ResNet blocks. Each blocks was consisted of the 2 layer CNN with 64 channels, kernel shape of 3 and stride 1. And the output flattened and passed to 1 hidden layer MLP with 256 units, before computing the policy and baseline.
MiniPacman We used the same model architecture as the one used for Boxworld with only difference being the initial CNN layer channels were (16, 32, 32) and ResNet blocks had 32 channels.
h.4 Cnn
The CNN network we use is similar to the encoder network for DRC with more layers. It has (32, 32, 64, 64, 64, 64, 64, 64, 64) channels, (8, 4, 4, 4, 4, 4, 4, 4, 4) kernel sizes and strides of (4, 2, 1, 1, 1, 1, 1, 1, 1).
h.5 I2a
We use the I2A implementation available at https://github.com/vasiloglou/mltrainnips2017/tree/master/sebastien_racaniere with the following modifications:

We replace the FrameProcessing class with a layer CNN with kernel sizes , strides and channels ; the output of the last convolution is then flattened and passed through a linear layer with outputs.

The modelfree path passed to the I2A class was a FrameProcessing as describe above.

The RolloutPolicy is a layer CNN with kernel sizes , strides and channels ; the output of the last convolution is flattened and passed through an MLP with one hidden layer of size , and output size (the number of actions in Sokoban).

The environment model is similar to the one used in (Racanière et al., 2017)
. It consists of a deterministic model that given a frame and onehot encoded action, outputs a predicted next frame and reward. The input frame is first reduced in size using a convolutional layer with kernel size
, stride and channels. The action is then combined with the output of this layer using poolandinject (see (Racanière et al., 2017)). This is followed by size preserving residual blocks with convolutions of shape . The output of the last residual block is then passed to two separate networks to predict frame and reward. The frame prediction head uses a convolution layer with shape , stride , followed by a deconvolution with stride , kernel shape and channels. The reward prediction head uses a convolution layer with shape , stride , followed by a linear layer with output size . We predict rewards binned in categories (less than , between and greater than ), reducing it to a classification problem. (Racanière et al., 2017).
Appendix I Details about figures
Unless otherwise noted, each curve is the average of five independent runs (with identical parameters). We also show a 95% confidence interval (using a shaded area) around each averaged curve. Before independent runs are averaged, we smoothen them with a window size of 5 million steps. Steps in RL learning curves refer to the total number of environment steps seen by the actors.