Findings in developmental psychology have revealed that infants are endowed with a small number of separable systems of core knowledge for reasoning about objects, actions, number, space, and possibly social interactions (Spelke and Kinzler, 2007). These systems enable infants to learn many skills and acquire knowledge rapidly. The most coherent explanation of this phenomenon is that the learning (or optimization) process of evolution has led to the emergence of components that enable fast and varied forms of learning. In psychology, learning to learn has a long history (Ward, 1937; Harlow, 1949; Kehoe, 1988).
Inspired by this, many researchers have attempted to build agents capable of learning to learn (Schmidhuber, 1987; Naik and Mammone, 1992; Thrun and Pratt, 1998; Hochreiter et al., 2001; Santoro et al., 2016; Duan et al., 2016; Wang et al., 2016; Ravi and Larochelle, 2017; Li and Malik, 2017)
. The scope of research under the umbrella of learning to learn is very broad. The learner can implement and be trained by many different algorithms, including gradient descent, evolutionary strategies, simulated annealing, and reinforcement learning.
. In the former, one uses supervised learning at the meta-level to learn an algorithm for supervised learning, while in the latter, one uses supervised learning at the meta-level to learn an algorithm for unsupervised learning.
Learning to learn can be used to learn both models and algorithms. In Andrychowicz et al. (2016) the output of meta-learning is a trained recurrent neural network (RNN), which is subsequently used as an optimization algorithm to fit other models to data. In contrast, in Zoph and Le (2017) the output of meta-learning can also be an RNN model, but this new RNN is subsequently used as a model that is fit to data using a classical optimizer. In both cases the output of meta-learning is an RNN, but this RNN is interpreted and applied as a model or as an algorithm. In this sense, learning to learn with neural networks blurs the classical distinction between models and algorithms.
In this work, the goal of meta-learning is to produce an algorithm for global black-box optimization. Specifically, we address the problem of finding a global minimizer of an unknown (black-box) loss function. That is, we wish to compute where is some search space of interest. The black-box function is not available to the learner in simple closed form at test time, but can be evaluated at a query point in the domain. This evaluation produces either deterministic or stochastic outputs such that . In other words, we can only observe the function through unbiased noisy point-wise observations .
. It is a sequential model-based decision making approach with two components. The first component is a probabilistic model, consisting of a prior distribution that captures our beliefs about the behavior of the unknown objective function and an observation model that describes the data generation mechanism. The model can be a Beta-Bernoulli bandit, a random forest, a Bayesian neural network, or a Gaussian process (GP)(Shahriari et al., 2016). Bayesian optimization is however often associated with GPs, to the point of sometimes being referred to as GP bandits (Srinivas et al., 2010).
The second component is an acquisition function, which is optimized at each step so as to trade-off exploration and exploitation. Here again we encounter a huge variety of strategies, including Thompson sampling, information gain, probability of improvement, expected improvement, upper confidence bounds(Shahriari et al., 2016). The requirement for optimizing the acquisition function at each step can be a significant cost, as shown in the empirical section of this paper. It also raises some theoretical concerns (Wang et al., 2014).
In this paper, we present a learning to learn approach for global optimization of black-box functions and contrast it with Bayesian optimization. In the meta-learning phase, we use a large number of differentiable functions generated with a GP to train RNN optimizers by gradient descent. We consider two types of RNN: long-short-term memory networks (LSTMs) byHochreiter and Schmidhuber (1997) and differentiable neural computers (DNCs) by Graves et al. (2016).
During meta-learning, we choose the horizon (number of steps) of the optimization process. We are therefore considering the finite horizon setting that is popular in AB tests (Kohavi et al., 2009; Scott, 2010) and is often studied under the umbrella of best arm identification in the bandits literature (Bubeck et al., 2009; Gabillon et al., 2012).
The RNN optimizer learns to use its memory to store information about previous queries and function evaluations, and learns to access its memory to make decisions about which parts of the domain to explore or exploit next. That is, by unrolling the RNN, we generate new candidates for the search process. The experiments will show that this process is much faster than applying standard Bayesian optimization, and in particular it does not involve either matrix inversion or optimization of acquisition functions.
In the experiments we also investigate distillation of acquisition functions to guide the process of training the RNN optimizers, and the use of parallel optimization schemes for expensive training of deep networks.
The experiments show that the learned optimizers can transfer to optimize a large and diverse set of black-box functions arising in global optimization, control, and hyper-parameter tuning. Moreover, withing the training horizon, the RNN optimizers are competitive with state-of-the-art heavily engineered packages such as Spearmint, SMAC and TPE (Snoek et al., 2014; Hutter et al., 2011a; Bergstra et al., 2011)
2 Learning Black-box Optimization
A black-box optimization algorithm can be summarized by the following loop:
Given the current state of knowledge propose a query point
Observe the response
Update any internal statistics to produce
This easily maps onto the classical frameworks presented in the previous section where the update step computes statistics and the query step uses these statistics for exploration. In this work we take this framework as a starting point and define a combined update and query rule using a recurrent neural network parameterized by such that
Intuitively this rule can be seen to update its hidden state using data from the previous time step and then propose a new query point. In what follows we will apply this RNN, with shared parameters, to many steps of a black-box optimization process. An example of this computation is shown in Figure 1. Additionally, note that in order to generate the first query we arbitrarily set the initial “observations” to dummy values and ; this is a point we will return to in Section 2.3.
2.1 Loss Function
Given this rule we now need a way to learn the parameters
with stochastic gradient descent for any given distribution of differentiable functions. Perhaps the simplest loss function one could use is the loss of the final iteration: for some time-horizon . This loss was considered by Andrychowicz et al. (2016) in the context of learning first-order optimizers, but ultimately rejected in favor of the summed loss
A key reason to prefer is that the amount of information conveyed by is temporally very sparse. By instead utilizing a sum of losses to train the optimizer we are able to provide information from every step along this trajectory. Although at test time the optimizer typically only has access to the observation , at training time the true loss can be used. Note that optimizing the summed loss is equivalent to finding a strategy which minimizes the expected cumulative regret. Finally, while in many optimization tasks the loss associated with the best observation is often desired, the cumulative regret can be seen as a proxy for this quantity.
By using the above objective function we will be encouraged to trade off exploration and exploitation and hence globally optimize the function . This is due to the fact that in expectation, any method that is better able to explore and find small values of will be rewarded for these discoveries. However the actual process of optimizing the above loss can be difficult due to the fact that nothing explicitly encourages the optimizer itself to explore.
We can encourage exploration in the space of optimizers by encoding an exploratory force directly into the meta learning loss function. Many examples exist in the bandit and Bayesian optimization communities, for example
where is the expected posterior improvement of querying given observations up to time . This can encourage exploration by giving an explicit bonus to the optimizer rather than just implicitly doing so by means of function evaluations. Alternatively, it is possible to use the observed improvement (OI)
We also studied a loss based on GP-UCB (Srinivas et al., 2010) but in preliminary experiments this did not perform as well as the EI loss and is thus not included in the later experiments.
The illustration of Figure 1 shows the optimizer unrolled over many steps, ultimately culminating in the loss function. To train the optimizer we will simply take derivatives of the loss with respect to the RNN parameters and perform stochastic gradient descent (SGD). In order to evaluate these derivatives we assume that derivatives of
can be computed with respect to its inputs. This assumption is made only in order to backpropagate errors from the loss to the parameters, but crucially isnot needed at test time. If the derivatives of are also not available at training time then it would be necessary to approximate these derivatives via an algorithm such as REINFORCE (Williams, 1992).
2.2 Training Function Distribution
To this point we have made no assumptions about the distribution of training functions . In this work we are interested in learning general-purpose black-box optimizers, and we desire our distribution to be quite broad.
As a result we propose the use of GPs as a suitable training distribution. Under the GP prior, the joint distribution of function values at any finite set of query points follows a multivariate Gaussian distribution(Rasmussen and Williams, 2006)
, and we generate a realization of the training function incrementally at the query points using the chain rule with a total time complexity offor every function sample.
The use of functions sampled from a GP prior also provides functions whose gradients can be easily evaluated at training time as noted above. Further, the posterior expected improvement used within can be easily computed (Močkus, 1982) and differentiated as well. Search strategies based on GP losses, such as , can be thought of as a distilled strategies. The major downside of search strategies which are based on GP inference is their cubic complexity.
While training with a GP prior grants us the convenience to assess the efficacy of our training algorithm by comparing head-to-head with GP-based methods, it is worth noting that our model can be trained with any distribution that permits efficient sampling and function differentiation. The flexibility could become useful when considering problems with specific prior knowledge and/or side information.
2.3 Parallel Function Evaluation
The use of parallel function evaluation is a common technique in Bayesian optimization, often used for costly, but easy to simulate functions. For example, as illustrated in the experiments, when searching for hyper-parameters of deep networks, it is convenient to train several deep networks in parallel.
Suppose we have
workers, and that the process of proposing candidates for function evaluation is much faster than evaluating the functions. We augment our RNN optimizer’s input with a binary variableas follows:
For the first steps, we set , arbitrarily set the inputs to dummy values and , and generate parallel queries . As soon as a worker finishes evaluating a query, the query and its evaluation are fed back to the network by setting , resulting in a new query . Figure 2 displays a single iteration of this algorithm.
A well-trained optimizer must learn to condition on in order to either generate initial queries or generate queries based on past observations. Another key point to consider is that the batch nature of the optimizer can result in the ordering of queries being permuted: i.e. although is proposed before it is entirely plausible that is evaluated first. In order to account for this at training time and not allow the optimizer to rely on a specific ordering, we simulate a runtime associated with the -th query. Observations are then made based on the order in which they complete.
It is worth noting that the sequential setting is a special case of this parallel policy where and every observation is made with . Note also that we have kept the number of workers fixed for simplicity of explanation only. The architecture allows for the number of workers to vary.
It is instructive to contrast this strategy with what is done in parallel Bayesian optimization (Desautels et al., 2014; Snoek et al., 2012). There care must be taken to ensure that a diverse set of queries are utilized—in the absence of additional data the standard sequential strategy would propose queries at the same point. Exactly computing the optimal
-step query is typically intractable, and as a result hand-engineered heuristics are employed. Often this involves synthetically reducing the uncertainty associated with outstanding queries in order to simulate later observations. In contrast, our RNN optimizer can store in its hidden state any relevant information about outstanding observations. Decisions about what to store are learned during training and as a result should be more directly related to later losses.
We present several experiments that show the breadth of generalization that is achieved by our learned algorithms. We train our algorithms to optimize very simple functions—samples from a GP with a fixed length scale—and show that the learned algorithms are able to generalize from these simple objective functions to a wide variety of other test functions that were not seen during training.
We experimented with two different RNN architectures: LSTMs and DNCs. However, we found the DNCs to perform slightly (but not significantly) better. For clarity, we only show plots for DNCs in most of the figures.
We train each RNN optimizer with trajectories of steps, and update the RNN parameters using BPTT with Adam. We use a curriculum to increase the length of trajectories gradually from to . We repeat this process for each of the loss functions discussed in Section 2. Hyper-parameters for the RNN optimization algorithm (such as learning rate, number of hidden units, and memory size for the DNC models) are found through grid search during training. When ready to be used as an optimizer, the RNN requires neither tuning of hyper-parameters nor hand-engineering. It is fully automatic.
In the following experiments, DNC sum refers to the DNC network trained using the summed loss , DNC OI to the network trained using the loss , and DNC EI to the network trained with the loss .
We compare our learning to learn approach with popular state-of-the-art Bayesian optimization packages, including Spearmint with automatic inference of the GP hyper-parameters and input warping to deal with non-stationarity (Snoek et al., 2014), Hyperopt (TPE) (Bergstra et al., 2011), and SMAC (Hutter et al., 2011b). For test functions with integer inputs, we treat them as piece-wise constant functions and round the network output to the closest values. We evaluate the performance at a given search step , according to the minimum observed function value up to step , .
3.1 Performance on Functions Sampled from the Training Distribution
We first evaluate performance on functions sampled from the training distribution. Notice, however, that these functions are never observed during training. Figure 3 shows the best observed function values as a function of search step , averaged over sampled functions for RNN models and 100 sampled functions for other models (we can afford to do more for RNNs because they are very fast optimizers). For Spearmint, we consider both the default setting with a prior distribution that estimates the GP hyper-parameters by Monte Carlo and a setting with the same hyper-parameters as those used in training. For the second setting, Spearmint knows the ground truth and thus provides a very competitive baseline. As expected Spearmint with a fixed prior proves to be one of the best models under most settings. When the input dimension is 6 or higher, however, neural network models start to outperform Spearmint. We suspect it is because in higher dimensional spaces, the RNN optimizer learns to be more exploitative given the fixed number of iterations. Among all RNNs, those trained with expected/observed improvement perform better than those trained with direct function observations.
Figure 4 shows the query trajectories , , for different black-box optimizers in a one-dimensional example. All of the optimizers explore initially, and later settle in one mode and search more locally. The DNCs trained with EI behave most similarly to Spearmint. DNC with direct function observations (DNC sum) tends to explore less than the other optimizers and often misses the global optimum, while the DNCs trained with the observed improvement (OI) keep exploring even in later stages.
3.2 Transfer to Global Optimization Benchmarks
We compare the algorithms on four standard benchmark functions for black-box optimization with dimensions ranging from 2 to 6. To obtain a more robust evaluation of the performance of each model, we generate multiple instances for each benchmark function by applying a random translation (–), scaling (–), flipping, and dimension permutation in the input domain.
The lef hand side of Figure 5 shows the minimum observed function values achieved by the learned DNC optimizers, and contrasts these against the ones attained by Spearmint, TPE and SMAC. All methods appear to have similar performance with Spearming doing slightly better in low dimensions. As the dimension increases, we see that the DNC optimizers converge at at a much faster rate within the horizon of steps.
We also observe that DNC OI and DNC EI both outperform DNC with direct obsevations of the loss (DNC sum). It is encouraging that the curves for DNC OI and DNC EI are so close. While DNC EI is distilling a popular acquisition function from the EI literature, the DNC OI variant is much easier to train as it never requires the GP computations necessary to construct the EI acquisition function.
The right hand side of Figure 5 shows that the neural network optimizers run about times faster than Spearmint and times faster than TPE and SMAC with the DNC architecture. There is an additional 5 times speedup when using the LSTM architecture, as shown in Table 1. The negligible runtime of our optimizers suggests new areas of application for global optimization methods that require both high sample efficiency and real-time performance.
3.3 Transfer to a Simple Control Problem
We also consider an application to a simple reinforcement learning task described by (Hoffman et al., 2009). In this problem we simulate a physical system consisting of a number of repellers which affect the fall of particles through a 2D-space. The goal is to direct the path of the particles through high reward regions of the state space and maximize the accumulated discounted reward. The four-dimensional state-space in this problem consists of a particle’s position and velocity. The path of the particles can be controlled by the placement of repellers which push the particles directly away with a force inversely proportional to their distance from the particle. At each time step the particle’s position and velocity are updated using simple deterministic physical forward simulation. The control policy for this problem consists of 3 learned parameters for each repeller: 2d location and the strength of the repeller.
In our experiments we consider a problem with 2 repellers, i.e. 6 parameters. An example trajectory along with the reward structure (contours) and repeller positions (circles) is displayed in Figure 6. We apply the same perturbation as in the previous subsection to study the average performance. The loss (minimal negative reward) of all models are also plotted in Figure 6. Neural network models outperform all the other competitors in this problem.
3.4 Transfer to ML Hyper-parameter Tuning
Lastly, we consider hyper-parameter tuning for machine learning problems. We include the three standard benchmarks in the HPOLib package(Eggensperger et al., 2013): SVM, online LDA, and logistic regression with 3, 3, and 4 hyper-parameters respectively. We also consider the problem of training a 6-hyper-parameter residual network for classification on the CIFAR-100 dataset.
For the first three problems, the objective functions have already been pre-computed on a grid of hyper-parameter values, and therefore evaluation with different random seeds (100 for Spearmint, 1000 for TPE and SMAC) is cheap. For the last experiment, however, it takes at least 16 GPU hours to evaluate one hyper-parameter setting. For this reason, we test the parallel proposal idea introduced in Section 2.3, with 5 parallel proposal mechanisms. This approach is about five times more efficient.
For the first three tasks, our model is run once because the setup is deterministic. For the residual network task, there is some random variation so we consider three runs per method.
The results are shown in Figure 7. The plots report the negative accuracy against number of function evaluations up to a horizon of . The neural network models especially when trained with observed improvement show competitive performance against the engineered solutions. In the ResNet experiment, we also compare our sequential DNC optimizers with the parallel versions with 5 workers. In this experiment we find that the learned and engineered parallel optimizers perform as well if not slightly better than the sequential ones. These minor differences arise from random variation.
4 Conclusions and Future Work
The experiments have shown that up to the training horizon the learned RNN optimizers are able to match the performance of heavily engineered Bayesian optimization solutions, including Spearmint, SMAC and TPE. The trained RNNs rely on neither heuristics nor hyper-parameters when being deployed as black-box optimizers.
The optimizers trained on synthetic functions were able to transfer successfully to a very wide class of black-box functions, associated with GP bandits, control, global optimization benchmarks, and hyper-parameter tuning.
The experiments have also shown that the RNNs are massively faster than other Bayesian optimization methods. Hence, for applications involving a known horizon and where speed is crucial, we recommend the use of the RNN optimizers. The parallel version of the algorithm also performed well when tuning the hyper-parameters of an expensive-to-train residual network.
However, the current RNN optimizers also have some shortcomings. Training for very long horizons is difficult. This issue was also documented recently in (Duan et al., 2016)
. We believe curriculum learning should be investigated as a way of overcoming this difficulty. In addition, a new model has to be trained for every input dimension with the current network architecture. While training optimizers for every dimension is not prohibitive in low dimensions, future works should extend the RNN structure to allow a variable input dimension. A promising solution is to serialize the input vectors along the search steps.
- Andrychowicz et al.  M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
- Bengio et al.  Y. Bengio, S. Bengio, J. Cloutier, and J. Gecsei. On the optimization of a synaptic learning rule. In Conference on Optimality in Biological and Artificial Networks, 1992.
- Bergstra et al.  J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, pages 2546–2554, 2011.
- Brochu et al.  E. Brochu, V. M. Cora, and N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Technical Report UBC TR-2009-23 and arXiv:1012.2599v1, Dept. of Computer Science, University of British Columbia, 2009.
- Bubeck et al.  S. Bubeck, R. Munos, and G. Stoltz. Pure exploration in multi-armed bandits problems. In International Conference on Algorithmic Learning Theory, 2009.
- Desautels et al.  T. Desautels, A. Krause, and J. Burdick. Parallelizing exploration-exploitation tradeoffs with Gaussian process bandit optimization. Journal of Machine Learning Research, 2014.
- Duan et al.  Y. Duan, J. Schulman, X. Chen, P. Bartlett, I. Sutskever, and P. Abbeel. Rl: Fast reinforcement learning via slow reinforcement learning. Technical report, UC Berkeley and OpenAI, 2016.
Eggensperger et al. 
K. Eggensperger, M. Feurer, F. Hutter, J. Bergstra, J. Snoek, H. Hoos, and
Towards an empirical foundation for assessing bayesian optimization of hyperparameters.In NIPS workshop on Bayesian Optimization in Theory and Practice, 2013.
- Gabillon et al.  V. Gabillon, M. Ghavamzadeh, and A. Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. In Advances in Neural Information Processing Systems, pages 3212–3220, 2012.
- Graves et al.  A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-BarwiÅska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 2016.
- Harlow  H. F. Harlow. The formation of learning sets. Psychological review, 56(1):51, 1949.
- Hochreiter and Schmidhuber  S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
- Hochreiter et al.  S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87–94. Springer, 2001.
Hoffman et al. 
M. W. Hoffman, H. Kueck, N. de Freitas, and A. Doucet.
New inference strategies for solving Markov decision processes using reversible jump MCMC.In
Uncertainty in Artificial Intelligence, pages 223–231, 2009.
- Hutter et al. [2011a] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In LION, pages 507–523, 2011a.
- Hutter et al. [2011b] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In International Conference on Learning and Intelligent Optimization, pages 507–523. Springer, 2011b.
E. J. Kehoe.
A layered network model of associative learning: learning to learn and configuration.Psychological review, 95(4):411, 1988.
- Kohavi et al.  R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery, 18(1):140–181, 2009.
- Li and Malik  S. Li and J. Malik. Learning to optimize. In International Conference on Learning Representations, 2017.
- Močkus  J. Močkus. The Bayesian approach to global optimization. In Systems Modeling and Optimization, volume 38, pages 473–481. Springer, 1982.
- Naik and Mammone  D. K. Naik and R. Mammone. Meta-neural networks that learn by learning. In International Joint Conference on Neural Networks, volume 1, pages 437–442, 1992.
- Rasmussen and Williams  C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
- Ravi and Larochelle  S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
- Santoro et al.  A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning, 2016.
- Schmidhuber  J. Schmidhuber. Evolutionary Principles in Self-Referential Learning. On Learning how to Learn: The Meta-Meta-Meta…-Hook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
- Scott  S. L. Scott. A modern Bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry, 26(6):639–658, 2010.
- Shahriari et al.  B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2016.
- Snoek et al.  J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pages 2951–2959, 2012.
- Snoek et al.  J. Snoek, K. Swersky, R. S. Zemel, and R. P. Adams. Input warping for Bayesian optimization of non-stationary functions. In International Conference on Machine Learning, 2014.
- Spelke and Kinzler  E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 10(1):89–96, 2007.
- Srinivas et al.  N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning, pages 1015–1022, 2010.
- Thrun and Pratt  S. Thrun and L. Pratt. Learning to learn. Springer Science & Business Media, 1998.
- Wang et al.  J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. arXiv Report 1611.05763, 2016.
- Wang et al.  Z. Wang, B. Shakibi, L. Jin, and N. de Freitas. Bayesian multi-scale optimistic optimization. In AI and Statistics, pages 1005–1014, 2014.
- Ward  L. B. Ward. Reminiscence and rote learning. Psychological Monographs, 49(4), 1937.
- Williams  R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992.
- Zoph and Le  B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.