Optimization is a bottleneck for almost all tasks in machine learning, as well as in many other fields, including engineering, design, operations research, and statistics. Advances in optimization therefore have broad impact. Historically, optimization has been performed using hand-designed algorithms. Recent results in machine learning show that, given sufficient data, well-trained neural networks often outperform hand-tuned approaches on supervised tasks. This raises the tantalizing possibility that neural networks may be able to outperform hand-designed optimizers.
Despite the promise in this approach, previous work on learned RNN optimizers for gradient descent has failed to produce neural network optimizers that generalize to new problems, or that continue to make progress on the problems for which they were meta-trained when run for large numbers of steps (see Figure 2). Current neural network optimizers are additionally too costly in both memory and computation to scale to larger problems.
We address both of these issues. Specifically, we improve upon existing learned optimizers by:
Developing a meta-training set that consists of an ensemble of small tasks with diverse loss landscapes
Introducing a hierarchical RNN architecture with lower memory and compute overhead, and which is capable of capturing inter-parameter dependencies.
Incorporating features motivated by successful hand-designed optimizers into the RNN, so that it can build on existing techniques. These include dynamically adapted input and output scaling, momentum at multiple time scales, and a cross between Nesterov momentum and RNN attention mechanisms.
Improving the meta-optimization pipeline, for instance by introducing a meta-objective that better encourages exact convergence of the optimizer, and by drawing the number of optimization steps during training from a heavy tailed distribution.
2 Related work
Learning to learn has a long history in psychology (Ward, 1937; Harlow, 1949; Kehoe, 1988; Lake et al., 2016). Inspired by it, machine learning researchers have proposed meta-learning techniques for optimizing the process of learning itself. Schmidhuber (1987), for example, considers networks that are able to modify their own weights. This leads to end-to-end differentiable systems which allow, in principle, for extremely general update strategies to be learned. There are many works related to this idea, including (Sutton, 1992; Naik & Mammone, 1992; Thrun & Pratt, 1998; Hochreiter et al., 2001; Santoro et al., 2016).
A series of papers from Bengio et al. (1990, 1992, 1995) presents methods for learning parameterized local neural network update rules that avoid back-propagation. Runarsson & Jonsson (2000) extend this to more complex update models. The result of meta learning in these cases is an algorithm, i.e. a local update rule.
Andrychowicz et al. (2016) learn to learn by gradient descent by gradient descent. Rather than trying to distill a global objective into a local rule, their work focuses on learning how to integrate gradient observations over time in order to achieve fast learning of the model. The component-wise structure of the algorithm allows a single learned algorithm to be applied to new problems of different dimensionality. While Andrychowicz et al. (2016)
consider the issue of transfer to different datasets and model structures, they focus on transferring to problems of the same class. In fact, they report negative results when transferring optimizers, meta-trained to optimize neural networks with logistic functions, to networks with ReLU functions.
, around the same time, but they rely on policy search to compute the meta-parameters of the optimizer. That is, they learn to learn by gradient descent by reinforcement learning.
Zoph & Le (2017) also meta-train a controller RNN, but this time to produce a string in a custom domain specific language (DSL) for describing neural network architectures. An architecture matching the produced configuration (the “child” network) is instantiated and trained in the ordinary way. In this case the meta-learning happens only at the network architecture level.
for 1 and 5-shot learning tasks. They use test error to optimize the meta learner. These tasks have the nice property that the recurrent neural networks only need to be unrolled for a small number of steps.
Wang et al. (2016) show that it is possible to learn to solve reinforcement learning tasks by reinforcement learning. They demonstrate their approach on several examples from the bandits and cognitive science literature. A related approach was proposed by Duan et al. (2016).
Finally, Chen et al. (2016) also learn reinforcement learning, but by supervised meta-training of the meta-learner. They apply their methods to black-box function optimization tasks, such as Gaussian process bandits, simple low-dimensional controllers, and hyper-parameter tuning.
At a high level, a hierarchical RNN is constructed to act as a learned optimizer, with its architecture matched to the parameters in the target problem. The hierarchical RNN’s parameters (called meta-parameters) are shared across all target problems, so despite having an architecture that adapts to the target problem, it can be applied to new problems. At each optimization step, the learned optimizer receives the gradients for every parameter along with some additional quantities derived from the gradients, and outputs an update to the parameters. Figure 1 gives an overview.
3.1 Hierarchical architecture
In order to effectively scale to large problems, the optimizer RNN must stay quite small while maintaining enough flexibility to capture inter-parameter dependencies that shape the geometry of the loss surface. Optimizers that account for this second order information are often particularly effective (e.g. quasi-Newton approaches). We propose a novel hierarchical architecture to enable both low per-parameter computational cost, and aggregation of gradient information and coordination of update steps across parameters (Figure 1). At the lowest level of the hierarchy, we have a small Parameter RNN that receives direct per-parameter (scalar) gradient inputs. One level up, we have an intermediate Tensor RNN that incorporates information from a subset of the Parameter RNNs (where the subsets are problem specific). For example, consider a feedforward fully-connected neural network. There would be a Tensor RNN for each layer of the network, where each layer contains an () weight matrix and therefore Parameter RNNs.
At the highest level of the hierarchy is a Global RNN which receives output from every Tensor RNN. This allows the Parameter RNN to have very few hidden units with larger Tensor and Global RNNs keeping track of problem-level information. The Tensor and Global RNNs can also serve as communication channels between Parameter and Tensor RNNs respectively. The Tensor RNN outputs are fed as biases to the Parameter RNN, and the new parameter state is averaged and fed as input to the Tensor RNN. Similarly, the Global RNN state is fed as a bias to each Tensor RNN, and the output of the Tensor RNNs is averaged and fed as input to the Global RNN (Figure 1).
The architecture used in the experimental results has a Parameter RNN hidden state size of 10, and a Tensor and Global RNN state size of 20 (the architecture used by Andrychowicz et al. (2016) had a two layer RNN for each parameter, with 20 units per layer). These sizes showed the best generalization to ConvNets and other complex test problems. Experimentally, we found that we could make the Parameter RNN as small as 5, and the Tensor RNN as small as 10 and still see good performance on most problems. We also found that the performance decreased slightly even on simple test problems if we removed the Global RNN entirely. We used a GRU architecture (Cho et al., 2014) for all three of the RNN levels.
3.2 Features inspired by optimization literature
The best performing neural networks often have knowledge about task structure baked into their design. Examples of this include convolutional models for image processing (Krizhevsky et al., 2012; He et al., 2016), causal models (RNNs) for modeling causal time series data, and the merging of neural value functions with Monte Carlo tree search in AlphaGo (Silver et al., 2016).
We similarly incorporate knowledge of effective strategies for optimization into our network architecture. We emphasize that these are not arbitrary design choices. The features below are motivated by results in optimization and recurrent network literature. They are also individually important to the ability of the learned optimizer to generalize to new problems, as is illustrated by the ablation study in Section 5.5 and Figure 6.
Let be the loss of the target problem, where is the set of all parameter tensors
(e.g. all weight matrices and bias vectors in a neural network). At each training iteration, each parameter tensor is updated as , where the update step is set by the learned optimizer (Equation 5 below).
3.2.1 Attention and Nesterov Momentum
Nesterov momentum (Nesterov, 1983a) is a powerful optimization approach, where parameter updates are based not on the gradient evaluated at the current iterate , but rather at a location which is extrapolated ahead of the current iterate. Similarly, attention mechanisms have proven extremely powerful in recurrent translation models (Bahdanau et al., 2015), decoupling the iteration of RNN dynamics from the observed portion of the input sequence. Motivated by these successes, we incorporate an attention mechanism that allows the optimizer to explore new regions of the loss surface by computing gradients away (or ahead) from the current parameter position. At each training step the attended location is set as , where the offset is further described by Equation 6 below. Note that the attended location is an offset from the previous parameter location rather than the previous attended location .
The gradient of the loss with respect to the attended parameter values will provide the only input to the learned optimizer, though it will be further transformed before being passed to the hierarchical RNN. For every parameter tensor , .
3.2.2 Momentum on multiple timescales
Momentum with an exponential moving average is typically motivated in terms of averaging away minibatch noise or high frequency oscillations, and is often a very effective feature (Nesterov, 1983b; Tseng, 1998). We provide the learned optimizer with exponential moving averages of the gradients on several timescales, where indexes the timescale of the average. The update equation for the moving average is
where thefor the shortest timescale is output by the RNN, and the remaining timescales each increase by a factor of two from that baseline.
By comparing the moving averages at multiple timescales, the learned optimizer has access to information about how rapidly the gradient is changing with training time (a measure of loss surface curvature), and about the degree of noise in the gradient.
3.2.3 Dynamic input scaling
We would like our optimizer to be invariant to parameter scale. Additionally, RNNs are most easily trained when their inputs are well conditioned, and have a similar scale as their latent state. In order to aid each of these goals, we rescale the average gradients in a fashion similar to what is done in RMSProp (Tieleman & Hinton, 2012), ADAM (Kingma & Ba, 2015), and SMORMS3 (Funk, 2015),
where is a running average of the square average gradient, is the scaled averaged gradient, and the momentum logit for the shortest timescale will be output by the RNN, similar to how the timescales for momentum are computed in the previous section.
It may be useful for the learned optimizer to have access to how gradient magnitudes are changing with training time. We therefore provide as further input a measure of relative gradient magnitudes at each averaging scale . Specifically, we provide the relative log gradient magnitudes,
3.2.4 Decomposition of output into direction and step length
Another aspect of RMSProp and ADAM is that the learning rate corresponds directly to the characteristic step length. This is true because the gradient is scaled by a running estimate of its standard deviation, and after scaling has a characteristic magnitude of 1. The length of update steps therefore scales linearly with the learning rate, but is invariant to any scaling of the gradients.
We enforce a similar decomposition of the parameter updates into update directions and for parameters and attended parameters, with corresponding step lengths and ,
where is the number of elements in the parameter tensor . The directions and are read directly out of the RNN (though see B.1 for subtleties).
Relative learning rate
We want the performance of the optimizer to be invariant to parameter scale. This requires that the optimizer judge the correct step length from the history of gradients, rather than memorizing the range of step lengths that were useful in its meta-training ensemble. The RNN therefore controls step length by outputing a multiplicative (additive after taking a logarithm) change, rather than by outputing the step length directly,
where for stability reasons, the log step length is specified relative to an exponential running average with meta-learned momentum . The attended parameter log step length is related to by a meta-learned constant offset ,
To further force the optimizer to dynamically adapt the learning rate rather than memorizing a learning rate trajectory, the learning rate is initialized from a log uniform distribution fromto . We emphasize that the RNN has no direct access to the learning rate, so it must adjust it based purely on its observations of the statistics of the gradients.
In order to aid in coordination across parameters, we do provide the RNN as an input the relative log learning rate of each parameter, compared to the remaining parameters, .
3.3 Optimizer inputs and outputs
As described in the preceding sections, the full set of Parameter RNN inputs for each tensor are , corresponding to the scaled averaged gradients, the relative log gradient magnitudes, and the relative log learning rate.
The full set of Parameter RNN outputs for each tensor are , corresponding to the parameter and attention update directions, the change in step length, and the momentum logits. Each of the outputs in is read out via a learned affine transformation of the Parameter RNN hidden state. The readout biases are clamped to 0 for and . The RNN update equations are then:
where is the hidden state for each level of the RNN, as described in Section 3.1, and and are learned weights of the affine transformation from the lowest level hidden state to output.
3.4 Compute and memory cost
The computational cost of the learned optimizer is , where is the minibatch size, is the total number of parameters, is the number of parameter tensors, and , , and are the latent sizes for Parameter, Tensor, and Global RNNs respectively. Typically, we are in the regime where , in which case the computational cost simplifies to . Note that as the minibatch size is increased, the computational cost of the learned optimizer approaches that of vanilla SGD, as the cost of computing the gradient dominates the cost of computing the parameter update.
The memory cost of the learned optimizer is , which similarly to computational cost typically reduces to . So long as the latent size of the Parameter RNN can be kept small, the memory overhead will also remain small.
We show experimental results for computation time in Section 5.6.
The RNN optimizer is meta-trained by a standard optimizer on an ensemble of target optimization tasks. We call this process meta-training, and the parameters of the RNN optimizer the meta-parameters.
4.1 Meta-training set
Previous learned optimizers have failed to generalize beyond the problem on which they were meta-trained. In order to address this, we meta-train the optimizer on an ensemble of small problems, which have been chosen to capture many commonly encountered properties of loss landscapes and stochastic gradients. By meta-training on small toy problems, we also avoid memory issues we would encounter by meta-training on very large, real-world problems.
Except where otherwise indicated, all target problems were designed to have a global minimum of zero (in some cases a constant offset was added to make the minimum zero). The code defining each of these problems is included in the open source release. See A.
4.1.1 Exemplar problems from literature
We included a set of 2-dimensional problems which have appeared in optimization literature (Surjanovic & Bingham, 2013) as toy examples of various loss landscape pathologies. These consisted of Rosenbrock, Ackley, Beale, Booth, Styblinski-Tang, Matyas, Branin, Michalewicz, and log-sum-exp functions.
4.1.2 Well behaved problems
We included a number of well-behaved convex loss functions, consisting of quadratic bowls of varying dimension with randomly generated coupling matrices, and logistic regression on randomly generated, generally linearly separable data. For the logistic regression problem, when the data is not fully linearly separable, the global minimum is greater than 0.
4.1.3 Noisy gradients and minibatch problems
For problems with randomly generated data, such as logistic regression, we fed in minibatches of various sizes, from 10 to 200. We also used a minibatch quadratic task, where the minibatch loss consisted of the square inner product of the parameters with random input vectors.
For full-batch problems, we sometimes added normally distributed noise with standard deviations from 0.1 to 2.0 in order to simulate noisy minibatch loss.
4.1.4 Slow convergence problems
We included several tasks where optimization could proceed only very slowly, despite the small problem size. This included a many-dimensional oscillating valley whose global minimum lies at infinity, and a problem with a loss consisting of a very strong coupling terms between parameters in a sequence. We additionally included a task where the loss only depends on the minimum and maximum valued parameter, so that gradients are extremely sparse and the loss has discontinuous gradients.
4.1.5 Transformed problems
We also included a set of problems which transform the previously defined target problems in ways which map to common situations in optimization.
To simulate problems with sparse gradients, one transformation sets a large fraction of the gradient entries to 0 at each training step. To simulate problems with different scaling across parameters, we added a transformation which performs a linear change of variables so as to change the relative scale of parameters. To simulate problems with different steepness-profiles over the course of learning, we added a transformation which applied monotonic transformations (such as raising to a power) to the final loss. Finally, to simulate complex tasks with diverse parts, we added a multi-task transformation, which summed the loss and concatenated the parameters from a diverse set of problems.
For the meta-training loss, used to train the meta-parameters of the optimizer, we used the average log loss across all training problems,
where the second term is a constant, and where is the full set of meta-parameters for the learned optimizer, consisting of , where indicates the GRU weights and biases for the Parameter, Tensor, or Global RNN, is the learning rate momentum and is the attended step offset (Section 3.2.4).
Minimizing the average log function value, rather than the average function value, better encourages exact convergence to minima and precise dynamic adjustment of learning rate based on gradient history (Figure 6). The average logarithm also more closely resembles minimizing the final function value, while still providing a meta-learning signal at every training step, since very small values of make an outsized contribution to the average after taking the logarithm.
4.3 Partial unrolling
Meta-learning gradients were computed via backpropagation through partial unrolling of optimization of the target problem, similarly toAndrychowicz et al. (2016). Note that Andrychowicz et al. (2016)
4.4 Heavy-tailed distribution over training steps
The optimizers were meta-trained for at least 40M meta-iterations (each meta-iteration consists of loading a random problem from the meta-training set, running the learned optimizer on that target problem, computing the meta-gradient, and then updating the meta-parameters). The meta-objective was minimized with asynchronous RMSProp across 1000 workers, with a learning rate of .
5.1 Failures of existing learned optimizers
Previous learned optimizer architectures like Andrychowicz et al. (2016) perform well on the problems on which they are meta-trained. However, they do not generalize well to new architectures or scale well to longer timescales. Figure 2
shows the performance of an optimizer meta-trained on a 2-layer perceptron with sigmoid activations on the same problem type with ReLU activations and a new problem type (a 2-layer convolutional network). In both cases, the same dataset (MNIST) and minibatch size (64) was used. In contrast, our optimizer, which has not been meta-trained on this dataset orany neural network problems, shows performance comparable with ADAM and RMSProp, even for numbers of iterations not seen during meta-training (Section 4.4).
5.2 Performance on training set problems
The learned optimizer matches or outperforms ADAM and RMSProp on problem types from the meta-training set (Figure 3). The exact setup for each problem type can be seen in the python code in the supplementary materials.
5.3 Generalization to new problem types
The meta-training problem set did not include any convolutional or fully-connected layers. Despite this, we see comparable performance to ADAM, RMSProp, and SGD with momentum on simple convolutional multi-layer networks and multi-layer fully connected networks both in terms of final loss and number of iterations to convergence (Figure 3(a) and Figure 2).
We also tested the learned optimizer on Inception V3 (Szegedy et al., 2016) and on ResNet V2 (He et al., 2016). Figure 3(b) shows the learned optimizer is able to stably train these networks for the first 10K to 20K steps, with performance similar to traditional optimizers tuned for the specific problem. Unfortunately, we find that later in training the learned optimizer stops making effective progress, and the loss approaches a constant (approximately 6.5 for Inception V3). Addressing this issue would be a goal of future work.
5.4 Performance is robust to choice of learning rate
One time-consuming aspect of training neural networks with current optimizers is choosing the right learning rate for the problem. While the learned optimizer is also sensitive to initial learning rate, it is much more robust. Figure 5 shows the learned optimizer’s training loss curve on a quadratic problem with different initial learning rates compared to those same learning rates on other optimizers.
5.5 Ablation experiments
The design choices described in Section 3 matter for the performance of the optimizer. We ran experiments in which we removed different features and re-meta-trained the optimizer from scratch. We kept the features which, on average, made performance better on a variety of test problems. Specifically, we kept all of the features described in 3.2 such as attention (3.2.1), momentum on multiple timescales (gradient scl) (3.2.2), dynamic input scaling (variable scl decay) (3.2.3), and a relative learning rate (relative lr) (3.2.4). We found it was important to take the logarithm of the meta-objective (log obj) as described in 4.2. In addition, we found it helpful to let the RNN learn its own initial weights (trainable weight init) and an accumulation decay for multiple gradient timescales (inp decay). Though all features had an effect, some features were more crucial than others in terms of consistently improved performance. Figure 6 shows one test problem (a 2-layer convolutional network) on which all final features of the learned optimizer matter.
5.6 Wall clock comparison
In experiments, for small minibatches, we significantly underperform ADAM and RMSProp in terms of wall clock time. However, consistent with the prediction in 3.4, since our overhead is constant in terms of minibatch we see that the overhead can be made small by increasing the minibatch size.
We have shown that RNN-based optimizers meta-trained on small problems can scale and generalize to early training on large problems like ResNet and Inception on the ImageNet dataset. To achieve these results, we introduced a novel hierarchical architecture that reduces memory overhead and allows communication across parameters, and augmented it with additional features shown to be useful in previous optimization and recurrent neural network literature. We also developed an ensemble of small optimization problems that capture common and diverse properties of loss landscapes. Although the wall clock time for optimizing new problems lags behind simpler optimizers, we see the difference decrease with increasing batch size. Having shown the ability of RNN-based optimizers to generalize to new problems, we look forward to future work on optimizing the optimizers.
- Andrychowicz et al. (2016) Andrychowicz, Marcin, Denil, Misha, Gomez, Sergio, Hoffman, Matthew W, Pfau, David, Schaul, Tom, Shillingford, Brendan, and de Freitas, Nando. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
- Bahdanau et al. (2015) Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. iclr, 2015.
- Bengio et al. (1995) Bengio, S., Bengio, Y., and Cloutier, J. On the search for new learning rules for ANNs. Neural Processing Letters, 2(4):26–30, 1995.
- Bengio et al. (1990) Bengio, Yoshua, Bengio, Samy, and Cloutier, Jocelyn. Learning a synaptic learning rule. Université de Montréal, Département d’informatique et de recherche opérationnelle, 1990.
- Bengio et al. (1992) Bengio, Yoshua, Bengio, Samy, Cloutier, Jocelyn, and Gecsei, Jan. On the optimization of a synaptic learning rule. In in Conference on Optimality in Biological and Artificial Networks, 1992.
- Chen et al. (2016) Chen, Yutian, Hoffman, Matthew W., Colmenarejo, Sergio Gomez, Denil, Misha, Lillicrap, Timothy P., and de Freitas, Nando. Learning to learn for global optimization of black box functions. arXiv Report 1611.03824, 2016.
- Cho et al. (2014) Cho, Kyunghyun, Van Merriënboer, Bart, Bahdanau, Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
- Duan et al. (2016) Duan, Yan, Schulman, John, Chen, Xi, Bartlett, Peter, Sutskever, Ilya, and Abbeel, Pieter. Rl: Fast reinforcement learning via slow reinforcement learning. Technical report, UC Berkeley and OpenAI, 2016.
- Funk (2015) Funk, Simon. RMSprop loses to SMORMS3 - beware the epsilon!, 2015. URL sifter.org/$∼$simon/journal/20150420.html.
- Harlow (1949) Harlow, Harry F. The formation of learning sets. Psychological review, 56(1):51, 1949.
He et al. (2016)
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian.
Identity mappings in deep residual networks.
European Conference on Computer Vision, pp. 630–645. Springer, 2016.
- Hochreiter et al. (2001) Hochreiter, Sepp, Younger, A Steven, and Conwell, Peter R. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87–94. Springer, 2001.
Kehoe, E James.
A layered network model of associative learning: learning to learn and configuration.Psychological review, 95(4):411, 1988.
- Kingma & Ba (2015) Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. iclr, 2015.
- Krizhevsky et al. (2012) Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
- Lake et al. (2016) Lake, Brenden M, Ullman, Tomer D, Tenenbaum, Joshua B, and Gershman, Samuel J. Building machines that learn and think like people. arXiv Report 1604.00289, 2016.
- Li & Malik (2017) Li, SKe and Malik, Jitendra. Learning to optimize. In International Conference on Learning Representations, 2017.
- Naik & Mammone (1992) Naik, Devang K and Mammone, RJ. Meta-neural networks that learn by learning. In International Joint Conference on Neural Networks, volume 1, pp. 437–442. IEEE, 1992.
- Nesterov (1983a) Nesterov, Yurii. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983a.
- Nesterov (1983b) Nesterov, Yurii. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983b.
- Ravi & Larochelle (2017) Ravi, Sachin and Larochelle, Hugo. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
Runarsson & Jonsson (2000)
Runarsson, Thomas Philip and Jonsson, Magnus Thor.
Evolution and design of distributed learning rules.
IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks, pp. 59–63. IEEE, 2000.
- Santoro et al. (2016) Santoro, ADAM, Bartunov, Sergey, Botvinick, Matthew, Wierstra, Daan, and Lillicrap, Timothy. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning, 2016.
- Schmidhuber (1987) Schmidhuber, Jurgen. Evolutionary Principles in Self-Referential Learning. On Learning how to Learn: The Meta-Meta-Meta…-Hook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
- Silver et al. (2016) Silver, David, Huang, Aja, Maddison, Chris J, Guez, Arthur, Sifre, Laurent, Van Den Driessche, George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam, Veda, Lanctot, Marc, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.
- Surjanovic & Bingham (2013) Surjanovic, Sonja and Bingham, Derek. Optimization test functions and datasets, 2013. URL http://www.sfu.ca/~ssurjano/optimization.html.
- Sutton (1992) Sutton, Richard S. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Association for the Advancement of Artificial Intelligence, pp. 171–176, 1992.
Szegedy et al. (2016)
Szegedy, Christian, Vanhoucke, Vincent, Ioffe, Sergey, Shlens, Jon, and Wojna,
Rethinking the inception architecture for computer vision.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826, 2016.
- Thrun & Pratt (1998) Thrun, Sebastian and Pratt, Lorien. Learning to learn. Springer Science and Business Media, 1998.
- Tieleman & Hinton (2012) Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4:2, 2012.
- Tseng (1998) Tseng, Paul. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. Journal on Optimization, 8(2):506–531, 1998.
- Wang et al. (2016) Wang, Jane X., Kurth-Nelson, Zeb, Tirumala, Dhruva, Soyer, Hubert, Leibo, Joel Z., Munos, Rémi, Blundell, Charles, Kumaran, Dharshan, and Botvinick, Matt. Learning to reinforcement learn. arXiv Report 1611.05763, 2016.
- Ward (1937) Ward, Lewis B. Reminiscence and rote learning. Psychological Monographs, 49(4):i, 1937.
- Zoph & Le (2017) Zoph, Barret and Le, Quoc V. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
Appendix A Code
The code for the meta-training procedure and meta-train problem set is available at https://git.io/v5oq5.
Appendix B Additional details of RNN architecture
b.1 Shortcut connection
Since we expect to be the primary driver of update step direction, and in order to further reduce the information which must be stored in the Parameter RNN hidden state, we included a meta-trainable linear projection from the average rescaled gradients and the update directions and .
Appendix C Additional details of meta-training process
c.1 Heavy-tailed distribution over training steps
Appendix D Architecture updates
The Inception V3 experiment in Figure 3(b) used a slightly newer version of the learned optimizer codebase. The changes were:
d.1 Parameter noise during training
Due to the use of small meta-training problems in Section 4.1, during meta-training the learned optimizer is often able to optimize the problem almost exactly early in the unrolled optimization, after which the meta-loss s becomes relatively uninformative. In order to better simulate tasks which take many steps to optimize, small Gaussian noise is added to the parameters during each optimization step. This effectively moves the loss landscape underneath the optimizer, providing a more informative learning signal after many unrolls, and forcing the learned optimizer to be robust to a new type of noise. Specifically, the parameter update becomes
where the noise scale is drawn from a log uniform distribution between and for each problem.
d.2 Momentum from previous timescale
In Equation 3 we scale the average gradients by a running estimate of the root-mean-square magnitude of . This is a mismatch with Adam, where the average gradient is scaled by a running estimate of the root-mean-square magnitude of the non-averaged gradients. In order to be consistent with this, and in order to encourage better use of the dynamic range of (as defined in the text body, it spends much of its time with values near or ), we modify Equation 3 to normalize the average gradient by from the immediately faster timescale,
and where we define the average gradient at the fastest time scale to be the raw gradient,
d.3 No normalization of step length
In order to simplify interactions between parameters, we no longer force a normalization of the parameter and attention update directions and . We do still decompose the update into the product of a learning rate and a step. Since the attended update direction is now able to take on a different magnitude, the separate attention log learning rate is no longer required, and is eliminated. Equations 5 and 6 thus become
d.4 More stable meta-training hyper-parameters
The distribution over meta-loss gradients is observed to be assymmetrical and heavy tailed. This combination is known to cause biased parameter updates in RMSProp and Adam, since both optimizers underweight the contribution from extremely rare extremely large gradients. In order to reduce this tendency, we updated the mean-quare-gradient momentum term to be 0.999, rather than 0.9 in the meta-optimizer RMSProp (Section 4.5).