Disentangling Adaptive Gradient Methods from Learning Rates

02/26/2020 ∙ by Naman Agarwal, et al. ∙ Google Princeton University Tel Aviv University 6

We investigate several confounding factors in the evaluation of optimization algorithms for deep learning. Primarily, we take a deeper look at how adaptive gradient methods interact with the learning rate schedule, a notoriously difficult-to-tune hyperparameter which has dramatic effects on the convergence and generalization of neural network training. We introduce a "grafting" experiment which decouples an update's magnitude from its direction, finding that many existing beliefs in the literature may have arisen from insufficient isolation of the implicit schedule of step sizes. Alongside this contribution, we present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods, aimed at bringing more clarity to this space.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Adaptive gradient methods are a cornerstone of optimization for deep learning. However, precise characterization of their behavior proves to be elusive; theoretical and empirical accounts of their preconditioning, generalization, and auto-tuning properties have seldom resulted in the unanimous adoption of new practices. In the pursuit of state-of-the-art results, the choice of optimizer is essentially treated as a black-box hyperparameter. To make matters worse, each optimizer implicitly specifies a different implicit sequence of step sizes; this interferes with the tuning of another notoriously opaque hyperparameter, the learning rate schedule.

In this work, we present methods to decouple an optimizer’s preconditioning effect from its implicit step size schedule. The chief purpose of this contribution is to provide a more rigorous way to conduct research in optimizer evaluation, but we explore the diagnostic and algorithmic potential of mixing optimizers. In the final section, we provide a broader retrospective on existing beliefs about adaptive gradient methods, and point out instances where experimental conclusions could have arisen from insufficient disentanglement of the optimizer from the learning rate schedule.

1.1 Our contributions

To better understand how adaptive gradient methods interact with learning rates, we present the following:

Methodology of learning-rate grafting.

We propose several variants of a simple grafting experiment, which combines the step magnitude and direction of two different optimizers. An immediate application is to control for the high-dimensional learning rate schedule parameter when evaluating any optimizer’s performance. We also highlight the potential of grafting to enable the discovery of new algorithmic tools. In particular, we find a linearly increasing step size schedule correction for AdaGrad.

Empirical insights about preconditioning.

Applying our grafting meta-optimizer to large-scale problems as a diagnostic tool, we find that the foremost

determining factor of an optimizer’s training curve is its implicit step size schedule. Controlling for this enables us to take a deeper look at the individual role of adaptivity. We shed light on why optimizer choice appears to be brittler in natural language processing than in computer vision.

Review of oft-cited evidence.

We provide a broader discussion on the existing literature on the comparison and evaluation of adaptive optimizers. We identify a few common confounding factors, along with easy ways to mitigate them. Centrally, we provide a retrospective on theoretical and empirical evidence from [wilson2017marginal], including a replication study of their experiments.

Although the core of our work comprises a relatively simple experimental idea, the discussion and implications are quite wide-ranging. For convenience, we collect in Appendix A a short summary all of our empirical findings.

1.2 Related work

Adaptive regularization.

Adaptive gradient methods were introduced by the online learning community [duchi2011adaptive, mcmahan2010adaptive]; see [hazan2016introduction]

for an introduction. A flurry of extensions, heuristics and modifications followed, most notably RMSprop 

[tieleman2012lecture] and Adam [kingma2014adam].

Adaptive optimization in deep learning.

Adaptive methods have turned out to be extremely robust in training deep neural networks, receiving tens of thousands of citations for this reason. In particular, Adam has been the de facto standard in fields such as NLP [devlin18, yang2019xlnet, liu2019roberta], deep generative modeling [karras2017progressive, brock2018large, kingma2018glow], and deep RL [haarnoja2018soft]. Adaptive methods have seen adoption in extremely large-scale settings, necessitating modifications to reduce resource consumption [shazeer2018adafactor, anil2019memory, chen2019extreme].

The empirical debate on adaptive methods.

An important discussion was sparked by [wilson2017marginal], who presented empirical and theoretical situations where adaptive methods generalize poorly. Building on this premise, [keskar2017improving] suggest switching from Adam and SGD during training. [smith2019super] develop a doctrine of “superconvergence” which eschews adaptive methods. [reddi2018convergence] construct, then mitigate, a pathological setting where Adam fails to converge. However, in the vast majority of cases, out-of-the-box adaptive methods are perfectly suitable for practitioners.

Learning rate schedules.

Choosing learning rate schedules has become a vexing empirical problem. Beyond classical optimization, [ge2019step] provide a fine-grained theoretical account for quadratic losses; [li2019exponential, arora2018theoretical]

study the interaction of learning rates with batch normalization

[ioffe2015batch]. The interaction of learning rates with batch size is explored in [krizhevsky2014one, goyal2017accurate, bottou2018optimization]. Learning rate warmups and restarts are popular state-of-the-art heuristics [gotmare2018closer, loshchilov2016sgdr]. [shampoopp] use the grafting method as one of many heuristics to stabilize an exotic full-matrix optimizer; we comment further in Section 4.1.

Discarding the backprop magnitude.

Several recent lines of work propose optimization procedures which discard magnitude information from the gradients (without replacing it with another optimizer’s magnitude). [blier2018learning] train models with randomized learning rates, motivated by ensemble learning intuitions. See the discussion on exotic optimizers in Section 4.1 for more examples. Our grafting meta-algorithm might help to shed more light on why these optimizers converge successfully.

2 Preliminaries

2.1 Stochastic first-order optimization

We will work in the usual abstraction of the stochastic optimization problem:

where the expectation is over a random variable

whose distribution

is initially unknown. In supervised learning,

represents an example-label pair , drawn from an unknown population. A stochastic optimization algorithm is given samples from the underlying distribution, and produces a point whose population loss is as close as possible to that of the minimizer . We will consider domains .

We will focus exclusively on methods which maintain a sequence of iterates , iteratively querying a stochastic gradient , for which . In modern deep learning, this is typically averaged over a mini-batch. Notationally, a gradient optimizer is a map from the current iterate and gradient to the next iterate .111Note that our notation is “imperative” rather than “functional”: an update can depend on the algorithm’s internal state, hyperparameters, or randomness. Then, vanilla SGD with learning rate schedule is the algorithm specified by

2.2 Adaptive gradient methods

It will be convenient to introduce some unifying notation for the standard family of second-moment-based adaptive optimizers. Algorithm 

1 captures the usual formulations of Adam, AdaGrad, and RMSprop, up to scaling conventions which can be absorbed into ; see Section 5.2 for a discussion on these, and [choi2019empirical] for a comprehensive unifying treatment. Additionally, by omitting the denominator

, we recover SGD with momentum. All operations on vectors are understood to be entrywise;

indicates entrywise squaring.

1:  Input: Initializer , learning rate schedule , hyperparameters .
2:  Initialize .
3:  for  do
4:     Receive stochastic gradient at .
5:     Update accumulators:
6:     Update:
7:  end for
Algorithm 1 Generic second-moment adaptive optimizer

The exponential window parameter denotes heavy-ball momentum [polyak1964some]. We set this to be throughout our experiments, which remains an uncontroversial rule of thumb. Importantly, we introduce momentum in AdaGrad, even though this is not possible by default in most deep learning packages (see Section 5.2). We do not attempt to disentangle the role of momentum in deep learning, nor the Nesterov-style alternative update [nesterov1983method, dozat2016incorporating]; we view this as an important but much subtler orthogonal question.

Crucially, for AdaGrad we have ; the accumulator does not decay. Adam, RMSprop, and countless variants opt for a slow attenuation like , as we use for all Adam experiments. The role of this is controversial; some works [kingma2014adam, agarwal2017second, staib2019escaping] view this a feature (forgetting stale information about the loss surface’s curvature), while [reddi2018convergence] present pathological failure cases that arise from this same forgetfulness. One minor conclusion from our main empirical study (Section 3) is that AdaGrad’s accumulator rule is not necessarily harmful in deep learning.

The constant is used to promote numerical stability in low-precision division, but can also be interpreted as a way to discard

adaptivity and “interpolate” an algorithm with SGD. We discuss the confounding perils of this in Section 

5.2, and use in our experiments, as outlined there.

2.3 Step size schedules, implicit and explicit

It is well-known that adaptive methods imply effective step sizes, especially in light of the discussion on the second-moment accumulator. This is motivated by the online formulation, in which a decay of the update magnitude gives optimal worst-case regret bounds [zinkevich2003online, duchi2011adaptive]. Going even earlier, step-size sequences are needed for classical convergence guarantees in first-order non-smooth and/or stochastic optimization (see, e.g., [boyd2004convex, ghadimi2013stochastic]). Table 1 gives a survey of some theoretically and/or empirically-motivated step size schedules.

 Schedule  Setting
Smooth, convex GD
Strongly convex SGD
Smooth non-convex SGD
Non-smooth convex GD/SGD
SGD, least squares
Batch norm + weight decay
Image models
Table 1: Some known learning rate schedules from classical and recent optimization literature.

However, a general theory of how to set the learning rate schedule is notoriously elusive. This -dimensional hyperparameter is generally selected by human search, augmented by architecture-specific heuristics.222It is difficult to find an authoritative citation for this, but refer to the step size heuristics in [goyal2017accurate, popel2018training] to see how delicate they are in state-of-the-art pipelines. For example, stepwise exponentially-decaying schedules are very popular on vision tasks, but seldom seen on language tasks. To add further complexity, learning rate warmups have been a recent popular choice across domains.

3 Decoupling Step Size from Direction

Concretely, we point out the following tension in the practice of optimization in deep learning: that the role of overlaps with the implicit step size schedule induced by the optimizer. In this section, we propose decoupling these factors via learning rate grafting.

3.1 AdaGraft: a meta-optimizer

We begin by presenting our experimental methodology, which revolves around Algorithm 2. At each iteration, it computes a single gradient, passes it to two optimizers, and makes a grafted step, combining the magnitude of ’s step and direction of ’s step.

1:  Input: Optimizers ; initializer ; .
2:  Initialize at .
3:  for  do
4:     Receive stochastic gradient at .
5:     Query steps from and :
6:     Update with grafted step:
7:  end for
Algorithm 2 AdaGraft meta-optimizer

Layer-wise vs. global grafting.

Two natural variants of Algorithm 2 come to mind, especially if one is concerned about efficient implementation (see Appendix C.1). In the layer-wise version, we view

as a single parameter group (usually a tensor-shaped variable specified by the architecture), and apply AdaGraft and its child optimizers to each group. In the

global version, contains all of the model’s weights. We discuss and evaluate both variants, but our main experimental results use layer-wise grafting.

The first experimental question addressed by AdaGraft is the following: To what degree does an optimizer’s implicit step size schedule determine its training curve? To this end, given a set of base optimizers, we can perform training runs for all pairs , where grafting is understood as simply running . For the main experiments, we use SGD, Adam, and AdaGrad, all with momentum .

All experiments were carried out on 32 cores of a TPU-v3 Pod [jouppi2017datacenter], using the Lingvo [shen2019lingvo]

sequence-to-sequence framework built on top of TensorFlow


3.2 ImageNet classification experiments

We ran all pairs of grafted optimizers on a 50-layer residual network [he2016identity]

with 26M parameters, trained on ImageNet classification

[imagenet]. We used a batch size of 4096, enabled by the large-scale training infrastructure, and a learning rate schedule consisting of a linear warmup and stepwise exponential decay. All details can be found in Appendix C.2.

Table 2 shows top-1 and top-5 accuracies at convergence. The final accuracies at convergence, as well as training loss curves, are very stable ( deviation) across runs, due to the large batch size. Figure 1 shows at a glance our main empirical observation: that the shapes of the training curves are clustered by the choice of , the optimizer which supplies the step magnitude.

We stress that no additional hyperparameter tuning was done in these experiments; not even the global scalar learning rate needed adjustment. Thus, starting with tuned optimizer setups, grafting produces a table of setups with no additional effort. Each row of this table controls for the implicit step size schedule.

Figure 1: Training curves for grafted optimizers on ResNet-50 for ImageNet classification. To first order, the shape of the training curve is determined by the step size schedule; most dramatically, AdaGrad’s poor convergence appears to be entirely due to its implicit step size schedule.
SGD Adam AdaGrad
SGD 75.4 92.6 72.8 91.2 73.7 91.4
Adam 74.1 91.9 73.0 91.3 73.7 91.6
AdaGrad 65.0 85.9 65.1 86.0 65.3 86.3
Table 2: Top-1 and top-5 accuracies at training step K for ImageNet experiments. Averaged over 3 trials; no accuracy varied by more than .

3.3 WMT14 English-French translation experiments

For a realistic large-scale NLP setting, we trained all grafted optimizers on a 6-layer Transformer network

[vaswani2017attention] with 375M parameters, on the WMT14 English-French translation task, which has 36.3M sentence pairs. Again, we use a large batch size (384 sequences), enabling very robust training setups. More details can be found in Appendix C.3.

Figure 2: Training curves for grafted optimizers on Transformer for WMT14 English-French translation. Convergence and solution quality are very sensitive to implicit step size schedule.
SGD Adam AdaGrad
SGD 39.8 0.1 40.0 0.3 39.4 0.2
Adam 40.6 0.2 41.2 0.2 40.1 0.2
AdaGrad 41.4 0.2 41.8 0.1 41.6 0.1
Table 3: Test set BLEU scores for Transformer grafting experiments at

K training steps. Averaged over 3 trials; standard deviations shown.

Interestingly, beyond demonstrating the same clustering of performance metrics by the choice of , these experiments show that it is possible for a grafted optimizer to outperform both base methods and ; see Figure 2 for loss curves, and Table 3 for the downstream BLEU metric, with which our results are consistent. Again, we are not making claims of categorical superiority under careful tuning, and only the power of bootstrapping; we stress that we did not even tune the global learning rate scalar.

3.4 Bootstrapping a learning rate schedule

So far, Algorithm 2 has taken the role of an exploratory tool for more principled optimizer comparison. In this section, we illustrate how it can be used in optimizer discovery. Focusing on the underrepresentation of AdaGrad in computer vision, we see that grafting can be repurposed to find a suitable learning rate schedule from scratch, which might not have been found from blind search or first principles.

It has been noted many times [zeiler2012adadelta, wilson2017marginal, bottou2018optimization] that AdaGrad’s step size schedule is “too aggressive” for deep learning. Using the global variant of Algorithm 2 with = (SGD, AdaGrad) and keeping track of the norms of the steps produced by and (Figure 4), we arrive at a linear correction , with , which can be overlaid on the usual warmup and stepwise-exponential annealing.

Figure 3: Global grafting experiments to fix AdaGrad. Bootstrapping with a well-tuned =SGD, we find an offline correction for AdaGrad’s implicit step size schedule; corrected AdaGrad competes with SGD or Adam. Grafting is denoted by .
Figure 4: Empirical discovery of linear learning rate correction from Figure 3. Norms of proposed steps are measured in a global (SGD, AdaGrad) grafting experiment. The ratio is found empirically to scale linearly with ; this adjustment can be applied directly to AdaGrad, without any grafting.

We have not encountered a linearly increasing learning rate schedule anywhere in the literature (see Table 1), and its interaction with AdaGrad’s decay is unintuitive. We try this again in the CIFAR-10 experiment in Section 5.3, but leave a closer look for future work.

Insufficiency of global learning rates in translation.

Repeating this experiment in the machine translation setup, we discover a learning rate correction from SGD to AdaGrad using per-layer grafting; however, we could not get SGD with this offline correction to train like AdaGrad; see Figure 5. Visualizing the per-layer norm corrections prescribed by Algorithm 2 sheds some light on why this might be the case; see Section 4.2 and Appendix B.1 for further discussion and some additional visualizations.

Figure 5: Training curves for global grafting and learning rate schedule correction, as in Figure 3, for the Transformer/WMT setup, layer-wise grafting with =AdaGrad, =SGD. Unlike the analogous experiment for ImageNet, we could not get corrected SGD to perform as well as AdaGrad.

4 Discussion and Further Experiments

4.1 Better evaluation for the zoo of optimizers

We highlight the potential of grafting to control for the implicit step size schedules of optimizers whose dynamics are not well-understood. In Figure 6, we show that by grafting the step size schedule of AdaGrad onto several recently proposed adaptive optimizers [zaheer2018adaptive, reddi2018convergence, shazeer2018adafactor], showing that their step directions are not significantly different.

Importantly, note that this does not invalidate any empirical findings from those works; it only suggests that these optimizers do not propose better directions, in the sense of coordinate-wise step size scalings. Running meaningful experiments with these optimizers playing the role of requires exact replications of their setups, which we do not attempt in this work.

Figure 6: Grafting experiments with an assortment of optimizers, fixing =AdaGrad. The training curves are nearly indistinguishable for the closely related adaptive methods, which we did not tune at all.

We conclude this section with more connections to the recent large-scale optimization literature.

More exotic update directions.

We highlight the possibility of simplifying the problem of searching for a good learning rate schedule when faced with an exotic optimizer. These might arise from quantization when one is concerned with communication efficiency [bernstein2018signsgd, wen2017terngrad], optimizer search [bello2017neural], or various theoretically-motivated optimizers which deviate from usual second-moment statistics [chen2018closing].

Normalizing steps by layer weights.

A string of recent empirical successes in accelerating large-scale neural network training tie the size of the update step to that of the layer , rather than the statistics of the gradients. These include LARS [you2017large], LAMB [you2019reducing], and Novograd [ginsburg2019stochastic]. Our experiments, particularly when grafting SGD, synergize with these works: that layer-wise scaling is powerful enough to train many deep learning architectures.

Beyond diagonal-matrix preconditioning.

An important family of optimizers comes from recent efforts to bring full-matrix preconditioning from theoretical origins to state-of-the-art deep learning [martens2015optimizing, martens2018kronecker, gupta2018shampoo, pmlr-v97-agarwal19b]. As mentioned in 1.2, learning-rate grafting was one of the many heuristics reported by [shampoopp] in order to make Shampoo converge in an industry-scale setting.

4.2 Understanding the role of adaptivity in NLP

In state-of-the-art methods in natural language processing powered by Transformers or recurrent neural networks, Adam is a versatile go-to choice. We propose using the per-layer grafting experiments, which were successful in both domains, to help explain why global learning rate correction is insufficient for Transformers.

Figure 7 visualizes the per-layer learning rates used to get successful convergence and matching training curves in the layer-wise grafting experiments. The effect of adaptive methods on Transformer training varies in magnitude and by layer; thus, the trajectory correction is higher-dimensional than on a ResNet, where can be summarized by a time-varying scalar. This provides a finer-grained view of the difficulty of optimization in deep NLP as compared to computer vision [pascanu2013difficulty, pmlr-v97-agarwal19b].

Figure 7: Layer-wise and global step size corrections for grafting experiments from Sections 3.2 and 3.3. Curves for individual layers are shaded proportionally to their average squared step norm from . Left: ResNet setup, running layer-wise grafting with = SGD, = AdaGrad. The global ratio is well-correlated with the the individual layers’ curves. Darkest curve is the softmax layer. Right: Transformer setup, running layer-wise grafting with = SGD, = AdaGrad. Unlike for ImageNet, the per-layer corrections are many orders of magnitude apart; note that the -axis is logarithmic. Darkest curves are the softmax and embedding layers.

5 Revisiting Beliefs about Adaptivity

The remainder of this work can be thought of as a “position paper”: a medley of broader observations on the confounding interactions between adaptive methods and learning rate schedules, arising from the grafting (plus some additional) experiments, as well as recent literature.

5.1 Theoretical accounts

Pathological examples.

A frequently-cited justification for the belief that SGD enjoys privileged generalization performance is a pathological case in which adaptive methods overfit dramatically:

Proposition 1 ([wilson2017marginal]; informal).

There exists an instance of least-squares regression () on which SGD generalizes better than AdaGrad.

Intuitively, SGD converges to a solution with small

norm. As is illustrated in the example above this allows SGD to not find a solution in a spurious subspace embedded inside the problem overall making it robust to outliers.

To present a counterpoint to this example, we construct a case in which SGD underfits:

Proposition 2 (Counter-counterexample; informal).

There exists an instance of hinge-loss linear classification () on which AdaGrad generalizes better than SGD.

Intuitively, in this scenario the test error depends on seeing every coordinate and some coordinates are chosen to be rare. While AdaGrad adapts quickly, SGD is slower to adapt, leading to comparatively very slow convergence in test error.

Thus, although artificial examples may be illuminating, especially in order to identify edge cases and qualitative understandings for when certain optimizers might fail, one must be careful not to apply these intuitions too generally. We review the first example and present the second in more detail in Appendix D.1.

Literature on the trajectory analysis of SGD.

Finally, we point to some recent literature on the generalization of SGD, which employ varied theoretical lenses to analyze its optimization trajectory. These include analyses based on stability and early stopping [hardt2015train], implicit bias in basic settings [gunasekar2018characterizing, gunasekar2018implicit], and the specific structure of neural networks [allen2019can, arora2019fine]. Recent work [belkin2019reconciling, mei2019generalization] studies the generalization of gradient descent in the overparameterized interpolation regime. These works show that SGD generalizes in various abstractions of neural network training, but do not make negative claims about adaptive methods. Since optimization is so robust to distortion of the gradient signal via preconditioning, our experiments suggest that a useful theoretical account of optimization in deep learning should not be too sensitive to the fact that SGD is the object of interest. It remains an intriguing question whether the trajectories of grafted optimizers match (in some well-chosen metric); we leave a careful experimental probe for future work.

5.2 Superficial discrepancies between optimizers

Even though we have taken a unifying view of the usual family of adaptive optimizers, the choice of optimizer is often seen as a black-box discrete hyperparameter. We show here that modulo the learning rate schedule, the popular second-moment adaptive optimizers are equivalent.

Warm-up and bias correction.

Adam [kingma2014adam]

takes the explicit view of trying to estimate the first and second exponentially-attenuated moments of the gradients. To do so, applies a

bias correction adjustment factor of to the learning rate. We note that for common choices of , this sequence is effectively a learning rate warmup on the scale of iterations; see Figure 8.

Figure 8: Adam’s bias correction factor with . For common values of , this takes the shape of a redundant learning rate warmup on top of RMSprop or AdaGrad.

Accumulator conventions.

In popular implementations, RMSprop and Adam’s accumulator updates multiply the new squared gradient by ; Adam also does this for the momentum . This is equivalent to a global scaling of and for , but prevents a unification with AdaGrad for no fundamental reason. Furthermore, in commonly available implementations of AdaGrad, momentum is not implemented, perhaps due to historical reasons.

The parameter.

This numerical stability hyperparameter, as defined and discussed briefly in Section 2.2, may be a hidden confounding factor in experiments. Between frameworks and individual optimizers, the denominator in Algorithm 1 may be or , and default values vary wildly (see Appendix B of [wilson2017marginal]).

We suggest trying , to remove a confounding factor: if the accumulator is 0, then the movement in that coordinate should also be 0. This can be viewed as taking the square root of the Moore-Penrose pseudoinverse of the accumulator as the preconditioner. In Appendix D.2, we verify that this has no bearing on the original AdaGrad regret analysis.

Figure 9 illustrates how sensitive training can be to the choice of , providing a situation where is safest. We do not prescribe this in all situations; dramatically, with a wide-ranging hyperparameter search, [choi2019empirical] settle upon in one setting.

Figure 9: Training curves for AdaGrad in the Transformer machine translation setup of Section 3.3, varying the hyperparameter; this has a dramatic effect on training dynamics. To best demonstrate the interpolation, we did not do any further tuning or grafting.

5.3 Replicating experiments from [wilson2017marginal]

Finally, we report on an attempt to replicate the experimental findings of [wilson2017marginal], in which they found adaptive methods to generalize poorly in four distinct deep learning setups. Although we were able to replicate some of their findings on these smaller-scale experiments, others appear to be sensitive to hyperparameter tuning, and perhaps subtle changes in the deep learning software and hardware stack that have occurred during the two years since the publication of that paper. We summarize these findings below, deferring some details to Appendix B.2.

Figure 10: Preview of results of the replication study. Circles show numbers reported by [wilson2017marginal]. In some cases, outdated software/hardware stacks result in mild discrepancies; in others, they may reverse the conclusions.

For the smaller-scale CIFAR-10 experiment with a VGGNet, we were able to replicate their results more or less perfectly (Figure 10, left). For the character-level language modeling benchmark on War & Peace, the training setups converged, but suggest the opposite conclusion about the generalization of SGD. We had difficulty with the possibly stale codebases for the other two experiments: in the generative parsing experiment, the paper’s reported hyperparameters sometimes resulted in non-convergence; even after retuning, we could not match the reported validation losses. We could not get the discriminative parsing experiment, which was written in DyNet and no longer maintained, to finish training before encountering a memory error.

Finally, as an addendum to the CIFAR-10 replication, in Appendix B.2, we apply a linear learning rate correction (as in Section 3.4) to AdaGrad in their setup, resulting in convergence that competes with Adam and RMSprop.

The purpose of this replication attempt, in regards to the thesis of this paper, is to illustrate the unreliability of typical small-scale optimizer comparisons, not only due to the implicit step size schedule; [choi2019empirical] explore this point far more broadly.

6 Conclusion

We hope to initiate the practice of learning-rate grafting whenever a new optimizer is introduced. If one observes dramatic differences in the shapes of training curves, we suggest normalizing the implicit step size schedule. This can go both ways:

  • Grafting a well-performing algorithm with SGD acts as a sanity check, determining if ’s superiority comes from finding better step sizes.

  • Meanwhile, letting supply the direction allows one to isolate its preconditioning dynamics, and might aid in the discovery of new update rules and schedules.

More broadly, we believe that more experiments in optimization for deep learning will benefit from adopting this bootstrapping methodology, which depends on tuning one baseline, rather than all candidates. We hope that this will aid in developing more robust beliefs about both adaptive methods and learning rate schedules.


We are grateful to Sanjeev Arora, Yi Zhang, Zhiyuan Li, Wei Hu, Yoram Singer, Kunal Talwar, Roger Grosse, Karthik Narasimhan, Mark Braverman, Surbhi Goel, and Sham Kakade for helpful discussions and illuminating perspectives.


Appendix A Summary of Empirical Results

For convenience, we provide a summary of all of the assorted experiments, which are interleaved with discussion and definitions in the main paper. Pointers to the relevant sections, figures, and tables accompany each main point.

Layer-wise grafting: training curves depend strongly on .

In both the ResNet and Transformer settings, training curves and final performance metrics depend significantly more heavily on , the sub-algorithm in AdaGraft supplying the magnitude. No additional tuning was done on the grafted optimizers. This is our primary result. SGD remains the best-performing optimizer on ResNet (by a smaller margin), but the AdaGrad grafting instances perform better than plain AdaGrad. References: Sections 3.2 and 3.3, Figures 1 and Figure 2, and Tables 2 and 3. Hyperparameter details are in Appendix C.

Successfully training a Transformer with (grafted) SGD.

A minor side note is that it has been an empirical challenge to train a state-of-the-art Transformer NLP model using SGD; the empirical papers we have cited use Adam and AdaGrad, and [you2019reducing] corroborate this in passing (motivating their LAMB optimizer as an improvement over LARS). We show that SGD can train such a model if it is allowed to take bootstrapped layer-wise learning rates from an algorithm that trains successfully. Completing the story and training with a aptly-corrected pure SGD is an interesting line of future work.

Global grafting: training curves depend on in vision but not NLP.

In the ResNet setting, the same result as layer-wise grafting is seen with global grafting, but the global learning rate (one scalar) needs to be decreased slightly (factor of 0.5). This weaker, secondary result is shown for only SGD, AdaGrad. Global grafting did not work in the Transformer setting, even upon tuning the global learning rate multiplier. References: Section 3.4 and B.1, Figures 3 and 5.

Linear schedule for AdaGrad on image models.

From the global grafting experiments, we can find a (purely empirical) correction for AdaGrad’s step sizes: a linearly increasing schedule , to be multiplied on top of the existing schedule . Also shown to work on a VGGNet on CIFAR-10, as a side note. References: Section 3.4, Figures 3 and 12.

Isolating the preconditioning behavior of recently proposed optimizers.

Letting be Yogi, Adafactor, or AMSGrad, but fixing AdaGrad, we get nearly indistinguishable training curves in the Transformer setting. If these optimizers are performing well on their own (ungrafted), it is because they are finding good layer-wise step sizes. References: Section 4.1, Figure 6.

Varying the parameter.

In the Transformer setting, tuning can have a dramatic effect on convergence. In this case, below , the curves are the same. Thus, we suggest trying “” to simplify the hyperparameter search space. References: Section 5.2, Figure 9.

Exploratory visualization of layerwise step corrections.

A visualization that sheds light on why adaptive optimizer choice is brittler in the Transformer setting than the ResNet setting. For the ResNet, the scalar corrections implied by layer-wise grafting roughly match that of global grafting; they are all well-approximated by the linear correction heuristic. For the Transformer, these sequences can differ by up to 5 orders of magnitude. In particular, AdaGrad takes hundreds of times larger steps on the softmax and embedding layers, compared to SGD. Also, the layer-wise correction schedules have high-dimensional structure at the beginning of training. References: Section 4.2, Figure 7.

Replications of [wilson2017marginal] experiments.

We could reproduce the CIFAR-10 experiments, but not the other three. Note that our main ImageNet experiments challenge the suggestion from the CIFAR-10 experiments that adaptive methods are insufficient for image models, and that the linear correction to AdaGrad transfers to this setting. References: Appendix B.2, Figures 12, 13, 14, Tables 4, 5, 6.

Appendix B Supplemental Figures and Tables

b.1 Global grafting and schedule correction for machine translation

We show the negative result we encountered when applying the learning rate schedule bootstrapping experiment of Section 3.4 to the machine translation model. We begin by repeating the procedure of discovering a learning rate schedule correction; as seen in Figure 11, this results in a near-perfect fit for an inverse polynomial schedule correction, with power approximately 0.27.

Figure 11: Step norm visualization, as in Figure 4, for the Transformer/WMT setup, layer-wise grafting with =AdaGrad, =SGD. The global norm ratio suggests an offline learning rate schedule correction proportional to for , but applying this does not cause SGD to train well.

However, this learning rate correction, despite tuning, did not result in improved training; see Figure 5 in the main paper. Even after tuning the global scalar learning rate multiplier, we could not get SGD with the learning rate schedule correction to match the training curve of AdaGrad. Only layer-wise grafted AdaGrad could match (in fact, outperform) AdaGrad itself. In Section 4, we develop more fine-grained empirical evidence for why one might expect learning rate schedule tuning to be more challenging in natural language processing than in vision.

b.2 Replication of [wilson2017marginal] experiments

CIFAR-10: positive replication.

On the classic benchmark task of CIFAR-10 classification with a VGG network [simonyan2014very], we were able to replicate the [wilson2017marginal] results perfectly, using the same codebase333https://github.com/szagoruyko/cifar.torch. We repeated the hyperparameter search reported in the paper, found the same optimal base learning rates for each optimizer, and found the same stratification in performance between non-adaptive methods, Adam & RMSprop, and AdaGrad. [choi2019empirical] have scrutinized the same experiment at greater depth, testing the effects of tuning.

Figure 12: Replication of CIFAR-10 image classification experiment from [wilson2017marginal]. Results were consistent with those reported in the paper.
Optimizer Test error (original) Test error (replication)
SGD (1) (1)
HB (2) (2)
AdaGrad (5) (5)
RMSprop (3) (3)
Adam (4) (4)
AdaGrad (linear correction)
Table 4: Data for replication of CIFAR-10 experiments. Standard deviations are computed over 5 trials. The linear schedule correction discovered in Section 3.4, with , improves AdaGrad’s convergence significantly.

Char-RNN: negative replication.

Curiously, our replication of the language modeling experiment using the same popular repository444https://github.com/jcjohnson/torch-rnn was successful in reproducing the optimal hyperparameter settings, but resulted in the opposite conclusion. Here, SGD found the objective with the smallest training loss, but Adam exhibited the best generalization performance. We believe that software version discrepancies (our setup: CUDA 10.1, cuDNN 7.5.1) may account for these differences.

Figure 13: Failed replication of War & Peace character-level language modeling experiment from [wilson2017marginal]. Although every model converged, results were not consistent with the paper’s findings.
Optimizer Val loss (original) Val loss (replication)
SGD (1) (4)
HB (3) (3)
AdaGrad (5) (5)
RMSprop (2) (2)
Adam (4) (1)
Table 5: Data for replication of War & Peace experiments. Standard deviations are computed over 5 trials.

PTB generative parsing: positive replication.

Next, for the experiments on the Penn Treebank [marcus1994penn] constituency parsing code555https://github.com/cdg720/emnlp2016 accompanying [charniak2016parsing], using the same architectural and training protocol modifications as specified in [wilson2017marginal], we were able to get the model to converge with each optimizer. However, for two of the settings (SGD and RMSprop), the best reported learning rates exhibited non-convergence (the fainter curves in Figure 14). Similarly as the above experiment, the ranking of optimizers’ training and generalization performance differs from that seen in the original report.

Figure 14: Failed replication of generative parsing experiments. Convergence seems to be very unstable; even after retuning hyperparameters for the wayward optimizers, results were not consistent with the paper’s findings.
Optimizer Val ppl. (original) Val ppl. (replication)
SGD (1) (2)
HB (2) (1)
AdaGrad (3) (3)
RMSprop (4) (5)
Adam (5) (4)
Table 6: Data for replication of Penn Treebank generative parsing experiments. Standard deviations are computed over 5 trials.

PTB discriminative parsing: could not run.

Finally, [wilson2017marginal] include a fourth set of experiments, generative parsing of Penn Treebank, using the code666https://github.com/jhcross/span-parser accompanying [cross2016span]. Unfortunately, this DyNet [dynet] implementation, which was last updated in 2016, encountered a fatal memory leak when training with our DyNet 2.1 setup. Along the same lines as the random-seed-tuning experiments of [henderson2018deep], this suggests that there are further technical complications to the problems of credible optimizer evaluation addressed by [schneider2019deepobs], even on well-known supervised learning benchmarks.

Appendix C Experiment Details

c.1 Efficient implementation of AdaGraft

We provide some notes on implementation concerns when applying AdaGraft to large-scale settings.

Black-box grafting.

In popular software packages, due to efficiency concerns, an optimizer implementation will often update a variable in-place. For sake of completeness, we show how to do this with some auxiliary space in Algorithm 3. Of course, this is not necessary when grafting two known optimizers; in that case, the operation can be streamlined significantly by computing the closed-form per-coordinate steps directly.

1:  Input: Optimizers ; initializer ; .
2:  Initialize at .
3:  for  do
4:     Receive stochastic gradient .
5:     Save to scratch space .
6:     Run in-place: .
7:     Compute step norm: .
8:     Reload weights from scratch space: .
9:     Run in-place: .
10:     Compute step norm: .
11:     Update: .
12:  end for
Algorithm 3 AdaGraft, black-box implementation

Numerical considerations.

Although the global step size scalar supplied to can be arbitrary (it never appears in the), when using in-place operations, numerical precision can be lost when the update is much smaller than the itself. We suggest using a multiplier of to minimize this problem; choosing scales like causes grafting to be numerically unstable. This is especially pertinent with huge models trained using mixed-precision floating point numbers.


We note that global grafting can be significantly slower in realistic large-scale training pipelines, since it is harder to parallelize. In particular, the grafted step magnitudes depend on a round of synchronization after computing all of the per-layer gradients. Meanwhile, distributed training frameworks provide per-parameter abstractions which naturally support the straightforward implementation of per-layer grafting.

c.2 ImageNet classification

The ResNet-50 models were trained with batch size 4096, with regularization and label smoothing 0.1.

For the base (ungrafted) optimizers, we follows a staircase learning rate schedule where learning rate is ramped up linearly from 0 to 6.4 over the first 5 epochs, followed by

drop in learning rate at 30 epochs, 60 epochs and 80 epochs. Momentum is set to throughout. For Adam, we use the default . We used the rule we introduced for the adaptive optimizers. Global learning rate scalars for SGD, Adam, and AdaGrad are set to , , and , respectively; these were found via coarse grid search. In fact, global learning rate was the only hyperparameter we tuned separately; we derived the rest from the adaptive optimizers’ defaults, along with a well-tuned SGD setup. Importantly, note that no experimental conclusion depends on the premise that an optimizer setup’s hyperparameters have been tuned to the same degree as another’s (which is very hard to fulfill or certify).

c.3 WMT14 English-French translation

The Transformer models were trained with batch size 384, and residual, input, attention, and ReLU dropout 0.1. The architecture hyperparameters are as follows: hidden state dimension 8192, 32K subword encoding tokens, embedding dimension 1024, and 12 attention heads.

For all optimizers, we used a learning rate schedule with a linear warm-up over 40K steps. For SGD, we kept the learning rate constant at the peak; for AdaGrad and Adam, we used a decay like in [shazeer2018adafactor]. Like in the ImageNet experiments, momentum is set to throughout, Adam’s , rule was used for the adaptive methods. Again via grid search, we found global learning rates of 0.03, 0.015, and 0.03 for SGD, Adam, and AdaGrad, respectively.

Appendix D Proofs

d.1 Pathological constructions

In this section we provide two simple examples of stochastic convex problems where it can be seen that when it comes to generalization both AdaGrad and SGD can be significantly better than the other depending on the instance. Our purpose to provide both the examples is to stress our point that the issue of understanding the generalization performance of SGD vs. adaptive methods is more nuanced than what simple examples might suggest and hence such examples should be treated as qualitative indicators more for the purpose of providing intuition. Indeed which algorithm will perform better on a given problem, depends on various properties of the precise instance.

Example where SGD AdaGrad.

We provide a brief intuitive review of the construction provided by [wilson2017marginal]

; for a precise description, see Section 3.3 of that paper. Consider a setting of overparameterized linear regression, where the true output (i.e. dependent variable)

is the first coordinate of the feature vector (independent variable) . The next two coordinates of are “dummy” coordinates set to 1; then, the coordinates are arranged in blocks which only appear once per sample, taking the value of .

The key idea is that in this setting, the solution space that AdaGrad explores is always in the subspace of the sign vector of . As a result, AdaGrad treats the first three coordinates essentially indistinguishably putting equal mass on each. It can then be seen that for any new example the AdaGrad solution does not extract the true label information from the first three coordinates and hence gets the prediction wrong, leading to high generalization error; the other distinguishing features belong to the new unique block which are set to 0 for the AdaGrad solution, as it has not seen them.

Example where AdaGrad Sgd.

This example is motivated from the original AdaGrad paper [duchi2011adaptive], adapted to the overparameterized setting. Consider a distribution supported over with equal mass over vectors with exactly one and mass everywhere else. Let the label distribution be always . Consider sampling a dataset of size where (corresponding to the overparameterized setting) and consider the hinge loss

where denotes the -th (example, label) pair. Note that there is an optimal predictor given by the all-ones vector.

Running AdaGrad in such a setting, it can be seen that the first time a vector that has not appeared yet is sampled, AdaGrad quickly adapts by setting the coordinate corresponding to the vector to and thereby making error on the example. Therefore after one epoch of AdaGrad ( steps), the training error reduces to and the average test error becomes roughly . On the other hand, for SGD (with an optimal decay scheme) after say steps, the learning rate reduces to at most and therefore in the next steps SGD reduces the error at most by a factor of , leading to a total test error of at least after a total of steps. This is significantly smaller than the error achieved by AdaGrad at this stage. Further note that to get down to the same test error as that achieved by AdaGrad, it can be seen that SGD requires at least times more steps than AdaGrad.

d.2 Regret of AdaGrad with

In Section 5.2, we give a method for setting to 0, to ensure that an adaptive optimizer is not interpolating with SGD. We give a brief note on why is unnecessary to establish the regret bound from [duchi2011adaptive]. The general update is performed as follows:

where the preconditioning matrix is updated in two possible ways:

The following theorem whose proof is very similar to the original analysis in [duchi2011adaptive] shows that the above modification leads to no change in the regret guarantee of AdaGrad.

Theorem 3.

The regret of AdaGrad, with pseudoinverse updates, is bounded as


We now provide a quick proof sketch for the above highlighting the main parts of the proof that change from the standard version. As in the original proof we consider the case of linear loss functions at every step given by

. The first step is to note that the following relationship holds directly by the definition of the updates:

The analysis now follows in the standard way by summing the above over time and analyzing the first and second summation separately. The first term is the same as the standard analysis and therefore leads to no change.

Further more for the second term the idea of the original proof is to show that

The above statement follows in the same way as the original proof with care for pseudoinverses. For instance in the diagonal version we can apply Lemma 4 from [duchi2011adaptive] along each coordinate separately applying the lemma from the first time the coordinate sees a non-zero gradient and ignoring everything before as it is 0.