# On the Convergence of Adam and Beyond

## Authors

• 22 publications
• 14 publications
• 48 publications

Adaptive learning rate methods have been successfully applied in many fi...
01/01/2021 ∙ by Hui Zhong, et al. ∙ 0

• ### BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model

Recent methods have significantly reduced the performance degradation of...
09/29/2020 ∙ by Junjie Liu, et al. ∙ 0

In several recently proposed stochastic optimization methods (e.g. RMSPr...
04/11/2018 ∙ by Noam Shazeer, et al. ∙ 0

• ### ADAMT: A Stochastic Optimization with Trend Correction Scheme

01/17/2020 ∙ by Bingxin Zhou, et al. ∙ 0

• ### Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties

10/03/2020 ∙ by Brett Daley, et al. ∙ 0

• ### The Role of Memory in Stochastic Optimization

07/02/2019 ∙ by Antonio Orvieto, et al. ∙ 6

• ### Reducing the variance in online optimization by transporting past gradients

06/08/2019 ∙ by Sébastien M. R. Arnold, et al. ∙ 0

## Code Repositories

Tensorflow (python) implementation of the AMSGrad algorithm.

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Stochastic gradient descent (Sgd) is the dominant method to train deep networks today. This method iteratively updates the parameters of a model by moving them in the direction of the negative gradient of the loss evaluated on a minibatch. In particular, variants of Sgd that scale coordinates of the gradient by square roots of some form of averaging of the squared coordinates in the past gradients have been particularly successful, because they automatically adjust the learning rate on a per-feature basis. The first popular algorithm in this line of research is Adagrad (Duchi et al., 2011; McMahan & Streeter, 2010), which can achieve significantly better performance compared to vanilla Sgd when the gradients are sparse, or in general small.

works well for sparse settings, its performance has been observed to deteriorate in settings where the loss functions are nonconvex and gradients are dense due to rapid decay of the learning rate in these settings since it uses all the past gradients in the update. This problem is especially exacerbated in high dimensional problems arising in deep learning. To tackle this issue, several variants of

In this paper, we analyze this situation in detail. We rigorously prove that the intuition conveyed in the above paragraph is indeed correct; that limiting the reliance of the update on essentially only the past few gradients can indeed cause significant convergence issues. In particular, we make the following key contributions:

• We elucidate how the exponential moving average in the RMSprop and Adam algorithms can cause non-convergence by providing an example of simple convex optimization problem where RMSprop and Adam provably do not converge to an optimal solution. Our analysis easily extends to other algorithms using exponential moving averages such as Adadelta and Nadam as well, but we omit this for the sake of clarity. In fact, the analysis is flexible enough to extend to other algorithms that employ averaging squared gradients over essentially a fixed size window (for exponential moving averages, the influences of gradients beyond a fixed window size becomes negligibly small) in the immediate past. We omit the general analysis in this paper for the sake of clarity.

• The above result indicates that in order to have guaranteed convergence the optimization algorithm must have “long-term memory” of past gradients. Specifically, we point out a problem with the proof of convergence of the Adam algorithm given by Kingma & Ba (2015). To resolve this issue, we propose new variants of Adam which rely on long-term memory of past gradients, but can be implemented in the same time and space requirements as the original Adam algorithm. We provide a convergence analysis for the new variants in the convex setting, based on the analysis of Kingma & Ba (2015), and show a data-dependent regret bound similar to the one in Adagrad.

• We provide a preliminary empirical study of one of the variants we proposed and show that it either performs similarly, or better, on some commonly used problems in machine learning.

## 2 Preliminaries

#### Notation.

We use to denote the set of all positive definite

matrices. With slight abuse of notation, for a vector

and a positive definite matrix , we use to denote , to denote -norm of row of M and to represent . Furthermore, for any vectors , we use for element-wise square root, for element-wise square, to denote element-wise division and to denote element-wise maximum. For any vector , denotes its coordinate where . The projection operation for is defined as for . Finally, we say has bounded diameter if for all .

#### Optimization setup.

A flexible framework to analyze iterative optimization methods is the online optimization problem in the full information feedback setting. In this online setup, at each time step , the optimization algorithm picks a point (i.e. the parameters of the model to be learned) , where is the feasible set of points. A loss function (to be interpreted as the loss of the model with the chosen parameters in the next minibatch) is then revealed, and the algorithm incurs loss . The algorithm’s regret at the end of rounds of this process is given by . Throughout this paper, we assume that the feasible set has bounded diameter and is bounded for all and .

Our aim to is to devise an algorithm that ensures , which implies that on average, the model’s performance converges to the optimal one. The simplest algorithm for this setting is the standard online gradient descent algorithm (Zinkevich, 2003), which moves the point in the opposite direction of the gradient while maintaining the feasibility by projecting onto the set via the update rule , where denotes the projection of onto the set i.e., , and is typically set to for some constant . The aforementioned online learning problem is closely related to the stochastic optimization problem: , popularly referred to as empirical risk minimization (ERM), where is a training example drawn training sample over which a model with parameters is to be learned, and is the loss of the model with parameters on the sample . In particular, an online optimization algorithm with vanishing average regret yields a stochastic optimization algorithm for the ERM problem (Cesa-Bianchi et al., 2004). Thus, we use online gradient descent and stochastic gradient descent (Sgd) synonymously.

We now provide a framework of adaptive methods that gives us insights into the differences between different adaptive methods and is useful for understanding the flaws in a few popular adaptive methods. Algorithm 1 provides a generic adaptive framework that encapsulates many popular adaptive methods. Note the algorithm is still abstract because the “averaging” functions and have not been specified. Here and . For ease of exposition, we refer to as step size and as learning rate of the algorithm and furthermore, restrict ourselves to diagonal variants of adaptive methods encapsulated by Algorithm 1 where . We first observe that standard stochastic gradient algorithm falls in this framework by using:

 ϕt(g1,…,gt)=gt  and  ψt(g1,…,gt)=I, (Sgd)

and for all . While the decreasing step size is required for convergence, such an aggressive decay of learning rate typically translates into poor empirical performance. The key idea of adaptive methods is to choose averaging functions appropriately so as to entail good convergence. For instance, the first adaptive method Adagrad (Duchi et al., 2011), which propelled the research on adaptive methods, uses the following averaging functions:

and step size for all . In contrast to a learning rate of in Sgd, such a setting effectively implies a modest learning rate decay of for . When the gradients are sparse, this can potentially lead to huge gains in terms of convergence (see Duchi et al. (2011)). These gains have also been observed in practice for even few non-sparse settings.

#### Adaptive methods based on Exponential Moving Averages.

for some . This update can alternatively be stated by the following simple recursion:

 mt,i=β1mt−1,i+(1−β1)gt,i and vt,i=β2vt−1,i+(1−β2)g2t,i (1)

and and for all . and . A value of and is typically recommended in practice. We note the additional projection operation in Algorithm 1 in comparison to Adam. When , the projection operation is an identity operation and this corresponds to the algorithm in (Kingma & Ba, 2015). For theoretical analysis, one requires for , although, a more aggressive choice of constant step size seems to work well in practice. RMSprop, which appeared in an earlier unpublished work (Tieleman & Hinton, 2012) is essentially a variant of Adam with . In practice, especially in deep learning applications, the momentum term arising due to non-zero appears to significantly boost the performance. We will mainly focus on Adam algorithm due to this generality but our arguments also apply to RMSprop and other algorithms such as Adadelta, Nadam.

## 3 The Non-Convergence of Adam

With the problem setup in the previous section, we discuss fundamental flaw in the current exponential moving average methods like Adam. We show that Adam can fail to converge to an optimal solution even in simple one-dimensional convex settings. These examples of non-convergence contradict the claim of convergence in (Kingma & Ba, 2015), and the main issue lies in the following quantity of interest:

 Γt+1=(√Vt+1αt+1−√Vtαt). (2)

This quantity essentially measures the change in the inverse of learning rate of the adaptive method with respect to time. One key observation is that for Sgd and Adagrad, for all . This simply follows from update rules of Sgd and Adagrad in the previous section. In particular, update rules for these algorithms lead to “non-increasing” learning rates. However, this is not necessarily the case for exponential moving average variants like Adam and RMSprop i.e., can potentially be indefinite for . We show that this violation of positive definiteness can lead to undesirable convergence behavior for Adam and RMSprop. Consider the following simple sequence of linear functions for :

 ft(x)={Cx,for tmod3=1−x,otherwise,

where . For this function sequence, it is easy to see that the point provides the minimum regret. Suppose and . We show that Adam converges to a highly suboptimal solution of for this setting. Intuitively, the reasoning is as follows. The algorithm obtains the large gradient once every 3 steps, and while the other 2 steps it observes the gradient , which moves the algorithm in the wrong direction. The large gradient is unable to counteract this effect since it is scaled down by a factor of almost for the given value of , and hence the algorithm converges to rather than . We formalize this intuition in the result below.

###### Theorem 1.

There is an online convex optimization problem where Adam has non-zero average regret i.e., as .

We relegate all proofs to the appendix. A few remarks are in order. One might wonder if adding a small constant in the denominator of the update helps in circumventing this problem i.e., the update for Adam in Algorithm 1 of is modified as follows:

 ^xt+1=xt−αtmt/√Vt+ϵI. (3)

The algorithm in (Kingma & Ba, 2015) uses such an update in practice, although their analysis does not. In practice, selection of the parameter appears to be critical for the performance of the algorithm. However, we show that for any constant , there exists an online optimization setting where, again, Adam has non-zero average regret asymptotically (see Theorem 6 in Section F of the appendix).

The above examples of non-convergence are catastrophic insofar that Adam and RMSprop converge to a point that is worst amongst all points in the set . Note that above example also holds for constant step size . Also note that classic Sgd and Adagrad

do not suffer from this problem and for these algorithms, average regret asymptotically goes to 0. This problem is especially aggravated in high dimensional settings and when the variance of the gradients with respect to time is large. This example also provides intuition for why large

is advisable while using Adam algorithm, and indeed in practice using large helps. However the following result shows that for any constant and with , we can design an example where Adam has non-zero average rate asymptotically.

###### Theorem 2.

For any constant such that , there is an online convex optimization problem where Adam has non-zero average regret i.e., as .

The above results show that with constant and , momentum or regularization via will not help in convergence of the algorithm to the optimal solution. Note that the condition is benign and is typically satisfied in the parameter settings used in practice. Furthermore, such condition is assumed in convergence proof of Kingma & Ba (2015). We can strengthen this result by providing a similar example of non-convergence even in the easier stochastic optimization setting:

###### Theorem 3.

For any constant such that , there is a stochastic convex optimization problem for which Adam does not converge to the optimal solution.

These results have important consequences insofar that one has to use “problem-dependent” and in order to avoid bad convergence behavior. In high-dimensional problems, this typically amounts to using, unlike the update in Equation (3), a different and for each dimension. However, this defeats the purpose of adaptive methods since it requires tuning a large set of parameters. We would also like to emphasize that while the example of non-convergence is carefully constructed to demonstrate the problems in Adam, it is not unrealistic to imagine scenarios where such an issue can at the very least slow down convergence.

We end this section with the following important remark. While the results stated above use constant and , the analysis of Adam in (Kingma & Ba, 2015) actually relies on decreasing over time. It is quite easy to extend our examples to the case where is decreased over time, since the critical parameter is rather than , and as long as is bounded away from , our analysis goes through. Thus for the sake of clarity, in this paper we only prove non-convergence of Adam in the setting where is held constant.

## 4 A New Exponential Moving Average Variant: AMSGrad

In this section, we develop a new principled exponential moving average variant and provide its convergence analysis. Our aim is to devise a new strategy with guaranteed convergence while preserving the practical benefits of Adam and RMSprop. To understand the design of our algorithms, let us revisit the quantity in (2). For Adam and RMSprop, this quantity can potentially be negative. The proof in the original paper of Adam erroneously assumes that is positive semi-definite and is hence, incorrect (refer to Appendix D for more details). For the first part, we modify these algorithms to satisfy this additional constraint. Later on, we also explore an alternative approach where can be made positive semi-definite by using values of and that change with .

AMSGrad uses a smaller learning rate in comparison to Adam and yet incorporates the intuition of slowly decaying the effect of past gradients on the learning rate as long as is positive semi-definite. Algorithm 2 presents the pseudocode for the algorithm. The key difference of AMSGrad with Adam is that it maintains the maximum of all until the present time step and uses this maximum value for normalizing the running average of the gradient instead of in Adam. By doing this, AMSGrad results in a non-increasing step size and avoids the pitfalls of Adam and RMSprop i.e., for all even with constant . Also, in Algorithm 2, one typically uses a constant in practice (although, the proof requires a decreasing schedule for proving convergence of the algorithm).

###### Theorem 4.

Let and be the sequences obtained from Algorithm 2, , , for all and . Assume that has bounded diameter and for all and . For generated using the AMSGrad (Algorithm 2), we have the following bound on the regret

 RT≤D2∞√Tα(1−β1)d∑i=1^v1/2T,i+D2∞(1−β1)2T∑t=1d∑i=1β1t^v1/2t,iαt+α√1+logT(1−β1)2(1−γ)√(1−β2)d∑i=1∥g1:T,i∥2.

The following result falls as an immediate corollary of the above result.

###### Corollary 1.

Suppose in Theorem 4, then we have

 RT≤D2∞√Tα(1−β1)d∑i=1^v1/2T,i+β1D2∞G∞(1−β1)2(1−λ)2+α√1+logT(1−β1)2(1−γ)√(1−β2)d∑i=1∥g1:T,i∥2.

The above bound can be considerably better than regret of Sgd when and (Duchi et al., 2011). Furthermore, in Theorem 4, one can use a much more modest momentum decay of and still ensure a regret of . We would also like to point out that one could consider taking a simple average of all the previous values of instead of their maximum. The resulting algorithm is very similar to Adagrad except for normalization with smoothed gradients rather than actual gradients and can be shown to have similar convergence as Adagrad.

## 5 Experiments

In this section, we present empirical results on both synthetic and real-world datasets. For our experiments, we study the problem of multiclass classification using logistic regression and neural networks, representing convex and nonconvex settings, respectively.

Synthetic Experiments: To demonstrate the convergence issue of Adam, we first consider the following simple convex setting inspired from our examples of non-convergence:

 ft(x)={1010x,for tmod101=1−10x,otherwise,

with the constraint set . We first observe that, similar to the examples of non-convergence we have considered, the optimal solution is ; thus, for convergence, we expect the algorithms to converge to . For this sequence of functions, we investigate the regret and the value of the iterate for Adam and AMSGrad. To enable fair comparison, we set and for Adam and AMSGrad algorithm, which are typically the parameters settings used for Adam in practice. Figure 1 shows the average regret () and value of the iterate () for this problem. We first note that the average regret of Adam does not converge to with increasing . Furthermore, its iterates converge to , which unfortunately has the largest regret amongst all points in the domain. On the other hand, the average regret of AMSGrad converges to and its iterate converges to the optimal solution. Figure 1 also shows the stochastic optimization setting:

 ft(x)={1010x,with probability 0.01−10x,otherwise.

Similar to the aforementioned online setting, the optimal solution for this problem is . Again, we see that the iterate of Adam converges to the highly suboptimal solution .

Logistic Regression: To investigate the performance of the algorithm on convex problems, we compare AMSGrad with Adam on logistic regression problem. We use MNIST dataset for this experiment, the classification is based on 784 dimensional image vector to one of the 10 class labels. The step size parameter is set to for both Adam and AMSGrad in for our experiments, consistent with the theory. We use a minibatch version of these algorithms with minibatch size set to . We set and is chosen from the set , but they are fixed throughout the experiment. The parameters and are chosen by grid search. We report the train and test loss with respect to iterations in Figure 2. We can see that AMSGrad performs better than Adam with respect to both train and test loss. We also observed that AMSGrad is relatively more robust to parameter changes in comparison to Adam.

Neural Networks: For our first experiment, we trained a simple 1-hidden fully connected layer neural network for the multiclass classification problem on MNIST. Similar to the previous experiment, we use and is chosen from

. We use a fully connected 100 rectified linear units (ReLU) as the hidden layer for this experiment. Furthermore, we use constant

throughout all our experiments on neural networks. Such a parameter setting choice of Adam is consistent with the ones typically used in the deep learning community for training neural networks. A grid search is used to determine parameters that provides the best performance for the algorithm.

Finally, we consider the multiclass classification problem on the standard CIFAR-10 dataset, which consists of 60,000 labeled examples of images. We use Cifarnet

, a convolutional neural network (CNN) with several layers of convolution, pooling and non-linear units, for training a multiclass classifer for this problem. In particular, this architecture has 2 convolutional layers with 64 channels and kernel size of

followed by 2 fully connected layers of size 384 and 192. The network uses max pooling and layer response normalization between the convolutional layers (Krizhevsky et al., 2012). A dropout layer with keep probability of 0.5 is applied in between the fully connected layers (Srivastava et al., 2014). The minibatch size is also set to 128 similar to previous experiments. The results for this problem are reported in Figure 2. The parameters for Adam and AMSGrad are selected in a way similar to the previous experiments. We can see that AMSGrad performs considerably better than Adam on train loss and accuracy. Furthermore, this performance gain also translates into good performance on test loss.

An alternative approach is to use an increasing schedule of in Adam. This approach, unlike Algorithm 2 does not require changing the structure of Adam but rather uses a non-constant and . The pseudocode for the algorithm, AdamNc, is provided in the appendix (Algorithm 3). We show that by appropriate selection of and , we can achieve good convergence rates.

###### Theorem 5.

Let and be the sequences obtained from Algorithm 3, , and for all . Assume that has bounded diameter and for all and . Furthermore, let be such that the following conditions are satisfied:

1. for some and all , .

2. for all and .

Then for generated using the AdamNc (Algorithm 3), we have the following bound on the regret

 RT≤D2∞2α(1−β1)d∑i=1√Tv1/2T,i+D2∞(1−β1)2T∑t=1d∑i=1β1tv1/2t,iαt+2ζ(1−β1)3d∑i=1∥g1:T,i∥2.

The above result assumes selection of such that for all . However, one can generalize the result to deal with the case where this constraint is violated as long as the violation is not too large or frequent. Following is an immediate consequence of the above result.

###### Corollary 2.

Suppose and in Theorem 5, then we have

 D2∞2α(1−β1)d∑i=1∥g1:T,i∥2+β1D2∞G∞(1−β1)2(1−λ)2+2ζ(1−β1)3d∑i=1∥g1:T,i∥2.

The above corollary follows from a trivial fact that for all when . This corollary is interesting insofar that such a parameter setting effectively yields a momentum based variant of Adagrad. Similar to Adagrad, the regret is data-dependent and can be considerably better than regret of Sgd when (Duchi et al., 2011). It is easy to generalize this result for setting similar settings of . Similar to Corollary 1, one can use a more modest decay of and still ensure a data-dependent regret of .

## 6 Discussion

In this paper, we study exponential moving variants of Adagrad and identify an important flaw in these algorithms which can lead to undesirable convergence behavior. We demonstrate these problems through carefully constructed examples where RMSprop and Adam converge to highly suboptimal solutions. In general, any algorithm that relies on an essentially fixed sized window of past gradients to scale the gradient updates will suffer from this problem.

We proposed fixes to this problem by slightly modifying the algorithms, essentially endowing the algorithms with a long-term memory of past gradients. These fixes retain the good practical performance of the original algorithms, and in some cases actually show improvements.

The primary goal of this paper is to highlight the problems with popular exponential moving average variants of Adagrad from a theoretical perspective. RMSprop and Adam have been immensely successful in development of several state-of-the-art solutions for a wide range of problems. Thus, it is important to understand their behavior in a rigorous manner and be aware of potential pitfalls while using them in practice. We believe this paper is a first step in this direction and suggests good design principles for faster and better stochastic optimization.

## References

• Auer & Gentile (2000) Peter Auer and Claudio Gentile. Adaptive and self-confident on-line learning algorithms. In Proceedings of the 13th Annual Conference on Learning Theory, pp. 107–117, 2000.
• Cesa-Bianchi et al. (2004) Nicolò Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50:2050–2057, 2004.
• Dozat (2016) Timothy Dozat.

In Proceedings of 4th International Conference on Learning Representations, Workshop Track, 2016.
• Duchi et al. (2011) John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, 2011.
• Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of 3rd International Conference on Learning Representations, 2015.
• Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1097–1105, 2012.
• McMahan & Streeter (2010) H. Brendan McMahan and Matthew J. Streeter. Adaptive bound optimization for online convex optimization. In Proceedings of the 23rd Annual Conference On Learning Theory, pp. 244–256, 2010.
• Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.
• Tieleman & Hinton (2012) T. Tieleman and G. Hinton. RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
• Zinkevich (2003) Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning, pp. 928–936, 2003.

## Appendix A Proof of Theorem 1

###### Proof.

We consider the setting where are linear functions and . In particular, we define the following function sequence:

 ft(x)={Cx,for tmod3=1−x,otherwise,

where . For this function sequence, it is easy to see that the point provides the minimum regret. Without loss of generality, assume that the initial point is . This can be assumed without any loss of generality because for any choice of initial point, we can always translate the coordinate system such that the initial point is in the new coordinate system and then choose the sequence of functions as above in the new coordinate system. Also, since the problem is one-dimensional, we drop indices representing coordinates from all quantities in Algorithm 1. Consider the execution of Adam algorithm for this sequence of functions with

 β1=0,β2=11+C2 and αt=α√t

where . Note that since gradients of these functions are bounded, has bounded diameter and . Hence, the conditions on the parameters required for Adam are satisfied (refer to (Kingma & Ba, 2015) for more details).

Our main claim is that for iterates arising from the updates of Adam, we have for all and furthermore, for all . For proving this, we resort to the principle of mathematical induction. Since , both the aforementioned conditions hold for the base case. Suppose for some , we have for all and . Our aim is to prove that and are positive and . We first observe that the gradients have the following form:

 ∇fi(x)={C,for imod3=1−1,otherwise

From update of Adam in Equation (1), we obtain

 ^x3t+2=x3t+1−αC√(3t+1)(β2v3t+(1−β2)C2)=1−αC√(3t+1)(β2v3t+(1−β2)C2).

The equality follows from the induction hypothesis. We observe the following:

 αC√(3t+1)(β2v3t+(1−β2)C2) ≤αC√(3t+1)(1−β2)C2 =α√(3t+1)(1−β2))<1. (4)

The second inequality follows from the step size choice that . Therefore, we have and hence . Furthermore, after the and updates of Adam in Equation (1), we have the following:

 ^x3t+3=x3t+2+α√(3t+2)(β2v3t+1+(1−β2)), ^x3t+4=x3t+3+α√(3t+3)(β2v3t+2+(1−β2)).

Since , it is easy to see that . To complete the proof, we need to show that . In order to prove this claim, we show that , which readily translates to because and here is the simple Euclidean projection (note that in one-dimension, ). We observe the following:

 ^x3t+4=min(^x3t+3,1)+α√(3t+3)(β2v3t+2+(1−β2)).

The above equality is due to the fact that and property of projection operation onto the set . We consider the following two cases:

1. Suppose , then it is easy to see from the above equality that .

2. Suppose , then we have the following:

 ^x3t+4 =^x3t+3+α√(3t+3)(β2v3t+2+(1−β2)) =x3t+2+α√(3t+2)(β2v3t+1+(1−β2))+α√(3t+3)(β2v3t+2+(1−β2)) =1−αC√(3t+1)(β2v3t+(1−β2)C2)+α√(3t+2)(β2v3t+1+(1−β2)) +α√(3t+3)(β2v3t+2+(1−β2)).

The third equality is due to the fact that . Thus, to prove , it is enough to the prove:

 αC√(3t+1)(β2v3t+(1−β2)C2)T1 ≤α√(3t+2)(β2v3t+1+(1−β2)) +α√(3t+3)(β2v3t+2+(1−β2))T2

We have the following bound on term from Equation (4):

 T1≤α√(3t+1)(1−β2)). (5)

Furthermore, we lower bound in the following manner:

 T2 =α√(3t+2)(β2v3t+1+(1−β2))+α√(3t+3)(β2v3t+2+(1−β2)) ≥α√β2C2+(1−β2)(1√3t+2+1√3t+3) ≥α√β2C2+(1−β2)(1√2(3t+1)+1√2(3t+1)) =√2α√(3t+1)(β2C2+(1−β2))=α√(3t+1)(1−β2)≥T1. (6)

The first inequality is due to the fact that for all . The last inequality follows from inequality in Equation (5). The last equality is due to following fact:

 √β2C2+(1−β2)2=√1−β2

for the choice of . Therefore, we have and hence, .

Therefore, from both the cases, we see that . Therefore, by the principle of mathematical induction it holds for all . Thus, we have

 f3t+1(x3t+1)+f3t+2(x3t+2)+f3t+2(x3t+2)−f3t+1(−1)−f3t+2(−1)−f3t+3(−1)≥2C−4=2C−4.

Therefore, for every 3 steps, Adam suffers a regret of at least . More specifically, . Since , this regret can be very large and furthermore, as , which completes the proof. ∎

## Appendix B Proof of Theorem 2

###### Proof.

The proof generalizes the optimization setting used in Theorem 1. Throughout the proof, we assume , which is also a condition (Kingma & Ba, 2015) assume in their paper. In this proof, we consider the setting where are linear functions and . In particular, we define the following function sequence:

 ft(x)={Cx,for tmodC=1−x,otherwise,

where , satisfies the following:

 (1−β1)βC−11C≤1−βC−11, β(C−2)/22C2≤1, 3(1−β1)2√1−β2(1+γ(1−γC−1)1−γ)+βC/2−111−β1

where . It is not hard to see that these conditions hold for large constant that depends on and . Since the problem is one-dimensional, we drop indices representing coordinates from all quantities in Algorithm 1. For this function sequence, it is easy to see that the point provides the minimum regret since . Furthermore, the gradients have the following form:

 ∇fi(x)={C,for tmodC=1−1,otherwise

Our first observation is that for all . For , this holds trivially due to our initialization. For the general case, observe the following:

 mkC+C =−(1−β1)−(1−β1)β1−⋯−(1−β1)βC−21+(1−β1)βC−11C+βC1mkC (8) =−(1−βC−11)+(1−β1)βC−11C+βC1mkC. (9)

If , it can be easily shown that for our selection of in Equation (7) by using the principle of mathematical induction. With this observation we continue to the main part of the proof. Let be such that for all where . All our analysis focuses on iterations . Note that any regret before is just a constant because is independent of and thus, the average regret is negligible as . Consider an iterate at time step of the form after . Our claim is that

 xt+C≥min{xt+ct,1} (10)

for some . To see this, consider the updates of Adam for the particular sequence of functions we considered are:

 xt+1 =ΠF(xt−α√t(1−β1)C+β1mt√(1−β2)C2+β2vt), xt+i =ΠF(xt+i−1−α√t+i−1−(1−β1)+β1mt+i−1√(1−β2)+β2vt+i−1) for i∈{2,⋯,C}.

For , we use the following notation:

 δt =−α√t(1−β1)C+β1mt√(1−