In the past few years, online algorithms have emerged as state-of-the-art techniques for solving large-scale machine learning problems [2, 13, 16]. In addition to their simplicity and generality, online algorithms are natural choices for problems where new data is constantly arriving and rapid adaptation is imporant.
Compared to the study of convex optimization in the batch (offline) setting, the study of online convex optimization is relatively new. In light of this, it is not surprising that performance-improving techniques that are well known and widely used in the batch setting do not yet have online analogues. In particular, convergence rates in the batch setting can often be dramatically improved through the use of preconditioning. Yet, the online convex optimization literature provides no comparable method for improving regret (the online analogue of convergence rates).
A simple and effective form of preconditioning is to re-parameterize the loss function so that its magnitude is the same in all coordinate directions. Without this modification, a batch algorithm such as gradient descent will tend to take excessively small steps along some axes and to oscillate back and forth along others, slowing convergence. In the online setting, this rescaling cannot be done up front because the loss functions vary over time and are not known in advance. As a result, when existing no-regret algorithms for online convex optimization are applied to machine learning problems, they tend to overfit the data with respect to certain features and underfit with respect to others (we give a concrete example of this behavior in §2).
We show that this problem can be overcome in a principled way by using
online gradient descent111 When loss functions are drawn IID, as
when online gradient descent is applied to a batch learning problem,
the term stochastic gradient descent is often used.
When loss functions are drawn IID, as when online gradient descent is applied to a batch learning problem, the term stochastic gradient descent is often used.with adaptive, per-coordinate learning rates. Our algorithm comes with worst-case regret bounds (see Theorem 3
) that are never worse than those of standard online gradient descent, and are much better when the magnitude of the gradients varies greatly across coordinates (this structure is common in large-scale problems of practical interest). Extending this approach, we give improved bounds for generalized notions of strong convexity, bounds in terms of the variance of cost functions, and bounds on adaptive regret (regret against a drifting comparator). Experimentally, we show that our algorithm dramatically outperforms standard online gradient descent on real-world problems, and is competitive with state-of-the-art algorithms for online binary classification.
1.1 Background and notation
In an online convex optimization problem, we are given as input a closed, convex feasible set . On each round , we must pick a point . We then incur loss , where is a convex function. At the end of round , the loss function is revealed to us. Our regret at the end of rounds is the difference between our total loss and that of the best fixed in hindsight, that is
Sequential prediction using a generalized linear model is an important special case of online convex optimization. In this case, each
is a vector of weights, whereis the weight assigned to feature on round . On round , the algorithm makes a prediction , where is a feature vector and is a fixed link function (e.g.,
for logistic regression,
for linear regression). The algorithm then incurs loss that is some function of the predictionand the label of the example. For example, in logistic regression the loss is , and in least squares linear regression the loss is . In both of these examples, it can be shown that is a convex function of .
We are particularly interested in online gradient descent and generalizations thereof. Online gradient descent chooses arbitrarily, and thereafter plays
where , , …, is a sequence of learning rates, is a subgradient of , and is the projection operator, where is the L2 norm. When the learning rates are chosen appropriately, online gradient descent obtains regret , where is the diameter of the feasible set and is the maximum norm of the gradients. Thus, as , the average loss of the points selected by online gradient descent is as good as that of any fixed point in the feasible set. It is perhaps surprising that this performance guarantee holds for any sequence of loss functions, and in particular that the bounds holds even if the sequence is chosen adversarially.
It is well-known that batch gradient descent performs poorly in the presence of so-called ravines, surfaces that curve more steeply in some directions than in others . In this section we give examples showing that when the slope of the loss function or the size of the feasible set varies widely across coordinates, gradient descent incurs high regret in the online setting. These observations motivate the use of per-coordinate learning rates (which can be thought of as an adaptive diagonal preconditioner).
2.1 A motivating application
Consider the problem of trying to predict the probability that a user will click on an ad when it is shown alongside search results for a particular query, using a generalized linear model. For simplicity, imagine there is only one ad, and we wish to predict its click-through rate on many different queries. On a large search engine, a popular query will occur orders of magnitude more often than a rare query. For queries that occur rarely, it is necessary to use a relatively large learning rate in order for the associated feature weights to move significantly away from zero. But for popular queries, the use of such a large learning rate will cause the feature weights to oscillate wildly, and so the predictions made by the algorithm will be unstable. Thus, gradient descent with a global learning rate cannot simultaneously perform well on common queries and on rare ones. Because rare queries are more numerous than common ones, performing poorly on either category leads to substantial regret.
2.2 Tradeoffs in one dimension
We first consider gradient descent in one dimension, with a fixed learning rate (later we generalize to arbitrary non-increasing sequences of learning rates).
If is too large, the algorithm may oscillate about the optimal point and thereby incur high regret. As a simple example, suppose the feasible set is , and the loss function on each round is , for some small positive . Then if and if . It is easy to verify that if the algorithm plays initially, it will play
on odd rounds andon even rounds, assuming . Thus, after rounds the algorithm incurs total loss . Always playing would incur zero loss, so the regret is .
On the other hand, if is too small then may stay close to zero long after the data indicates that a larger would incur smaller loss. For example, suppose always. Then . For the first rounds, and therefore our per-round regret relative to the comparator is at least on these rounds. Thus, overall regret is at least , assuming that . Thus, for any choice of there exists a problem where
where the upper bound is adapted from Zinkevich . Thus, by setting (which minimizes the upper bound) we minimize worst-case regret up to a constant factor. Note that this choice of satisfies the constraints , as was assumed earlier.
The fact that the optimal choice of is proportional to captures a fundamental tradeoff. When the feasible set is large and the gradients are small, we must use a larger learning rate in order to be competitive with points in the far extremes of the feasible set. On the other hand, when the feasible set is small and the gradients are large, we must use a smaller learning rate in order to avoid the possibility of oscillating between the extremes and performing poorly relative to points in the center.
Because the relevant values of and will in general be different for different coordinates, a gradient descent algorithm that uses the same learning rate for all coordinates is doomed to either underfit on some coordinates or oscillate on others. To handle this, we must use different learning rates for different coordinates. Furthermore, because the magnitude of the gradients is not known in advance and can change over time, we must incorporate it into our choice of learning rate in an online fashion.
2.3 A bad example for global learning rates
We now exhibit a class of online convex optimization problems where the use of a coordinate-independent learning rate forces regret to grow at an asymptotically larger rate than with a per-coordinate learning rate. This result is summarized in the following theorem.
There exists a family of online convex optimization problems, parameterized by their lengths (number of rounds ), where gradient descent with a non-increasing global learning rate incurs regret at least , whereas gradient descent with an appropriate per-coordinate learning rate has regret .
The lower bound stated in Theorem 1 does not contradict the previously-stated upper bound on the regret of online gradient descent, because in this family of problems (and ).
Proof of Theorem 1.
To prove this theorem, we interleave instances of the two classes of one-dimensional subproblem discussed in §2.2, setting and setting the feasible set to . We have one subproblem of the first type, lasting for rounds, followed by subproblems of the second type, each lasting rounds. Each subproblem is assigned its own coordinate. Formally, the loss function is
On each round, only one component of the gradient vector is non-zero. Thus, running gradient descent with global learning rate is equivalent to running a separate copy of gradient descent on each subproblem, where each copy uses learning rate . Moreover, overall regret is simply the sum of the regret on each subproblem. Thus, by the lower bounds stated §2.2, regret is at least
(note that ).
If we set , this expression is . To see this, first note that if then the second term is already (note that ). Otherwise, a simple minimization over shows that the sum is . Because regret on the first subproblem is an increasing function of , and regret on all later subproblems is a decreasing function of , the same lower bound holds for any non-increasing sequence of per-round learning rates. Thus, we have proved the first part of the theorem.
Now consider the alternative of letting the learning rate for each coordinate vary independently. On a one-dimensional subproblem with feasible set and gradients of magnitude at most 1, gradient descent using learning rate on round of the subproblem obtains regret on a subproblem of length . Thus, if we ran an independent copy of this algorithm on each coordinate, we would obtain regret , which completes the proof. ∎
3 Improved Regret Bounds using Per-Coordinate Learning Rates
Zinkevich  proved bounds on the regret of online gradient descent (which chooses according to Equation (1)). Building on his analysis, we improve these bounds by adjusting the learning rates on a per-coordinate basis. Specifically, we obtain these bounds by constructing the vector by
where is a vector of learning rates, one for each coordinate. We then play . We prove bounds for feasible sets defined by axis-aligned constraints, . Many machine learning problems can be solved using feasible sets of this form, as our experiments demonstrate.222Our techniques can be extended to arbitrary feasible sets using a somewhat different algorithm, but the proofs are signicantly more technical .
3.1 A better global learning rate
We first give an improved regret bound for gradient descent with a global (coordinate-independent) learning rate. In the next subsection, we make use of this improved bound in order to prove the desired bounds on the regret of gradient descent with a per-coordinate learning rate.
Zinkevich  showed that if we run gradient descent with a non-increasing sequence of learning rates, regret is bounded by
To guard against the worst case, it is natural to choose our sequence of learning rates so as to minimize this bound. Doing so is problematic, however, because in the online setting the gradients are not known in advance. Perhaps surprisingly, we can come within a factor of of the optimal bound even without having this information up front, as the following theorem shows.
Setting yields regret , where .
We now compute . First, note that if for some then we could reduce the second term in by making smaller. Because the sequence is constrained to be non-increasing, it follows that the bound is minimized using a constant learning rate . A simple minimization then shows that it is optimal to set . which gives regret . ∎
A related result appears in , giving improved bounds in the case of strongly convex functions but worse constants than ours in the case of linear functions.
For any non-negative real numbers ,
The lemma is clearly true for . Fix some , and assume the lemma holds for . Thus,
where we define and . The derivative of the right hand side with respect to is , which is negative for . Thus, subject to the constraint , the right hand side is maximized at , and is therefore at most . ∎
3.2 A per-coordinate learning rate
We can improve the above bound by running, for each coordinate, a separate copy of gradient descent that uses the learning rate given in the previous section (see Algorithm 1). Specifically, we use the update of Equation (2) with , where is the diameter of the feasible set along coordinate .
The following theorem makes three important points about the performance of Algorithm 1: (i), its regret is bounded by a sum of per-coordinate bounds, each of the same form as (3); (ii) the algorithm’s choice of gives a regret bound that is only a factor of worse than if the bound had been optimized knowing in advance; and, (iii), the regret bound of Algorithm 1 is never worse than the bound for global learning rates stated in Theorem 2. Futhermore, as illustrated in Theorem 1, the per-coordinate bound can be better by an arbitrarily large factor if the magnitude of the gradients varies widely across coordinates.
Zinkevich showed that, so long as our algorithm only makes use of , we may assume without loss of generality that is linear, and therefore for all . If is a hypercube, then the projection operator simply projects each coordinate indepdently onto the interval . Thus, in this special case, we can think of each coordinate as solving a separate online convex optimization problem where the loss function on round is . Thus, Equation (3) implies that for each ,
Summing this bound over all , we get the regret bound
Applying Theorem 2 to each one-dimensional problem, we get .
4 Additional Improved Regret Bounds
The approach of bounding overall regret in terms of the sum of regret on a set of one-dimensional problems can be used to obtain additional regret bounds that improve over those of previous work, in the special case where the feasible set is a hypercube. The key observation is captured in the following lemma.
Consider an online optimization problem with feasible set and loss functions . For each , let be a lower bound on (i.e., for all ). Further suppose that for all , where is the sequence of points played by an online algorithm. Consider the composite online algorithm formed by running a 1-dimensional algorithm independently for each coordinate on feasible set , with loss function on round . Let
be the total regret of the composite algorithm, and let
be the regret incurred by the algorithm responsible for choosing the coordinate. Then .
Because , and ,
Importantly, for arbitrary convex functions, we can always construct such independent lower bounds by choosing , as long as we add a “bias” coordinate where . A similar observation was originally used by Zinkevich  to show that any algorithm for online linear optimization can be used for online convex optimization. We used this fact in the proof of Theorem 3, where we only analyzed the linear case.
This simple lemma has powerful ramifications. We now discuss several improved guarantees that can be obtained by applying it to known online algorithms. For simplicity, when stating these bounds we assume that the feasible set is and that the gradients of the loss functions are componentwise upper bounded by 1 (that is, for all and ).
4.1 More general notions of strong convexity
A function is -strongly convex if, for all , it holds that . Strongly convex functions arise, for example, when solving learning problems subject to L2 regularization.
Bartlett et al.  give an online convex optimization algorithm whose regret is
where is the largest constant such that each is -strongly convex. We can generalize the concept of strong convexity as follows. We say that is strongly convex with respect to the vector if, for all , . Suppose we run the algorithm of Bartlett et al. independently for each coordinate, feeding back to the algorithm responsible for choosing coordinate (we can always choose ). Applying Lemma 2, we obtain a regret bound
This bound is never worse than the previous one, and is better if the degree of strong convexity differs substantially across different coordinates (e.g., if using different L2 regularization parameters for different classes of features).
4.2 Tighter bounds in terms of variance
Hazan and Kale  give a bound on gradient descent’s regret in terms of the variance of the sequence of gradients. Specifically, their algorithm has regret , where and , where .
By running a separate copy of their algorithm on each coordinate, we can instead obtain a bound of , where .
To compare the bounds, let be a vector whose component is , and let be a vector whose components are all 1. Note that . Using the Cauchy-Schwarz inequality,
Thus, the bound obtained by running separate copies of the algorithm for each coordinate is never worse than the original bound, and is substantially better when the variance varies greatly across coordinates.
4.3 Adaptive regret
One weakness of standard regret bounds like those stated so far is that they bound performance only in terms of the static optimal solution over all rounds. In a non-stationary environment, it is desirable to obtain stronger guarantees. For example, suppose the feasible set is , for the first rounds and thereafter. Then an algorithm that plays for all has 0 regret, yet its loss on the final rounds is worse than if it had played the point for those rounds. Indeed, standard regret-minimizing algorithms fail to adapt in simple examples such as this.
Hazan and Seshadhri  define adaptive regret as the maximum, over all intervals , of the regret incurred over that interval. For -strongly convex functions, their algorithm achieves adaptive regret .
By running an independent copy of their algorithm on each coordinate, we can obtain the following guarantee. Consider an arbitrary sequence of points in , and let be the regret relative to that sequence. Holding constant for simplicity, the adaptive regret bound just stated implies that the algorithm of Hazan and Seshadhri  obtains , where is the number of values of for which (this follows by summing adaptive regret over the intervals where is constant). Using separate copies for each coordinate, we instead obtain
where is the number of values of where . This bound is never worse than the previous one, and is better when some coordinates of the vectors in change more frequently than others.
This provides an improved performance guarantee when the environment is stationary with respect to some coordinates and non-stationary with respect to others. This could happen, for example, if the effect of certain features (e.g., features for advertisers in certain business sectors) changes over time, but the effect of other features remains constant.
5 Experimental Evaluation
In this section, we evaluate gradient descent with per-coordinate learning rates experimentally on several machine learning problems.
5.1 Online binary classification
|Fraction of mistakes|
We first compare the performance of online gradient descent with that of two recent algorithms for text classification: the Passive-Aggressive (PA) algorithm , and confidence-weighted (CW) linear classification . The latter algorithm has been demonstrated to have state-of-the-art performance on large real-world problems .
We used four sentiment classification data sets (Books, Dvd, Electronics, and Kitchen), available from , each with 1000 positive examples and 1000 negative examples,333We used the features provided in processed_acl.tar.gz, and scaled each vector of counts to unit length. as well as the scaled versions of the rcv1.binary (677,399 examples) and news20.binary (19,996 examples) data sets from LIBSVM . For each data set, we shuffled the examples and then ran each algorithm for one pass over the data, computing the loss on each event before training on it.
For the online gradient descent algorithms, we set for . We found that the learning rate suggested by Theorem 3 was too aggressive in practice when the feasible set is large (note that it moves a feature’s weight to the maximum value the first time it sees a non-zero gradient for that feature). In order to improve performance, we did some parameter tuning. For Algorithm 1 (Per-Coord), we scaled the learning rate formula by a factor of , and for the global learning rate (Global) we scaled it by
. We estimate the diameterin the global learning rate formula online, based on the number of attributes seen so far. For CW, we found that the parameters and worked well in practice.
Table 1 presents average hinge loss and the fraction of classification mistakes for each algorithm. The Global and Per-Coord algorithms are designed to minimize hinge loss, and at this objective the Per-Coord algorithm consistently wins. CW and PA are designed to maximize classification accuracy, and on this objective Per-Coord and CW are the best algorithms. The fact that the classification accuracy of Per-Coord is comparable to that of a state-of-the-art binary classification algorithm is impressive given the former algorithm’s generality (i.e., its applicability to arbitrary online convex optimization problems such as online shortest paths).
5.2 Large-scale logistic regression
We collected data from a large search engine444No user-specific data was used in these experiments. consisting of random samples of queries that contained a particular phrase, for example “auto insurance”. Each data set has a few million examples. We transformed this data into an online logistic regression problem with a feature vector for each ad impression, using features based on the text of the ad and the query. The target label is 1 if the ad was clicked, and -1 otherwise. The loss function is the sum of the logistic loss, , and an L2 regularization term.
We compare gradient descent using the global learning rate from §3.1 with gradient descent using the per-coordinate rate given in §3.2. We scaled the formulas given in those sections by 0.1; this improved performance for both algorithms but did not change the relative comparison. The feasible set was .
Table 2 shows the regret incurred by the two algorithms on various data sets. Gradient descent with a per-coordinate learning rate consistently obtains an order of magnitude lower regret than with a global learning rate. To calculate regret, we computed the static optimal loss by running our per-coordinate algorithm through the data many times until convergence.
6 Related Work
The use of different learning rates for different coordinates has been investigated extensively in the neural network community. There the focus has been on empirical performance in the batch setting, and a large number of algorithms have been developed; see for example. These algorithms are not designed to perform well in an adversarial online setting, and for many of them it is straightforward to construct examples where the algorithm incurs high regret.
More recently, Hsu et al.  gave an algorithm for choosing per-coordinate learning rates for gradient descent, derive asymptotic rates of convergence in the batch setting, and present a number of positive experimental results.
Confidence-weighted linear classification  and AROW  are similar to our algorithm in that they make different-sized adjustments for different coordinates, and in that common features are updated less aggressively than rare ones. Unlike our algorithm, these algorithms apply only to classification problems and not to general online convex optimization, and the guarantees are in the form of mistake bounds rather than regret bounds.
-  Peter L. Bartlett, Elad Hazan, and Alexander Rakhlin. Adaptive online gradient descent. In NIPS, 2008.
-  Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems, volume 20. 2008.
-  Chih-Chung Chang and Chih-Jen Lin. LIBSVM data sets. http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/, 2010.
-  Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. Online passive-aggressive algorithms. J. Mach. Learn. Res., 7, 2006.
-  Koby Crammer, Alex Kulesza, and Mark Drezde. Adaptive regularization of weight vectors. In NIPS, 2009.
-  Mark Dredze. Multi-domain sentiment dataset (v2.0). http://www.cs.jhu.edu/~mdredze/datasets/sentiment/, 2010.
-  Mark Drezde, Koby Crammer, and Fernando Pereira. Confidence-weighted linear classification. In ICML, 2008.
-  John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Manuscript, 2010.
-  Elad Hazan and Satyen Kale. Extracting certainty from uncertainty: Regret bounded by variation in costs. In COLT, 2008.
-  Elad Hazan and C. Seshadhri. Efficient learning algorithms for changing environments. In ICML, 2009.
-  Chun-Nan Hsu, Han-Shen Huang, Yu-Ming Chang, and Yuh-Jye Lee. Periodic step-size adaptation in second-order gradient descent for single-pass on-line structured learning. Maching Learning, 77(2-3), 2009.
-  Robert A. Jacobs. Increased rates of convergence through learning rate adaptation. Neural Networks, 1, 1988.
-  Justin Ma, Lawrence K. Saul, Stefan Savage, and Geoffrey M. Voelker. Identifying suspicious URLs: an application of large-scale online learning. In ICML, 2009.
-  H. Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. Manuscript, 2010.
Richard S. Sutton.
Two problems with backpropagation and other steepest-descent learning procedures for networks.In Proc. Eighth Annual Conference of the Cognitive Science Society, 1986.
-  Tong Zhang. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In ICML, 2004.
-  Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In ICML, 2003.