DeepAI AI Chat
Log In Sign Up

Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuity

by   Benjamin Grimmer, et al.
cornell university

We generalize the classic convergence rate theory for subgradient methods to apply to non-Lipschitz functions via a new measure of steepness. For the deterministic projected subgradient method, we derive a global O(1/√(T)) convergence rate for any function with at most exponential growth. Our approach implies generalizations of the standard convergence rates for gradient descent on functions with Lipschitz or Hölder continuous gradients. Further, we show a O(1/√(T)) convergence rate for the stochastic projected subgradient method on functions with at most quadratic growth, which improves to O(1/T) under strong convexity.


page 1

page 2

page 3

page 4


Convergence rates for momentum stochastic gradient descent with noise of machine learning type

We consider the momentum stochastic gradient descent scheme (MSGD) and i...

Faster Subgradient Methods for Functions with Hölderian Growth

The purpose of this manuscript is to derive new convergence results for ...

A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method

In this note, we present a new averaging technique for the projected sto...

How To Make the Gradients Small Stochastically

In convex stochastic optimization, convergence rates in terms of minimiz...

Fast Rates of ERM and Stochastic Approximation: Adaptive to Error Bound Conditions

Error bound conditions (EBC) are properties that characterize the growth...

Dual optimization for convex constrained objectives without the gradient-Lipschitz assumption

The minimization of convex objectives coming from linear supervised lear...