Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization

by   Karan Chadha, et al.

We extend the Approximate-Proximal Point (aProx) family of model-based methods for solving stochastic convex optimization problems, including stochastic subgradient, proximal point, and bundle methods, to the minibatch and accelerated setting. To do so, we propose specific model-based algorithms and an acceleration scheme for which we provide non-asymptotic convergence guarantees, which are order-optimal in all problem-dependent constants and provide linear speedup in minibatch size, while maintaining the desirable robustness traits (e.g. to stepsize) of the aProx family. Additionally, we show improved convergence rates and matching lower bounds identifying new fundamental constants for "interpolation" problems, whose importance in statistical machine learning is growing; this, for example, gives a parallelization strategy for alternating projections. We corroborate our theoretical results with empirical testing to demonstrate the gains accurate modeling, acceleration, and minibatching provide.


page 1

page 2

page 3

page 4


Randomized Smoothing for Stochastic Optimization

We analyze convergence rates of stochastic optimization procedures for n...

Minibatch and Momentum Model-based Methods for Stochastic Non-smooth Non-convex Optimization

Stochastic model-based methods have received increasing attention lately...

Stochastic L-BFGS: Improved Convergence Rates and Practical Acceleration Strategies

We revisit the stochastic limited-memory BFGS (L-BFGS) algorithm. By pro...

Locally Accelerated Conditional Gradients

Conditional gradient methods form a class of projection-free first-order...

Stochastic Reformulations of Linear Systems: Algorithms and Convergence Theory

We develop a family of reformulations of an arbitrary consistent linear ...

Parameter-free Locally Accelerated Conditional Gradients

Projection-free conditional gradient (CG) methods are the algorithms of ...

Acceleration Methods

This monograph covers some recent advances on a range of acceleration te...