Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation

02/27/2018
by   Cong Fang, et al.
0

Asynchronous algorithms have attracted much attention recently due to the crucial demands on solving large-scale optimization problems. However, the accelerated versions of asynchronous algorithms are rarely studied. In this paper, we propose the "momentum compensation" technique to accelerate asynchronous algorithms for convex problems. Specifically, we first accelerate the plain Asynchronous Gradient Descent, which achieves a faster O(1/√(ϵ)) (v.s. O(1/ϵ)) convergence rate for non-strongly convex functions, and O(√(κ)(1/ϵ)) (v.s. O(κ(1/ϵ))) for strongly convex functions to reach an ϵ- approximate minimizer with the condition number κ. We further apply the technique to accelerate modern stochastic asynchronous algorithms such as Asynchronous Stochastic Coordinate Descent and Asynchronous Stochastic Gradient Descent. Both of the resultant practical algorithms are faster than existing ones by order. To the best of our knowledge, we are the first to consider accelerated algorithms that allow updating by delayed gradients and are the first to propose truly accelerated asynchronous algorithms. Finally, the experimental results on a shared memory system show that acceleration can lead to significant performance gains on ill-conditioned problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/24/2018

Asynchronous decentralized accelerated stochastic gradient descent

In this work, we introduce an asynchronous decentralized accelerated sto...
research
09/30/2021

Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization

We show that stochastic acceleration can be achieved under the perturbed...
research
05/25/2018

A New Analysis of Variance Reduced Stochastic Proximal Methods for Composite Optimization with Serial and Asynchronous Realizations

We provide a comprehensive analysis of stochastic variance reduced gradi...
research
10/01/2015

An Asynchronous Implementation of the Limited Memory CMA-ES

We present our asynchronous implementation of the LM-CMA-ES algorithm, w...
research
02/05/2019

Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problems

Nonconvex and nonsmooth problems have recently attracted considerable at...
research
08/04/2015

Asynchronous stochastic convex optimization

We show that asymptotically, completely asynchronous stochastic gradient...
research
07/26/2019

Taming Momentum in a Distributed Asynchronous Environment

Although distributed computing can significantly reduce the training tim...

Please sign up or login with your details

Forgot password? Click here to reset