Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Server

by   Arda Aytekin, et al.

This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss is strongly convex, we establish linear convergence rate, give explicit expressions for step-size choices that guarantee convergence to the optimum, and bound the associated convergence factors. The expressions have an explicit dependence on the degree of asynchrony and recover classical results under synchronous operation. Simulations and implementations on commercial compute clouds validate our findings.



page 1

page 2

page 3

page 4


An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimization

Mini-batch optimization has proven to be a powerful paradigm for large-s...

Asynchronous Federated Optimization

Federated learning enables training on a massive number of edge devices....

General Proximal Incremental Aggregated Gradient Algorithms: Better and Novel Results under General Scheme

The incremental aggregated gradient algorithm is popular in network opti...

On the Convergence of SARAH and Beyond

The main theme of this work is a unifying algorithm, abbreviated as L2S,...

Additive Schwarz Methods for Convex Optimization with Backtracking

This paper presents a novel backtracking strategy for additive Schwarz m...

Stochastic optimization and sparse statistical recovery: An optimal algorithm for high dimensions

We develop and analyze stochastic optimization algorithms for problems i...

(Near) Optimal Parallelism Bound for Fully Asynchronous Coordinate Descent with Linear Speedup

When solving massive optimization problems in areas such as machine lear...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.