Revisiting EXTRA for Smooth Distributed Optimization

by   Huan Li, et al.

EXTRA is a popular method for the dencentralized distributed optimization and has broad applications. This paper revisits the EXTRA. Firstly, we give a sharp complexity analysis for EXTRA with the improved O((L/μ+1/1-σ_2(W))log1/ϵ(1-σ_2(W))) communication and computation complexities for μ-strongly convex and L-smooth problems, where σ_2(W) is the second largest singular value of the weight matrix W. When the strong convexity is absent, we prove the O((L/ϵ+1/1-σ_2(W))log1/1-σ_2(W)) complexities. Then, we use the Catalyst framework to accelerate EXTRA and obtain the O(√(L/μ(1-σ_2(W)))log L/μ(1-σ_2(W))log1/ϵ) communication and computation complexities for strongly convex and smooth problems and the O(√(L/ϵ(1-σ_2(W)))log1/ϵ(1-σ_2(W))) complexities for non-strongly convex ones. Our communication complexities of the accelerated EXTRA are only worse by the factors of (logL/μ(1-σ_2(W))) and (log1/ϵ(1-σ_2(W))) from the lower complexity bounds for strongly convex and non-strongly convex problems, respectively.


page 1

page 2

page 3

page 4


Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization

We study stochastic decentralized optimization for the problem of traini...

Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization

Decentralized optimization over time-varying graphs has been increasingl...

Lower Bounds and Accelerated Algorithms for Bilevel Optimization

Bilevel optimization has recently attracted growing interests due to its...

Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems

While many distributed optimization algorithms have been proposed for so...

Optimal Algorithms for Stochastic Multi-Level Compositional Optimization

In this paper, we investigate the problem of stochastic multi-level comp...

Smooth Strongly Convex Regression

Convex regression (CR) is the problem of fitting a convex function to a ...

Revisiting Projection-free Online Learning: the Strongly Convex Case

Projection-free optimization algorithms, which are mostly based on the c...