Revisiting EXTRA for Smooth Distributed Optimization

by   Huan Li, et al.

EXTRA is a popular method for the dencentralized distributed optimization and has broad applications. This paper revisits the EXTRA. Firstly, we give a sharp complexity analysis for EXTRA with the improved O((L/μ+1/1-σ_2(W))log1/ϵ(1-σ_2(W))) communication and computation complexities for μ-strongly convex and L-smooth problems, where σ_2(W) is the second largest singular value of the weight matrix W. When the strong convexity is absent, we prove the O((L/ϵ+1/1-σ_2(W))log1/1-σ_2(W)) complexities. Then, we use the Catalyst framework to accelerate EXTRA and obtain the O(√(L/μ(1-σ_2(W)))log L/μ(1-σ_2(W))log1/ϵ) communication and computation complexities for strongly convex and smooth problems and the O(√(L/ϵ(1-σ_2(W)))log1/ϵ(1-σ_2(W))) complexities for non-strongly convex ones. Our communication complexities of the accelerated EXTRA are only worse by the factors of (logL/μ(1-σ_2(W))) and (log1/ϵ(1-σ_2(W))) from the lower complexity bounds for strongly convex and non-strongly convex problems, respectively.


page 1

page 2

page 3

page 4


Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization

We study stochastic decentralized optimization for the problem of traini...

Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization

Decentralized optimization over time-varying graphs has been increasingl...

Lower Bounds and Accelerated Algorithms for Bilevel Optimization

Bilevel optimization has recently attracted growing interests due to its...

Optimal Algorithms for Distributed Optimization

In this paper, we study the optimal convergence rate for distributed con...

Optimal Algorithms for Stochastic Multi-Level Compositional Optimization

In this paper, we investigate the problem of stochastic multi-level comp...

Smooth Strongly Convex Regression

Convex regression (CR) is the problem of fitting a convex function to a ...

Space and complexities of territorial systems

The spatial character of territorial systems plays a crucial role in the...

Please sign up or login with your details

Forgot password? Click here to reset