Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization

04/06/2021
by   Huan Li, et al.
0

Decentralized optimization over time-varying graphs has been increasingly common in modern machine learning with massive data stored on millions of mobile devices, such as in federated learning. This paper revisits and extends the widely used accelerated gradient tracking. We prove the O(γ^2/(1-σ_γ)^2√(L/ϵ)) and O((γ/1-σ_γ)^1.5√(L/μ)log1/ϵ) complexities for the practical single loop accelerated gradient tracking over time-varying graphs when the problems are nonstrongly convex and strongly convex, respectively, where γ and σ_γ are two common constants charactering the network connectivity, ϵ is the desired precision, and L and μ are the smoothness and strong convexity constants, respectively. Our complexities improve significantly on the ones of O(1/ϵ^5/7) and O((L/μ)^5/71/(1-σ)^1.5log1/ϵ) proved in the original literature only for static graph. When combining with a multiple consensus subroutine, the dependence on the network connectivity constants can be further improved. When the network is time-invariant, our complexities exactly match the lower bounds without hiding any poly-logarithmic factor for both nonstrongly convex and strongly convex problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/24/2020

Revisiting EXTRA for Smooth Distributed Optimization

EXTRA is a popular method for the dencentralized distributed optimizatio...
02/18/2021

ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks

We propose ADOM - an accelerated method for smooth and strongly convex d...
02/07/2021

Lower Bounds and Accelerated Algorithms for Bilevel Optimization

Bilevel optimization has recently attracted growing interests due to its...
07/26/2022

DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization for Time-Varying Gossips

DADAO is a novel decentralized asynchronous stochastic algorithm to mini...
08/17/2018

Decentralized Dictionary Learning Over Time-Varying Digraphs

This paper studies Dictionary Learning problems wherein the learning tas...
04/02/2016

Centralized and Decentralized Global Outer-synchronization of Asymmetric Recurrent Time-varying Neural Network by Data-sampling

In this paper, we discuss the outer-synchronization of the asymmetricall...
11/19/2020

A Stable High-order Tuner for General Convex Functions

Iterative gradient-based algorithms have been increasingly applied for t...