
CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression
Due to the high communication cost in distributed and federated learning...
read it

Federated Accelerated Stochastic Gradient Descent
We propose Federated Accelerated Stochastic Gradient Descent (FedAc), a ...
read it

Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization
We consider the setting of distributed empirical risk minimization where...
read it

Federated Learning with Compression: Unified Analysis and Sharp Guarantees
In federated learning, communication cost is often a critical bottleneck...
read it

From Averaging to Acceleration, There is Only a Stepsize
We show that accelerated gradient descent, averaged gradient descent and...
read it

Compressed Distributed Gradient Descent: CommunicationEfficient Consensus over Networks
Network consensus optimization has received increasing attention in rece...
read it

Accelerated Gradient Descent Learning over Multiple Access Fading Channels
We consider a distributed learning problem in a wireless network, consis...
read it
Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
Due to the high communication cost in distributed and federated learning problems, methods relying on compression of communicated messages are becoming increasingly popular. While in other contexts the best performing gradienttype methods invariably rely on some form of acceleration/momentum to reduce the number of iterations, there are no methods which combine the benefits of both gradient compression and acceleration. In this paper, we remedy this situation and propose the first accelerated compressed gradient descent (ACGD) methods. In the single machine regime, we prove that ACGD enjoys the rate O((1+ω)√(L/μ)log1/ϵ) for μstrongly convex problems and O((1+ω)√(L/ϵ)) for convex problems, respectively, where L is the smoothness constant and ω is the compression parameter. Our results improve upon the existing nonaccelerated rates O((1+ω)L/μlog1/ϵ) and O((1+ω)L/ϵ), respectively, and recover the optimal rates of accelerated gradient descent as a special case when no compression (ω=0) is applied. We further propose a distributed variant of ACGD (called ADIANA) and prove the convergence rate O(ω+√(L/μ) +√((ω/n+√(ω/n))ω L/μ)), where n is the number of devices/workers and O hides the logarithmic factor log1/ϵ. This improves upon the previous best result O(ω + L/μ+ω L/nμ) achieved by the DIANA method of Mishchenko et al (2019). Finally, we conduct several experiments on realworld datasets which corroborate our theoretical results and confirm the practical superiority of our methods.
READ FULL TEXT
Comments
There are no comments yet.