A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!

by   Dmitry Kovalev, et al.

Decentralized optimization methods enable on-device training of machine learning models without a central coordinator. In many scenarios communication between devices is energy demanding and time consuming and forms the bottleneck of the entire system. We propose a new randomized first-order method which tackles the communication bottleneck by applying randomized compression operators to the communicated messages. By combining our scheme with a new variance reduction technique that progressively throughout the iterations reduces the adverse effect of the injected quantization noise, we obtain the first scheme that converges linearly on strongly convex decentralized problems while using compressed communication only. We prove that our method can solve the problems without any increase in the number of communications compared to the baseline which does not perform any communication compression while still allowing for a significant compression factor which depends on the conditioning of the problem and the topology of the network. Our key theoretical findings are supported by numerical experiments.



There are no comments yet.


page 1

page 2

page 3

page 4


Decentralized Composite Optimization with Compression

Decentralized optimization and communication compression have exhibited ...

Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication

We consider decentralized stochastic optimization with the objective fun...

Scalable Average Consensus with Compressed Communications

We propose a new decentralized average consensus algorithm with compress...

Natural Compression for Distributed Deep Learning

Due to their hunger for big data, modern deep learning models are traine...

SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization

In this paper, we propose and analyze SPARQ-SGD, which is an event-trigg...

PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning

Lossy gradient compression has become a practical tool to overcome the c...

Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor

In order to mitigate the high communication cost in distributed and fede...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.