Quantized Push-sum for Gossip and Decentralized Optimization over Directed Graphs

02/23/2020
by   Hossein Taheri, et al.
0

We consider a decentralized stochastic learning problem where data points are distributed among computing nodes communicating over a directed graph. As the model size gets large, decentralized learning faces a major bottleneck that is the heavy communication load due to each node transmitting large messages (model updates) to its neighbors. To tackle this bottleneck, we propose the quantized decentralized stochastic learning algorithm over directed graphs that is based on the push-sum algorithm in decentralized consensus optimization. More importantly, we prove that our algorithm achieves the same convergence rates of the decentralized stochastic learning algorithm with exact-communication for both convex and non-convex losses. A key technical challenge of the work is to prove exact convergence of the proposed decentralized learning algorithm in the presence of quantization noise with unbounded variance over directed graphs. We provide numerical evaluations that corroborate our main theoretical results and illustrate significant speed-up compared to the exact-communication methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/13/2020

Push-SAGA: A decentralized stochastic algorithm with variance reduction over directed graphs

In this paper, we propose Push-SAGA, a decentralized stochastic first-or...
research
06/29/2018

Quantized Decentralized Consensus Optimization

We consider the problem of decentralized consensus optimization, where t...
research
07/24/2019

Robust and Communication-Efficient Collaborative Learning

We consider a decentralized learning problem, where a set of computing n...
research
03/06/2021

Decentralized Langevin Dynamics over a Directed Graph

The prevalence of technologies in the space of the Internet of Things an...
research
08/04/2017

Efficient Variance-Reduced Learning for Fully Decentralized On-Device Intelligence

This work develops a fully decentralized variance-reduced learning algor...
research
08/25/2021

Decentralized optimization with non-identical sampling in presence of stragglers

We consider decentralized consensus optimization when workers sample dat...
research
10/05/2022

Personalized Decentralized Bilevel Optimization over Stochastic and Directed Networks

While personalization in distributed learning has been extensively studi...

Please sign up or login with your details

Forgot password? Click here to reset