Coded Gradient Aggregation: A Tradeoff Between Communication Costs at Edge Nodes and at Helper Nodes

05/06/2021
by   Birenjith Sasidharan, et al.
0

The increasing amount of data generated at the edge/client nodes and the privacy concerns have resulted in learning at the edge, in which the computations are performed at edge devices and are communicated to a central node for updating the model. The edge nodes have low bandwidth and may be available only intermittently. There are helper nodes present in the network that aid the edge nodes in the communication to the server. The edge nodes communicate the local gradient to helper nodes which relay these messages to the central node after possible aggregation. Recently, schemes using repetition codes and maximum-distance-separable (MDS) codes, respectively known as Aligned MDS Coding (AMC) scheme and Aligend Repetition Coding (ARC) scheme, were proposed. It was observed that in AMC scheme the communication between edge nodes and helper nodes is optimal but with an increased cost of communication between helper and master. An upper bound on the communication cost between helpers and master was obtained. In this paper, a tradeoff between communication costs at edge nodes and helper nodes is established with the help of pyramid codes, a well-known class of locally repairable codes. The communication costs at both the helper nodes and edge nodes are exactly characterized. Using the developed technique, the exact communication cost at helper nodes can be computed for the scheme using MDS codes. In the end, we provide two improved aggregation strategies for the existing AMC and ARC schemes, yielding significant reduction in communication cost at helpers, without changing any of the code parameters.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

05/24/2018

Coded FFT and Its Communication Overhead

We propose a coded computing strategy and examine communication costs of...
10/08/2019

Timely Distributed Computation with Stragglers

We consider a status update system in which the update packets need to b...
07/06/2020

Deep Partial Updating

Emerging edge intelligence applications require the server to continuous...
02/06/2019

CodedReduce: A Fast and Robust Framework for Gradient Aggregation in Distributed Learning

We focus on the commonly used synchronous Gradient Descent paradigm for ...
07/12/2017

Gradient Coding from Cyclic MDS Codes and Expander Graphs

Gradient Descent, and its variants, are a popular method for solving emp...
03/02/2021

Optimal Communication-Computation Trade-Off in Heterogeneous Gradient Coding

Gradient coding allows a master node to derive the aggregate of the part...
08/20/2018

Improved Latency-Communication Trade-Off for Map-Shuffle-Reduce Systems with Stragglers

In a distributed computing system operating according to the map-shuffle...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.