Coded Gradient Aggregation: A Tradeoff Between Communication Costs at Edge Nodes and at Helper Nodes

by   Birenjith Sasidharan, et al.

The increasing amount of data generated at the edge/client nodes and the privacy concerns have resulted in learning at the edge, in which the computations are performed at edge devices and are communicated to a central node for updating the model. The edge nodes have low bandwidth and may be available only intermittently. There are helper nodes present in the network that aid the edge nodes in the communication to the server. The edge nodes communicate the local gradient to helper nodes which relay these messages to the central node after possible aggregation. Recently, schemes using repetition codes and maximum-distance-separable (MDS) codes, respectively known as Aligned MDS Coding (AMC) scheme and Aligend Repetition Coding (ARC) scheme, were proposed. It was observed that in AMC scheme the communication between edge nodes and helper nodes is optimal but with an increased cost of communication between helper and master. An upper bound on the communication cost between helpers and master was obtained. In this paper, a tradeoff between communication costs at edge nodes and helper nodes is established with the help of pyramid codes, a well-known class of locally repairable codes. The communication costs at both the helper nodes and edge nodes are exactly characterized. Using the developed technique, the exact communication cost at helper nodes can be computed for the scheme using MDS codes. In the end, we provide two improved aggregation strategies for the existing AMC and ARC schemes, yielding significant reduction in communication cost at helpers, without changing any of the code parameters.



There are no comments yet.


page 1

page 2

page 3

page 4


Coded FFT and Its Communication Overhead

We propose a coded computing strategy and examine communication costs of...

Timely Distributed Computation with Stragglers

We consider a status update system in which the update packets need to b...

Deep Partial Updating

Emerging edge intelligence applications require the server to continuous...

CodedReduce: A Fast and Robust Framework for Gradient Aggregation in Distributed Learning

We focus on the commonly used synchronous Gradient Descent paradigm for ...

Gradient Coding from Cyclic MDS Codes and Expander Graphs

Gradient Descent, and its variants, are a popular method for solving emp...

Optimal Communication-Computation Trade-Off in Heterogeneous Gradient Coding

Gradient coding allows a master node to derive the aggregate of the part...

Improved Latency-Communication Trade-Off for Map-Shuffle-Reduce Systems with Stragglers

In a distributed computing system operating according to the map-shuffle...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.