Compressed Coded Distributed Computing

05/05/2018
by   Songze Li, et al.
0

Communication overhead is one of the major performance bottlenecks in large-scale distributed computing systems, in particular for machine learning applications. Conventionally, compression techniques are used to reduce the load of communication by combining intermediate results of the same computation task as much as possible. Recently, via the development of coded distributed computing (CDC), it has been shown that it is possible to enable coding opportunities across intermediate results of different computation tasks to further reduce the communication load. We propose a new scheme, named compressed coded distributed computing (in short, compressed CDC), which jointly exploits the above two techniques (i.e., combining the intermediate results of the same computation and coding across the intermediate results of different computations) to significantly reduce the communication load for computations with linear aggregation (reduction) of intermediate results in the final stage that are prevalent in machine learning (e.g., distributed training algorithms where partial gradients are computed distributedly and then averaged in the final stage). In particular, compressed CDC first compresses/combines several intermediate results for a single computation, and then utilizes multiple such combined packets to create a coded multicast packet that is simultaneously useful for multiple computations. We characterize the achievable communication load of compressed CDC and show that it substantially outperforms both combining methods and CDC scheme.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/26/2018

A Storage-Computation-Communication Tradeoff for Distributed Computing

This paper investigates distributed computing systems where computations...
research
05/13/2020

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

In large scale distributed computing systems, communication overhead is ...
research
08/20/2020

A Survey of Coded Distributed Computing

Distributed computing has become a common approach for large-scale compu...
research
01/17/2018

Coded Computing for Distributed Graph Analytics

Many distributed graph computing systems have been developed recently fo...
research
01/22/2019

CAMR: Coded Aggregated MapReduce

Many big data algorithms executed on MapReduce-like systems have a shuff...
research
08/02/2022

Distributed Computations with Layered Resolution

Modern computationally-heavy applications are often time-sensitive, dema...
research
01/10/2023

A Fundamental Tradeoff Among Storage, Computation, and Communication for Distributed Computing over Star Network

Coded distributed computing can alleviate the communication load by leve...

Please sign up or login with your details

Forgot password? Click here to reset