Fundamental Resource Trade-offs for Encoded Distributed Optimization

03/31/2018
by   A. Salman Avestimehr, et al.
0

Dealing with the shear size and complexity of today's massive data sets requires computational platforms that can analyze data in a parallelized and distributed fashion. A major bottleneck that arises in such modern distributed computing environments is that some of the worker nodes may run slow. These nodes a.k.a. stragglers can significantly slow down computation as the slowest node may dictate the overall computational time. A recent computational framework, called encoded optimization, creates redundancy in the data to mitigate the effect of stragglers. In this paper we develop novel mathematical understanding for this framework demonstrating its effectiveness in much broader settings than was previously understood. We also analyze the convergence behavior of iterative encoded optimization algorithms, allowing us to characterize fundamental trade-offs between convergence rate, size of data set, accuracy, computational load (or data redundancy), and straggler toleration in this framework.

READ FULL TEXT
10/27/2017

Near-Optimal Straggler Mitigation for Distributed Gradient Methods

Modern learning algorithms use gradient descent updates to train inferen...
11/14/2017

Straggler Mitigation in Distributed Optimization Through Data Encoding

Slow running or straggler tasks can significantly reduce computation spe...
03/14/2018

Redundancy Techniques for Straggler Mitigation in Distributed Optimization and Learning

Performance of distributed optimization and learning systems is bottlene...
10/08/2018

Time-Message Trade-Offs in Distributed Algorithms

This paper focuses on showing time-message trade-offs in distributed alg...
06/10/2020

Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimization

Distributed optimization is vital in solving large-scale machine learnin...
08/22/2016

Computational and Statistical Tradeoffs in Learning to Rank

For massive and heterogeneous modern datasets, it is of fundamental inte...
09/18/2019

Diversity-enabled sweet spots in layered architectures and speed-accuracy trade-offs in sensorimotor control

Nervous systems sense, communicate, compute, and actuate movement using ...