Bottleneck Time Minimization for Distributed Iterative Processes: Speeding Up Gossip-Based Federated Learning on Networked Computers

06/29/2021
by   Mehrdad Kiamari, et al.
0

We present a novel task scheduling scheme for accelerating computational applications involving distributed iterative processes that are executed on networked computing resources. Such an application consists of multiple tasks, each of which outputs data at each iteration to be processed by neighboring tasks; these dependencies between the tasks can be represented as a directed graph. We first mathematically formulate the problem as a Binary Quadratic Program (BQP), accounting for both computation and communication costs. We show that the problem is NP-hard. We then relax the problem as a Semi-Definite Program (SDP) and utilize a randomized rounding technique based on sampling from a suitably-formulated multi-variate Gaussian distribution. Furthermore, we derive the expected value of bottleneck time. Finally, we apply our proposed scheme on gossip-based federated learning as an application of iterative processes. Through numerical evaluations on the MNIST and CIFAR-10 datasets, we show that our proposed approach outperforms well-known scheduling techniques from distributed computing. In particular, for arbitrary settings, we show that it reduces bottleneck time by 91% compared to HEFT and 84% compared to throughput HEFT.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/29/2021

From Distributed Machine Learning to Federated Learning: A Survey

In recent years, data and computing resources are typically distributed ...
research
06/14/2022

Matching Pursuit Based Scheduling for Over-the-Air Federated Learning

This paper develops a class of low-complexity device scheduling algorith...
research
12/10/2020

Communication-Computation Efficient Secure Aggregation for Federated Learning

Federated learning has been spotlighted as a way to train neural network...
research
10/07/2022

Time Minimization in Hierarchical Federated Learning

Federated Learning is a modern decentralized machine learning technique ...
research
03/12/2021

Auction Based Clustered Federated Learning in Mobile Edge Computing System

In recent years, mobile clients' computing ability and storage capacity ...
research
02/28/2022

Computational Code-Based Privacy in Coded Federated Learning

We propose a privacy-preserving federated learning (FL) scheme that is r...

Please sign up or login with your details

Forgot password? Click here to reset