Throughput-Optimal Topology Design for Cross-Silo Federated Learning

by   Othmane Marfoq, et al.

Federated learning usually employs a client-server architecture where an orchestrator iteratively aggregates model updates from remote clients and pushes them back a refined model. This approach may be inefficient in cross-silo settings, as close-by data silos with high-speed access links may exchange information faster than with the orchestrator, and the orchestrator may become a communication bottleneck. In this paper we define the problem of topology design for cross-silo federated learning using the theory of max-plus linear systems to compute the system throughput—number of communication rounds per time unit. We also propose practical algorithms that, under the knowledge of measurable network characteristics, find a topology with the largest throughput or with provable throughput guarantees. In realistic Internet networks with 10 Gbps access links for silos, our algorithms speed up training by a factor 9 and 1.5 in comparison to the master-slave architecture and to state-of-the-art MATCHA, respectively. Speedups are even larger with slower access links.


page 30

page 31

page 40


Multigraph Topology Design for Cross-Silo Federated Learning

Cross-silo federated learning utilizes a few hundred reliable data silos...

Privatized Graph Federated Learning

Federated learning is a semi-distributed algorithm, where a server commu...

Optimal Client Sampling for Federated Learning

It is well understood that client-master communication can be a primary ...

Federated learning with multichannel ALOHA

In this paper, we study federated learning in a cellular system with a b...

Comfetch: Federated Learning of Large Networks on Memory-Constrained Clients via Sketching

A popular application of federated learning is using many clients to tra...

Optimising Communication Overhead in Federated Learning Using NSGA-II

Federated learning is a training paradigm according to which a server-ba...

Optimization with access to auxiliary information

We investigate the fundamental optimization question of minimizing a tar...