Aergia: Leveraging Heterogeneity in Federated Learning Systems

by   Bart Cox, et al.

Federated Learning (FL) is a popular approach for distributed deep learning that prevents the pooling of large amounts of data in a central server. FL relies on clients to update a global model using their local datasets. Classical FL algorithms use a central federator that, for each training round, waits for all clients to send their model updates before aggregating them. In practical deployments, clients might have different computing powers and network capabilities, which might lead slow clients to become performance bottlenecks. Previous works have suggested to use a deadline for each learning round so that the federator ignores the late updates of slow clients, or so that clients send partially trained models before the deadline. To speed up the training process, we instead propose Aergia, a novel approach where slow clients (i) freeze the part of their model that is the most computationally intensive to train; (ii) train the unfrozen part of their model; and (iii) offload the training of the frozen part of their model to a faster client that trains it using its own dataset. The offloading decisions are orchestrated by the federator based on the training speed that clients report and on the similarities between their datasets, which are privately evaluated thanks to a trusted execution environment. We show through extensive experiments that Aergia maintains high accuracy and significantly reduces the training time under heterogeneous settings by up to 27 respectively.


page 9

page 11


Latency Aware Semi-synchronous Client Selection and Model Aggregation for Wireless Federated Learning

Federated learning (FL) is a collaborative machine learning framework th...

FedPrune: Towards Inclusive Federated Learning

Federated learning (FL) is a distributed learning technique that trains ...

FedSS: Federated Learning with Smart Selection of clients

Federated learning provides the ability to learn over heterogeneous user...

Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data

We propose FEDENHANCE, an unsupervised federated learning (FL) approach ...

A Fair Federated Learning Framework With Reinforcement Learning

Federated learning (FL) is a paradigm where many clients collaboratively...

Online Meta-Learning for Model Update Aggregation in Federated Learning for Click-Through Rate Prediction

In Federated Learning (FL) of click-through rate (CTR) prediction, users...

Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms

This paper studies a new task of federated learning (FL) for semantic pa...

Please sign up or login with your details

Forgot password? Click here to reset