Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling

12/21/2021
by   Bing Luo, et al.
12

Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server's communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-clock convergence time. We obtain a new tractable convergence bound for FL algorithms with arbitrary client sampling probabilities. Based on the bound, we analytically establish the relationship between the total learning time and sampling probabilities, which results in a non-convex optimization problem for training time minimization. We design an efficient algorithm for learning the unknown parameters in the convergence bound and develop a low-complexity algorithm to approximately solve the non-convex problem. Experimental results from both hardware prototype and simulation demonstrate that our proposed sampling scheme significantly reduces the convergence time compared to several baseline sampling schemes. Notably, our scheme in hardware prototype spends 73 less time than the uniform sampling baseline for reaching the same target loss.

READ FULL TEXT

page 1

page 7

research
05/27/2022

Client Selection in Nonconvex Federated Learning: Improved Convergence Analysis for Optimal Unbiased Sampling Strategy

Federated learning (FL) is a distributed machine learning paradigm that ...
research
04/17/2023

Incentive Mechanism Design for Unbiased Federated Learning with Randomized Client Participation

Incentive mechanism is crucial for federated learning (FL) when rational...
research
07/26/2021

On The Impact of Client Sampling on Federated Learning Convergence

While clients' sampling is a central operation of current state-of-the-a...
research
01/19/2022

Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization

Federated learning (FL) is a useful tool in distributed machine learning...
research
08/16/2022

On the Convergence of Multi-Server Federated Learning with Overlapping Area

Multi-server Federated learning (FL) has been considered as a promising ...
research
12/28/2021

Adaptive Client Sampling in Federated Learning via Online Learning with Bandit Feedback

In federated learning (FL) problems, client sampling plays a key role in...
research
10/31/2022

Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms

This paper focuses on Bayesian inference in a federated learning context...

Please sign up or login with your details

Forgot password? Click here to reset