On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning

01/27/2022
by   Hanhan Zhou, et al.
0

One of the biggest challenges in Federated Learning (FL) is that client devices often have drastically different computation and communication resources for local updates. To this end, recent research efforts have focused on training heterogeneous local models obtained by pruning a shared global model. Despite empirical success, theoretical guarantees on convergence remain an open question. In this paper, we present a unifying framework for heterogeneous FL algorithms with arbitrary adaptive online model pruning and provide a general convergence analysis. In particular, we prove that under certain sufficient conditions and on both IID and non-IID data, these algorithms converges to a stationary point of standard FL for general smooth cost functions, with a convergence rate of O(1/√(Q)). Moreover, we illuminate two key factors impacting convergence: pruning-induced noise and minimum coverage index, advocating a joint design of local pruning masks for efficient training.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/11/2023

FedLP: Layer-wise Pruning Mechanism for Communication-Computation Efficient Federated Learning

Federated learning (FL) has prevailed as an efficient and privacy-preser...
research
05/27/2022

Towards Communication-Learning Trade-off for Federated Learning at the Network Edge

In this letter, we study a wireless federated learning (FL) system where...
research
12/25/2021

Towards Federated Learning on Time-Evolving Heterogeneous Data

Federated Learning (FL) is an emerging learning paradigm that preserves ...
research
04/25/2022

FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Server

Despite achieving remarkable performance, Federated Learning (FL) suffer...
research
06/10/2022

On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond

The FedProx algorithm is a simple yet powerful distributed proximal poin...
research
06/29/2022

AFAFed – Protocol analysis

In this paper, we design, analyze the convergence properties and address...
research
12/05/2022

FedTiny: Pruned Federated Learning Towards Specialized Tiny Models

Neural network pruning has been a well-established compression technique...

Please sign up or login with your details

Forgot password? Click here to reset