Achieving Personalized Federated Learning with Sparse Local Models

01/27/2022
by   Tiansheng Huang, et al.
0

Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge. Instead of training an intact (or dense) PFL model, FedSpa only maintains a fixed number of active parameters throughout training (aka sparse-to-sparse training), which enables users' models to achieve personalization with cheap communication, computation, and memory cost. We theoretically show that the iterates obtained by FedSpa converge to the local minimizer of the formulated SPFL problem at rate of 𝒪(1/√(T)). Comprehensive experiments demonstrate that FedSpa significantly saves communication and computation costs, while simultaneously achieves higher model accuracy and faster convergence speed against several state-of-the-art PFL methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/01/2022

DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training

Personalized federated learning is proposed to handle the data heterogen...
research
02/19/2021

Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques

We study the optimization aspects of personalized Federated Learning (FL...
research
04/26/2021

Communication-Efficient and Personalized Federated Lottery Ticket Learning

The lottery ticket hypothesis (LTH) claims that a deep neural network (i...
research
10/19/2021

User-Centric Federated Learning

Data heterogeneity across participating devices poses one of the main ch...
research
09/19/2023

Sparser Random Networks Exist: Enforcing Communication-Efficient Federated Learning via Regularization

This work presents a new method for enhancing communication efficiency i...
research
02/21/2023

Fusion of Global and Local Knowledge for Personalized Federated Learning

Personalized federated learning, as a variant of federated learning, tra...
research
05/05/2023

Now It Sounds Like You: Learning Personalized Vocabulary On Device

In recent years, Federated Learning (FL) has shown significant advanceme...

Please sign up or login with your details

Forgot password? Click here to reset