Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learning

12/11/2021
by   Shraman Pal, et al.
0

In recent years, there have been great advances in the field of decentralized learning with private data. Federated learning (FL) and split learning (SL) are two spearheads possessing their pros and cons, and are suited for many user clients and large models, respectively. To enjoy both benefits, hybrid approaches such as SplitFed have emerged of late, yet their fundamentals have still been illusive. In this work, we first identify the fundamental bottlenecks of SL, and thereby propose a scalable SL framework, coined SGLR. The server under SGLR broadcasts a common gradient averaged at the split-layer, emulating FL without any additional communication across clients as opposed to SplitFed. Meanwhile, SGLR splits the learning rate into its server-side and client-side rates, and separately adjusts them to support many clients in parallel. Simulation results corroborate that SGLR achieves higher accuracy than other baseline SL methods including SplitFed, which is even on par with FL consuming higher energy and communication costs. As a secondary result, we observe greater reduction in leakage of sensitive information via mutual information using SLGR over the baselines.

READ FULL TEXT

page 2

page 8

research
06/22/2023

Communication-Efficient Federated Learning through Importance Sampling

The high communication cost of sending model updates from the clients to...
research
03/27/2023

Adaptive Federated Learning via New Entropy Approach

Federated Learning (FL) has recently emerged as a popular framework, whi...
research
03/28/2023

Fast Convergent Federated Learning with Aggregated Gradients

Federated Learning (FL) is a novel machine learning framework, which ena...
research
02/13/2023

FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averaging

Federated learning (FL) is an emerging learning paradigm to tackle massi...
research
04/28/2023

Quality-Adaptive Split-Federated Learning for Segmenting Medical Images with Inaccurate Annotations

SplitFed Learning, a combination of Federated and Split Learning (FL and...
research
07/01/2022

Visual Transformer Meets CutMix for Improved Accuracy, Communication Efficiency, and Data Privacy in Split Learning

This article seeks for a distributed learning solution for the visual tr...
research
05/29/2023

Reducing Communication for Split Learning by Randomized Top-k Sparsification

Split learning is a simple solution for Vertical Federated Learning (VFL...

Please sign up or login with your details

Forgot password? Click here to reset