Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning

06/06/2022
by   Milad Sefidgaran, et al.
0

In this paper, we use tools from rate-distortion theory to establish new upper bounds on the generalization error of statistical distributed learning algorithms. Specifically, there are K clients whose individually chosen models are aggregated by a central server. The bounds depend on the compressibility of each client's algorithm while keeping other clients' algorithms un-compressed, and leverage the fact that small changes in each local model change the aggregated model by a factor of only 1/K. Adopting a recently proposed approach by Sefidgaran et al., and extending it suitably to the distributed setting, this enables smaller rate-distortion terms which are shown to translate into tighter generalization bounds. The bounds are then applied to the distributed support vector machines (SVM), suggesting that the generalization error of the distributed setting decays faster than that of the centralized one with a factor of 𝒪(log(K)/√(K)). This finding is validated also experimentally. A similar conclusion is obtained for a multiple-round federated learning setup where each client uses stochastic gradient Langevin dynamics (SGLD).

READ FULL TEXT

page 1

page 2

page 3

page 4

02/04/2022

Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning

We consider information-theoretic bounds on expected generalization erro...
02/10/2021

Learning under Distribution Mismatch and Model Misspecification

We study learning algorithms when there is a mismatch between the distri...
04/21/2021

Gradient Masked Federated Optimization

Federated Averaging (FedAVG) has become the most popular federated learn...
05/05/2020

Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learning

Machine learning algorithms operating on mobile networks can be characte...
08/16/2021

Client Selection Approach in Support of Clustered Federated Learning over Wireless Edge Networks

Clustered Federated Multitask Learning (CFL) was introduced as an effici...
06/30/2022

Cross-domain Federated Object Detection

Detection models trained by one party (server) may face severe performan...