Achieving Statistical Optimality of Federated Learning: Beyond Stationary Points

06/29/2021
by   Lili Su, et al.
0

Federated Learning (FL) is a promising framework that has great potentials in privacy preservation and in lowering the computation load at the cloud. FedAvg and FedProx are two widely adopted algorithms. However, recent work raised concerns on these two methods: (1) their fixed points do not correspond to the stationary points of the original optimization problem, and (2) the common model found might not generalize well locally. In this paper, we alleviate these concerns. Towards this, we adopt the statistical learning perspective yet allow the distributions to be heterogeneous and the local data to be unbalanced. We show, in the general kernel regression setting, that both FedAvg and FedProx converge to the minimax-optimal error rates. Moreover, when the kernel function has a finite rank, the convergence is exponentially fast. Our results further analytically quantify the impact of the model heterogeneity and characterize the federation gain - the reduction of the estimation error for a worker to join the federated learning compared to the best local estimator. To the best of our knowledge, we are the first to show the achievability of minimax error rates under FedAvg and FedProx, and the first to characterize the gains in joining FL. Numerical experiments further corroborate our theoretical findings on the statistical optimality of FedAvg and FedProx and the federation gains.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/22/2020

FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data

Federated Learning (FL) has become a popular paradigm for learning from ...
research
06/21/2021

Compositional Federated Learning: Applications in Distributionally Robust Averaging and Meta Learning

In the paper, we propose an effective and efficient Compositional Federa...
research
05/11/2020

FedSplit: An algorithmic framework for fast federated optimization

Motivated by federated learning, we consider the hub-and-spoke model of ...
research
06/23/2021

Behavior Mimics Distribution: Combining Individual and Group Behaviors for Federated Learning

Federated Learning (FL) has become an active and promising distributed m...
research
06/12/2022

Federated Learning on Riemannian Manifolds

Federated learning (FL) has found many important applications in smart-p...
research
09/22/2022

One-Shot Federated Learning for Model Clustering and Learning in Heterogeneous Environments

We propose a communication efficient approach for federated learning in ...
research
06/17/2021

Optimality and Stability in Federated Learning: A Game-theoretic Approach

Federated learning is a distributed learning paradigm where multiple age...

Please sign up or login with your details

Forgot password? Click here to reset