Achieving Statistical Optimality of Federated Learning: Beyond Stationary Points

06/29/2021
by   Lili Su, et al.
0

Federated Learning (FL) is a promising framework that has great potentials in privacy preservation and in lowering the computation load at the cloud. FedAvg and FedProx are two widely adopted algorithms. However, recent work raised concerns on these two methods: (1) their fixed points do not correspond to the stationary points of the original optimization problem, and (2) the common model found might not generalize well locally. In this paper, we alleviate these concerns. Towards this, we adopt the statistical learning perspective yet allow the distributions to be heterogeneous and the local data to be unbalanced. We show, in the general kernel regression setting, that both FedAvg and FedProx converge to the minimax-optimal error rates. Moreover, when the kernel function has a finite rank, the convergence is exponentially fast. Our results further analytically quantify the impact of the model heterogeneity and characterize the federation gain - the reduction of the estimation error for a worker to join the federated learning compared to the best local estimator. To the best of our knowledge, we are the first to show the achievability of minimax error rates under FedAvg and FedProx, and the first to characterize the gains in joining FL. Numerical experiments further corroborate our theoretical findings on the statistical optimality of FedAvg and FedProx and the federation gains.

READ FULL TEXT

Authors

page 1

page 2

page 3

page 4

05/22/2020

FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data

Federated Learning (FL) has become a popular paradigm for learning from ...
06/21/2021

Compositional Federated Learning: Applications in Distributionally Robust Averaging and Meta Learning

In the paper, we propose an effective and efficient Compositional Federa...
05/11/2020

FedSplit: An algorithmic framework for fast federated optimization

Motivated by federated learning, we consider the hub-and-spoke model of ...
06/12/2022

Federated Learning on Riemannian Manifolds

Federated learning (FL) has found many important applications in smart-p...
01/27/2022

On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning

One of the biggest challenges in Federated Learning (FL) is that client ...
06/17/2021

Optimality and Stability in Federated Learning: A Game-theoretic Approach

Federated learning is a distributed learning paradigm where multiple age...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.