Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms

by   Maruan Al-Shedivat, et al.

Federated learning is typically approached as an optimization problem, where the goal is to minimize a global loss function by distributing computation across client devices that possess local data and specify different parts of the global objective. We present an alternative perspective and formulate federated learning as a posterior inference problem, where the goal is to infer a global posterior distribution by having client devices each infer the posterior of their local data. While exact inference is often intractable, this perspective provides a principled way to search for global optima in federated settings. Further, starting with the analysis of federated quadratic objectives, we develop a computation- and communication-efficient approximate posterior inference algorithm – federated posterior averaging (FedPA). Our algorithm uses MCMC for approximate inference of local posteriors on the clients and efficiently communicates their statistics to the server, where the latter uses them to refine a global estimate of the posterior mode. Finally, we show that FedPA generalizes federated averaging (FedAvg), can similarly benefit from adaptive optimizers, and yields state-of-the-art results on four realistic and challenging benchmarks, converging faster, to better optima.



page 1

page 2

page 3

page 4


Server Averaging for Federated Learning

Federated learning allows distributed devices to collectively train a mo...

Bayesian Federated Learning via Predictive Distribution Distillation

For most existing federated learning algorithms, each round consists of ...

Local Averaging Helps: Hierarchical Federated Learning and Convergence Analysis

Federated learning is an effective approach to realize collaborative lea...

Fine-tuning is Fine in Federated Learning

We study the performance of federated learning algorithms and their vari...

Robust One Round Federated Learning with Predictive Space Bayesian Inference

Making predictions robust is an important challenge. A separate challeng...

Federated Adaptation of Reservoirs via Intrinsic Plasticity

We propose a novel algorithm for performing federated learning with Echo...

Federated Gaussian Process: Convergence, Automatic Personalization and Multi-fidelity Modeling

In this paper, we propose : a Federated Gaussian process (𝒢𝒫) regression...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.