Information-Theoretic Perspective of Federated Learning

11/15/2019
by   Linara Adilova, et al.
0

An approach to distributed machine learning is to train models on local datasets and aggregate these models into a single, stronger model. A popular instance of this form of parallelization is federated learning, where the nodes periodically send their local models to a coordinator that aggregates them and redistributes the aggregation back to continue training with it. The most frequently used form of aggregation is averaging the model parameters, e.g., the weights of a neural network. However, due to the non-convexity of the loss surface of neural networks, averaging can lead to detrimental effects and it remains an open question under which conditions averaging is beneficial. In this paper, we study this problem from the perspective of information theory: We measure the mutual information between representation and inputs as well as representation and labels in local models and compare it to the respective information contained in the representation of the averaged model. Our empirical results confirm previous observations about the practical usefulness of averaging for neural networks, even if local dataset distributions vary strongly. Furthermore, we obtain more insights about the impact of the aggregation frequency on the information flow and thus on the success of distributed learning. These insights will be helpful both in improving the current synchronization process and in further understanding the effects of model aggregation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/13/2023

Layerwise Linear Mode Connectivity

In the federated setup one performs an aggregation of separate local mod...
research
05/22/2022

Federated Learning Aggregation: New Robust Algorithms with Guarantees

Federated Learning has been recently proposed for distributed model trai...
research
10/24/2020

Local Averaging Helps: Hierarchical Federated Learning and Convergence Analysis

Federated learning is an effective approach to realize collaborative lea...
research
10/19/2021

Layer-wise Adaptive Model Aggregation for Scalable Federated Learning

In Federated Learning, a common approach for aggregating local models ac...
research
05/27/2022

FedControl: When Control Theory Meets Federated Learning

To date, the most popular federated learning algorithms use coordinate-w...
research
06/22/2022

How to Combine Variational Bayesian Networks in Federated Learning

Federated Learning enables multiple data centers to train a central mode...
research
04/01/2022

Robust and Efficient Aggregation for Distributed Learning

Distributed learning paradigms, such as federated and decentralized lear...

Please sign up or login with your details

Forgot password? Click here to reset