FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning

by   Wei Chen, et al.
The University of Texas at Austin
Carnegie Mellon University

In this paper, we identify a new phenomenon called activation-divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets.


page 1

page 2

page 3

page 4


FedCAT: Towards Accurate Federated Learning via Device Concatenation

As a promising distributed machine learning paradigm, Federated Learning...

FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximation

Federated Learning (FL) is a decentralized machine learning architecture...

Partial Variable Training for Efficient On-Device Federated Learning

This paper aims to address the major challenges of Federated Learning (F...

Resource-Efficient Federated Learning for Heterogenous and Resource-Constrained Environments

Federated Learning (FL) is a privacy-enforcing sub-domain of machine lea...

Learnings from Federated Learning in the Real world

Federated Learning (FL) applied to real world data may suffer from sever...

FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for Resource and Data Heterogeneity

Federated Learning (FL) enables training a global model without sharing ...

FedorAS: Federated Architecture Search under system heterogeneity

Federated learning (FL) has recently gained considerable attention due t...

Code Repositories


Source code for ECML-PKDD (2020) paper: FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning

view repo

Please sign up or login with your details

Forgot password? Click here to reset