No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier

by   Zexi Li, et al.

Data heterogeneity is an inherent challenge that hinders the performance of federated learning (FL). Recent studies have identified the biased classifiers of local models as the key bottleneck. Previous attempts have used classifier calibration after FL training, but this approach falls short in improving the poor feature representations caused by training-time classifier biases. Resolving the classifier bias dilemma in FL requires a full understanding of the mechanisms behind the classifier. Recent advances in neural collapse have shown that the classifiers and feature prototypes under perfect training scenarios collapse into an optimal structure called simplex equiangular tight frame (ETF). Building on this neural collapse insight, we propose a solution to the FL's classifier bias problem by utilizing a synthetic and fixed ETF classifier during training. The optimal classifier structure enables all clients to learn unified and optimal feature representations even under extremely heterogeneous data. We devise several effective modules to better adapt the ETF structure in FL, achieving both high generalization and personalization. Extensive experiments demonstrate that our method achieves state-of-the-art performances on CIFAR-10, CIFAR-100, and Tiny-ImageNet.


page 2

page 14


NeFL: Nested Federated Learning for Heterogeneous Clients

Federated learning (FL) is a promising approach in distributed learning ...

FedAug: Reducing the Local Learning Bias Improves Federated Learning on Heterogeneous Data

Federated Learning (FL) is a machine learning paradigm that learns from ...

Comparative Evaluation of Data Decoupling Techniques for Federated Machine Learning with Database as a Service

Federated Learning (FL) is a machine learning approach that allows multi...

Prototype Guided Federated Learning of Visual Feature Representations

Federated Learning (FL) is a framework which enables distributed model t...

No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data

A central challenge in training classification models in the real-world ...

Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clients

Federated learning (FL) is a distributed framework for collaboratively t...

Please sign up or login with your details

Forgot password? Click here to reset