Federated Learning (FL) is a distributed learning algorithm that trains models across multiple decentralized clients and keeps data private simultaneously (McMahan et al., 2017; Konečnỳ et al., 2016). One of the issues in FL is the distinct distributions where decentralized data is diverse due to different properties over each client (Kairouz et al., 2019; Liang et al., 2020; Deng et al., 2020; Li et al., 2019).
In the real world, data is inevitably long-tailed (Yang and Xu, 2020). Fig. 1 illustrates the data distribution of MIT-BIH (Goldberger et al., 2000), a real-world Electrocardiography (ECG) dataset for medical diagnosis. Each patient, which may have different arrhythmia issues over several ECGs, is viewed as a client under the FL setting. Fig. 1(a) visualizes the client distribution as each client point using t-SNE (Maaten and Hinton, 2008). It shows that the global distribution of clients is non-uniformly, where some clients are scattered and far away. Fig. 1(b) plots the class distribution of data from clients, which shares distinct distributions and provides different class data.
FedAvg (McMahan et al., 2017) is a classic FL algorithm where a single model tries to fit among clients by averaging the parameters from local training. Fig. 1(c) presents the Macro-F1 score of FedAvg for global and client-based (local) testing. Since conducting all clients equally, FedAvg ignores the various data distributions between clients, making the poor performance on the global test. Furthermore, FedAvg is easily dominated by major clients but gives up remaining clients, where the F1 score drops drastically on them. Fig. 1 shows this imbalanced issue that makes applying FL to practical applications challenging.
Even if encouraging worse clients to focus on global fairness (Mohri et al., 2019; Li et al., 2019), the performance gap between global and local tests is still significant (Jiang et al., 2019), which indicates that personalization is crucial in FL. The local training (Fallah et al., 2020; Khodak et al., 2019; Liang et al., 2020; Dinh et al., 2020) adopts personalization by training part of local models only on client data as customization. However, the local models, which directly minimize the local error, suffer from overfitting and lose the discrimination of those minor local classes.
In this paper, we introduce Global-Regularized Personalization (GRP-FED) to address client imbalance in FL. As shown in Fig. 2, GRP-FED contains a single global model and local models for each client to consider global fairness and local personalization. Since each client provides different amounts and aspects of class data, our GRP-FED presents Adaptive Aggregation to adaptively adjust the weight of each client and aggregate as a fairer global model. To do personalization, local models are only trained on the specific data for each client. In case of being customizing but overfitting, we present the Global-Regularized Discriminator (D) to distinguish that an extracted feature is from the global or the local model. By jointly optimizing to fool D, local models learn the specific distribution for each client and the general global feature to avoid overfitting.
We conduct the evaluation on real-world MIT-BIH (Goldberger et al., 2000) and synthesis CIFAR-10 (Krizhevsky and Hinton, 2009) datasets under FL setting. The experimental results show that our GRP-FED can improve both global and local tests. Furthermore, the proposed global-regularized discriminator addresses local overfitting effectively. In summary, our contributions are three-fold:
[topsep=0pt, noitemsep, leftmargin=*]
We present GRP-FED to simultaneously consider global fairness and local personalization for Federated Learning;
The proposed adaptively-aggregated global model and customized local models gain improvement under both global and local scenarios;
Extensive ablation studies on both real-world MIT-BIH and synthesis CIFAR-10 show that GRP-FED achieves better performance and deals with client imbalance.
2 Related Work
Federated Learning (FL) Federated Learning (FL) (McMahan et al., 2017; Konečnỳ et al., 2016), where models are trained across multiple decentralized clients, aims to preserve user privacy (Li et al., 2020; Basu et al., 2019) and lower communication costs (Li et al., 2020; Basu et al., 2019). Similar to imbalanced data distribution (Hanzely and Richtárik, 2020; Khodak et al., 2019; Wang et al., 2021; Duan et al., 2020), we investigate the global model used for all and new data and local models that are customized and support personalization for local clients.
Global Model for FL In FL, the global model is trained from all clients and fit the overall global distribution. FedAvg (McMahan et al., 2017) is the first to apply local SGD and build a single global model from a subset of clients. Moreover, they improve global fairness by adapting the global model better to each client (Fallah et al., 2020; Khodak et al., 2019) or treating clients with different importance weights (Mohri et al., 2019). Inspired by q-FFL (Li et al., 2019), which utilizes a constant power to tune the amount of fairness, our GRP-FED adaptively adjusts the power of loss to satisfy dynamic fairness during the global training.
Local Model for FL The performance gap between global and local tests indicates that personalization is crucial in FL (Jiang et al., 2019). Local fine-tuning (Smith et al., 2017; Chen et al., 2020; Khodak et al., 2019; Fallah et al., 2020; Liang et al., 2020) supports personalization by training each local model only on client data. While, pFedMe (Dinh et al., 2020) argues that directly minimizing local error is prone to overfit and adopts Moreau envelopes to help decouple personalization. Different from that, our GRPFED introduces the Global-Regularized Discriminator to regularize the local feature distribution and mitigate the local overfitting issue.
Task Definition Federated Learning (FL) is to learn from independent clients where client contains a local dataset . represents the pair of th data and its label where is the number of data in . We consider FL as classification task where is the class label of . Intuitively, each client captures a different view of the global data distribution . However, since each client provides with distinct distributions in real world, FL easily suffers from the client imbalance under practical applications.
GRP-FED To address the client imbalanced issue, we present Global-Regularized Personalization (GRP-FED) into FL. An overview of GRP-FED111: global feature extractor, : local feature extractor, : classifier, and : discriminator in client . is illustrated in Fig. 2. For a data point , the feature extractor extracts the lower-dimensional representation of
, and the classifierperforms the output prediction . GRP-FED consists of a fair global model that applies adaptive aggregation to consider different aspects from clients, and local models to customize each client. The proposed adaptive aggregation adjusts the aggregated proportion to ensure the fairness over distinct clients. The local models do personalization, where a global-regularized discriminator prevents it from overfitting when optimizing the client data.
3.2 Global Fairness
Global Training Global fairness aims at building a global model that can fairly cope with the global distribution over distinct clients. The global model includes the global feature extractor parameterized by and the classifier parameterized by . The global training is to train the global model in each client and then aggregated as a single global model:
where is the extracted global feature.
is calculated by the loss functionat time step and updates to receive . In conventional FL, FedAvg (Smith et al., 2017) considers the global model by averaging all trained global models from clients. However, under the real-world FL setting, client data is collected from different environments, scenarios, or applications. The data distribution from each client presents diversely, which results in poor generalization if treating them equally. To deal with the client imbalanced issue, we propose adaptive aggregation for a fairer aggregated proportion.
Adaptive Aggregation q-FFL (Li et al., 2019) adopts a constant power that tunes the amount of fairness. However, the training process for FL can be dynamic over distinct clients, where a fixed power of loss is difficult to satisfy the expected fairness under all situations. To overcome this issue, we present adaptive aggregation and consider a dynamic
to adaptively adjusts for better fairness. We treat fairness as the standard deviation () of the global training loss in all clients. If is high, the global training loss is quite different and the global model may suffer from client imbalance. Therefore, we adjust the loss of power :
Otherwise, if the fairness becomes relatively fairer, we should decrease for more robust training. Finally, we acquire the new global model by aggregating all trained global models, weighted by the global training loss and the adaptive power :
In this way, we can adaptively adjust to satisfy the dynamic fairness during the global training by considering the standard deviation of the global training loss over all clients.
3.3 Local Personalization
Local Training Apart from a single global model, since each client is collected from different sources and under various usages, local models that support personalization are also crucial. The local training is to train each local model only with the data in client for personalization:
where is the extracted personalized feature. Similar to the global training, is updated by from . Thereby, we personalize the local feature in the specific client. Note that we fix the classifier with during the local training for a personalized local feature distribution.
Global-Regularized Discriminator () After the local training, we can have the personalized feature . However, since under FL, the client data distribution is far from global , the learned may be just overfitting on that client but suffers from poor generalization for the global scenario. To mitigate this overfitting issue, we introduce Global-Regularized Discriminator (). Each client maintains its own parameterized by , which serves as a binary classifier to distinguish an extracted feature is from the global or the local feature extractor. We make the global feature by as the true case and the local feature by as the false case, and train as following:
where and are fixed, and only is updated during the training. With the help of , the local feature extractor can be regularized to prevent overfitting:
This time, should be freeze and is optimized to fool the discriminator . By updating the local training along with the global-regularized discriminator, the local feature extractor learns to personalize and imitate the global feature distribution, which can avoid client overfitting.
|Method||Global Test||Local Test||Personalization||Generalization||Global Test||Local Test||Personalization||Generalization|
3.4 Learning of GRP-FED and Inference
The learning process of GRP-FED is presented in Algo. 1. For each round , to make the federated learning stable, we follow (Smith et al., 2017) that randomly selects clients as for training. At first, the server copies global feature extractor and classifier to the clients for independent federated training. Both and are trained through data from all clients during the global training. For the local training, the local feature extractor is only trained by the client data and updated from the to do personalization on client . Also, is jointly trained from to prevent overfitting. The global-regularized discriminator () then updates from by discriminating an feature is extracted from the local or global model, where is the weight of loss between and .
After returning all trained , , and global training loss from each client , we aggregate and as the new global model over the aggregated weight . is updated from the adaptive power and the global training loss to force investigating different proportions of clients. In total, the entire training loss of GRP-FED is:
Inference During inference, given an example , we consider two testing types for both local and global scenario:
[topsep=0pt, noitemsep, leftmargin=*]
local test: if belongs to client , we apply the local model () for the best personalization;
global test: otherwise, is fed to the global model () as an unknown example from the global distribution.
We also conduct these two types of testing in our experiments to evaluate both global fairness (global test) and local personalization(local test) of our proposed GRP-FED.
4.1 Experimental Setting
Dataset We evaluate our GRP-FED on two federated classification datasets, real-world MIT-BIH (Goldberger et al., 2000), and synthesis CIFAR-10 (Krizhevsky and Hinton, 2009). MIT-BIH is an Electrocardiography (ECG) dataset for medical diagnosis, where each fragment belongs to one of 12 arrhythmia classes. There are 46 patients in MIT-BIH, containing different numbers of ECG fragments and presenting various class distributions. We treat each patient as a single client that supports both personalized evaluation for a specific patient and global evaluation over all clients.
We distribute the entire CIFAR-10 dataset to 50 clients as the FL setting. To imitate different client distributions, each client contains different numbers of total data (with decreasing over clients). The class distribution is randomly sampled (with decreasing over the number of each class). is that follows the distribution of MIT-BIH.
Evaluation Metrics Since the class distribution over real-world data is non-uniform, the classic accuracy (%) cannot reflect the proper performance of the prediction and may ignore those minor classes with less examples. We adopt macro-F1, the mean F1-score of each class, to treat them as the same importance. This evaluation is more suitable under data imbalance. For instance, we care more about those examples with different arrhythmia issues in MIT-BIH.
Testing Scenario We conduct global usage and local personalization under two testing scenarios:
[topsep=0pt, noitemsep, leftmargin=*]
Global Test (): the global model predicts the entire testing set to evaluate the fairness over global distribution;
Local test (): we consider both aspects of local personalization () and local generalization () to avoid overfitting. is the mean performance of local models under their clients; calculates from the mean macro-F1 score of each local model in the global test. Concerning both personalization and overfitting, the overall performance of the local test () is computed as:
Baselines We compare against various FL methods:
[topsep=0pt, noitemsep, leftmargin=*]
Local-only: Local and LG-FedAvg (Liang et al., 2020);
For global-only methods, the global model evaluates under each client to perform the local test. Following LG-FedAvg (Liang et al., 2020), we ensemble results from all local models as the global output for local-only algorithms. With GRP-FED or global-local frameworks, we apply the global model for the global test and local models for the local test.
Implementation Detail As the classification task, we apply the cross-entropy loss for the loss function . We adopt 5-layer 1D ResNet (He et al., 2016) to process ECG under MIT-BIH and ResNet-30 under CIFAR10 as the feature extractor . The classifier is a 2-layer fully-connected (FC) that projects the feature into class prediction. The global-regularized discriminator
is also 2-layer FC but projects to binary indication for the true/false discrimination. We set the local epochand the batch size 64. SGD optimizes all parameters with a learning rate () 5e-3, adjusting rate ( in Eq. 2) 0.5, momentum 0.9. The initial loss power is 10, which is the same as q-FFL.
4.2 Quantitative Results
Global Test Table 1 shows the results of GRP-FED and baselines on both real-world MIT-BIH and synthesis CIFAR-10 datasets. The global test () is to evaluate the fairness of the global model over the entire testing set. It shows that our GRP-FED achieves the highest macro-F1 score on both MIT-BIH () and CIFAR-10 (). Since the proposed adaptive aggregation adjusts the power of loss according to the dynamic fairness, it gains a significant improvement under and achieves better global fairness.
Local Test Local personalization is essential when regarding a specific client under the FL setting. At first, LG-FedAvg (Liang et al., 2020) performs the best in the local personalization test (96.4% ) on MIT-BIH. However, since the local features are merely learned from the client, they are easily overfitting and result in poor generalization in the local generalization test (16.9% ). q-FFL has the highest 55.1% but presents lower 60.2% without personalization. With the global-regularized discriminator, local models in our GRP-FED can extract personalized features but avoid overfitting. We surpass all baselines in the overall local test (55.3% ) with a comparable 86.4% and 42.4% . A similar trend can be found on CIFAR-10. Our GRP-FED achieves the highest 55.2% and strikes the most appropriate balance between personalization (61.1% ) and generalization (51.6% ).
4.3 Ablation Study
Is the Global Model Actually Fair? To ensure the global model is actually fair, we plot the learning curve of the mean and max global training loss in Fig. 3. Basically, all methods have a relatively low mean training loss during the global training. We can investigate the global fairness through the max global training loss. FedAvg treats each client equally and sacrifices those minor classes, resulting in a high max global training loss in the end. The adversarial aggregation in AFL is not fair enough and still remains high max training loss. q-FFL adopts a constnat loss power to tune the amount of fairness. While, a fixed power of loss cannot satisfy all fairness situations and instead increases the max training loss at last. Our adaptive aggregation considers a dynamic that can adaptively adjust, which keeps decreasing the max training loss as the fairer global model.
How affects the local models? We adopt to control the weight of loss between the personalization by the local training and the generalization by the global-regularized discriminator. Fig. 4 illustrates the effect of on MIT-BIH during the local personalization. There is a trade-off between local personalization () and local generalization (). When gets larger, we treat the personalization as more important and improve but hurt . On the other hand, increases when decreases if we consider the local feature should be more generalized with lower . leads to the best local test () in our GRP-FED.
Case Study Fig. 5 visualizes the performance of two clients on MIT-BIH. Since the upper client is similar to the global class distribution, all methods perform well under both the global test () and local personalization test (). The trained local model on this client also performs well in the overall local test (). However, in the lower client, which presents a distinct class distribution, LG-FedAvg is entirely overfitting and results in high but terrible . By contrast, our GRP-FED still outperforms the remaining baselines in and achieves the highest with the help of the global-regularized discriminator. Moreover, GRP-FED considers the dynamic fairness from different client distributions and leads to the best as a fairer global model.
In this paper, we introduce Global-Regularized Personalization (GRP-FED) to address the client imbalance issue under federated learning (FL). GRP-FED consists of a global model and local models for each client. The global model considers the dynamic fairness and investigates different proportions of clients with the adaptive aggregation. The local models do personalization by the local training, and the proposed global-regularized discriminator can prevent the overfitting issue. Extensive results show that our GRP-FED outperforms baselines under both global and local scenarios on real-world MIT-BIH and synthesis CIFAR-10 datasets.
- Qsparse-local-sgd: distributed SGD with quantization, sparsification and local computations. In Advances in Neural Information Processing Systems, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett (Eds.), Cited by: §2.
Federated multi-task learning with hierarchical attention for sensor data analytics.
2020 International Joint Conference on Neural Networks (IJCNN), External Links: Cited by: §2.
- Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461. Cited by: §1.
- Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Cited by: §1, §2, 3rd item.
- Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems 32 (1), pp. 59–71. Cited by: §2.
- Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. Advances in Neural Information Processing Systems. Cited by: §1, §2, §2, 3rd item.
- PhysioBank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101 (23). Cited by: §1, §1, §4.1.
- Federated learning of a mixture of global and local models. CoRR abs/2002.05516. External Links: Cited by: §2.
- Deep residual learning for image recognition. CVPR. Cited by: §4.1.
- Improving federated learning personalization via model agnostic meta learning. CoRR abs/1909.12488. External Links: Cited by: §1, §2.
- Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977. Cited by: §1.
- Adaptive gradient-based meta-learning methods. In Advances in Neural Information Processing Systems, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett (Eds.), Cited by: §1, §2, §2, §2.
- Federated learning: strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492. Cited by: §1, §2.
- Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto. Cited by: §1, §4.1.
- Fair resource allocation in federated learning. arXiv preprint arXiv:1905.10497. Cited by: §1, §1, §2, §3.2, 1st item.
Acceleration for compressed gradient descent in distributed and federated optimization.
Proceedings of the 37th International Conference on Machine Learning, ICML, Cited by: §2.
- Think locally, act globally: federated learning with local and global representations. arXiv preprint arXiv:2001.01523. Cited by: §1, §1, §2, 2nd item, §4.1, §4.2.
- Visualizing data using t-sne. Journal of machine learning research 9 (Nov). Cited by: §1.
Communication-efficient learning of deep networks from decentralized data.
Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS, A. Singh and X. (. Zhu (Eds.), Cited by: §1, §1, §2, §2.
- Agnostic federated learning. In Proceedings of the 36th International Conference on Machine Learning, ICML, K. Chaudhuri and R. Salakhutdinov (Eds.), Cited by: §1, §2, 1st item.
- Federated multi-task learning. In Advances in neural information processing systems, Cited by: §2, §3.2, §3.4, 1st item.
- Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, pp. 10165–10173. Cited by: §2.
- Rethinking the value of labels for improving class-imbalanced learning. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Cited by: §1.