Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization

02/23/2022
by   Jaehong Yoon, et al.
7

In practical federated learning scenarios, the participating devices may have different bitwidths for computation and memory storage by design. However, despite the progress made in device-heterogeneous federated learning scenarios, the heterogeneity in the bitwidth specifications in the hardware has been mostly overlooked. We introduce a pragmatic FL scenario with bitwidth heterogeneity across the participating devices, dubbed as Bitwidth Heterogeneous Federated Learning (BHFL). BHFL brings in a new challenge, that the aggregation of model parameters with different bitwidths could result in severe performance degeneration, especially for high-bitwidth models. To tackle this problem, we propose ProWD framework, which has a trainable weight dequantizer at the central server that progressively reconstructs the low-bitwidth weights into higher bitwidth weights, and finally into full-precision weights. ProWD further selectively aggregates the model parameters to maximize the compatibility across bit-heterogeneous weights. We validate ProWD against relevant FL baselines on the benchmark datasets, using clients with varying bitwidths. Our ProWD largely outperforms the baseline FL algorithms as well as naive approaches (e.g. grouped averaging) under the proposed BHFL scenario.

READ FULL TEXT

page 8

page 13

research
08/17/2020

Inverse Distance Aggregation for Federated Learning with Non-IID Data

Federated learning (FL) has been a promising approach in the field of me...
research
05/13/2023

A Federated Learning-based Industrial Health Prognostics for Heterogeneous Edge Devices using Matched Feature Extraction

Data-driven industrial health prognostics require rich training data to ...
research
02/01/2022

Factorized-FL: Agnostic Personalized Federated Learning with Kernel Factorization Similarity Matching

In real-world federated learning scenarios, participants could have thei...
research
12/23/2022

Deep Unfolding-based Weighted Averaging for Federated Learning under Heterogeneous Environments

Federated learning is a collaborative model training method by iterating...
research
06/03/2022

Supernet Training for Federated Image Classification under System Heterogeneity

Efficient deployment of deep neural networks across many devices and res...
research
08/22/2023

Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning

Federated learning (FL) inevitably confronts the challenge of system het...
research
06/30/2023

Improving Federated Aggregation with Deep Unfolding Networks

The performance of Federated learning (FL) is negatively affected by dev...

Please sign up or login with your details

Forgot password? Click here to reset