Bridging Differential Privacy and Byzantine-Robustness via Model Aggregation

04/29/2022
by   Heng Zhu, et al.
0

This paper aims at jointly addressing two seemly conflicting issues in federated learning: differential privacy (DP) and Byzantine-robustness, which are particularly challenging when the distributed data are non-i.i.d. (independent and identically distributed). The standard DP mechanisms add noise to the transmitted messages, and entangles with robust stochastic gradient aggregation to defend against Byzantine attacks. In this paper, we decouple the two issues via robust stochastic model aggregation, in the sense that our proposed DP mechanisms and the defense against Byzantine attacks have separated influence on the learning performance. Leveraging robust stochastic model aggregation, at each iteration, each worker calculates the difference between the local model and the global one, followed by sending the element-wise signs to the master node, which enables robustness to Byzantine attacks. Further, we design two DP mechanisms to perturb the uploaded signs for the purpose of privacy preservation, and prove that they are (ϵ,0)-DP by exploiting the properties of noise distributions. With the tools of Moreau envelop and proximal point projection, we establish the convergence of the proposed algorithm when the cost function is nonconvex. We analyze the trade-off between privacy preservation and learning performance, and show that the influence of our proposed DP mechanisms is decoupled with that of robust stochastic model aggregation. Numerical experiments demonstrate the effectiveness of the proposed algorithm.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/07/2023

Byzantine-Robust Federated Learning with Variance Reduction and Differential Privacy

Federated learning (FL) is designed to preserve data privacy during mode...
research
08/28/2023

On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learning

This paper jointly considers privacy preservation and Byzantine-robustne...
research
10/29/2022

Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks

In distributed learning systems, robustness issues may arise from two so...
research
02/16/2021

Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?

This paper addresses the problem of combining Byzantine resilience with ...
research
10/08/2021

Combining Differential Privacy and Byzantine Resilience in Distributed SGD

Privacy and Byzantine resilience (BR) are two crucial requirements of mo...
research
12/18/2020

Learning from History for Byzantine Robust Optimization

Byzantine robustness has received significant attention recently given i...
research
02/09/2023

Distributed Learning with Curious and Adversarial Machines

The ubiquity of distributed machine learning (ML) in sensitive public do...

Please sign up or login with your details

Forgot password? Click here to reset