DeepAI AI Chat
Log In Sign Up

Combining Differential Privacy and Byzantine Resilience in Distributed SGD

by   Rachid Guerraoui, et al.

Privacy and Byzantine resilience (BR) are two crucial requirements of modern-day distributed machine learning. The two concepts have been extensively studied individually but the question of how to combine them effectively remains unanswered. This paper contributes to addressing this question by studying the extent to which the distributed SGD algorithm, in the standard parameter-server architecture, can learn an accurate model despite (a) a fraction of the workers being malicious (Byzantine), and (b) the other fraction, whilst being honest, providing noisy information to the server to ensure differential privacy (DP). We first observe that the integration of standard practices in DP and BR is not straightforward. In fact, we show that many existing results on the convergence of distributed SGD under Byzantine faults, especially those relying on (α,f)-Byzantine resilience, are rendered invalid when honest workers enforce DP. To circumvent this shortcoming, we revisit the theory of (α,f)-BR to obtain an approximate convergence guarantee. Our analysis provides key insights on how to improve this guarantee through hyperparameter optimization. Essentially, our theoretical and empirical results show that (1) an imprudent combination of standard approaches to DP and BR might be fruitless, but (2) by carefully re-tuning the learning algorithm, we can obtain reasonable learning accuracy while simultaneously guaranteeing DP and BR.


page 1

page 2

page 3

page 4


Differential Privacy and Byzantine Resilience in SGD: Do They Add Up?

This paper addresses the problem of combining Byzantine resilience with ...

Byzantine Machine Learning Made Easy by Resilient Averaging of Momentums

Byzantine resilience emerged as a prominent topic within the distributed...

Practical Differentially Private and Byzantine-resilient Federated Learning

Privacy and Byzantine resilience are two indispensable requirements for ...

Bridging Differential Privacy and Byzantine-Robustness via Model Aggregation

This paper aims at jointly addressing two seemly conflicting issues in f...

Fast and Secure Distributed Learning in High Dimension

Modern machine learning is distributed and the work of several machines ...

Making Byzantine Decentralized Learning Efficient

Decentralized-SGD (D-SGD) distributes heavy learning tasks across multip...

Distributed Learning with Curious and Adversarial Machines

The ubiquity of distributed machine learning (ML) in sensitive public do...