Securing Distributed Machine Learning in High Dimensions

04/26/2018
by   Lili Su, et al.
0

We consider securing a distributed machine learning system wherein the data is kept confidential by its providers who are recruited as workers to help the learner to train a d--dimensional model. In each communication round, up to q out of the m workers suffer Byzantine faults; faulty workers are assumed to have complete knowledge of the system and can collude to behave arbitrarily adversarially against the learner. We assume that each worker keeps a local sample of size n. (Thus, the total number of data points is N=nm.) Of particular interest is the high-dimensional regime d ≫ n. We propose a secured variant of the classical gradient descent method which can tolerate up to a constant fraction of Byzantine workers. We show that the estimation error of the iterates converges to an estimation error O(√(q/N) + √(d/N)) in O( N) rounds. The core of our method is a robust gradient aggregator based on the iterative filtering algorithm proposed by Steinhardt et al. Steinhardt18 for robust mean estimation. We establish a uniform concentration of the sample covariance matrix of gradients, and show that the aggregated gradient, as a function of model parameter, converges uniformly to the true gradient function. As a by-product, we develop a new concentration inequality for sample covariance matrices of sub-exponential distributions, which might be of independent interest.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/16/2017

Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent

We consider the problem of distributed statistical machine learning in a...
research
05/16/2020

Byzantine-Resilient SGD in High Dimensions on Heterogeneous Data

We study distributed stochastic gradient descent (SGD) in the master-wor...
research
01/16/2023

A Robust Classification Framework for Byzantine-Resilient Stochastic Gradient Descent

This paper proposes a Robust Gradient Classification Framework (RGCF) fo...
research
08/11/2020

Holdout SGD: Byzantine Tolerant Federated Learning

This work presents a new distributed Byzantine tolerant federated learni...
research
05/23/2023

On the Optimal Batch Size for Byzantine-Robust Distributed Learning

Byzantine-robust distributed learning (BRDL), in which computing devices...
research
11/15/2020

Echo-CGC: A Communication-Efficient Byzantine-tolerant Distributed Machine Learning Algorithm in Single-Hop Radio Network

In this paper, we focus on a popular DML framework – the parameter serve...
research
07/25/2023

High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers

Robust distributed learning with Byzantine failures has attracted extens...

Please sign up or login with your details

Forgot password? Click here to reset