1 Introduction
The security of distributed machine learning has drawn increasing attention in recent years. Among the threat models, Byzantine failures (Lamport et al., 1982) are perhaps the most wellstudied. In the Byzantine model, workers can behave arbitrarily and maliciously. In addition, Byzantine workers are omniscient and can conspire. Most of the existing Byzantinetolerant machinelearning algorithms (Blanchard et al., 2017; Chen et al., 2017; Yin et al., 2018; Feng et al., 2014; Su & Vaidya, 2016a; b; Alistarh et al., 2018)
focus on the protection of distributed Stochastic Gradient Descent (SGD).
In this paper, we consider Byzantinetolerant SGD in a serverworker architecture (also known as the parameter server architecture (Li et al., 2014a; b)), depicted in Figure 3
. The system is composed of server nodes and worker nodes. In each epoch, the workers pull the latest model from the servers, estimate the gradients using the locally sampled training data, and then push the gradient estimators to the servers. The servers aggregate the gradient estimators, and update the model by using the aggregated gradients.
We consider Byzantine failures at a subset of the worker nodes. Byzantine workers send arbitrary values instead of the gradient estimators to the server. Such Byzantine gradients are potentially adversarial, and this can result in convergence to suboptimum models, or even lead to divergence. To make things worse, the Byzantine workers can spy on the information at any server or at any honest worker (omniscience). Byzantine gradients can thus be tailored to have similar variance and magnitude as the correct gradients, which makes them hard to distinguish. Additionally, in different iterations, different subsets of workers can behave in a Byzantine manner, evading detection. Existing literature assumes that less than half of the workers are Byzantine in any iteration.
Compared to traditional Byzantine tolerance in distributed systems (Lynch, 1996; Avizienis et al., 2004; Tanenbaum & Van Steen, 2007; Fischer et al., 1982), Byzantine tolerance in distributed machine learning has unique properties and challenges. Traditional Byzantine tolerance attempts to reach consensus on correct values. However, machine learning algorithms do not need to reach consensus. Further, even nonByzantinetolerant machine learning algorithms can naturally tolerate some noise in the input and during execution (Xing et al., 2016). Thus for distributed SGD, existing techniques for Byzantinetolerant execution guarantee an upperbound on the distance between the aggregated approximate gradient (under Byzantine workers) and the true gradient (Blanchard et al., 2017; Yin et al., 2018).
A deeper introspection reveals, however, that what really matters for gradient descent algorithms is the direction of the descent. As shown in Figure 2
, to let the gradient descent algorithm make progress, we need to guarantee that the direction of the aggregated vector agrees with the true gradient, i.e., the inner product between the aggregated vector and the true gradient must be nonnegative. This can be violated by an attack that makes the inner product negative. We call this class of new attacks “inner product manipulation attacks”.
We observe that the bounded distance between the aggregated value and the true gradient guaranteed by existing techniques is not enough to defend distributed synchronous SGD against inner product manipulation attacks. For example, for the coordinatewise median, if we put all the Byzantine values on the opposite side of the true gradient, the inner product between the aggregated vector and the true gradient can be manipulated to be negative.
In this paper, we study how inner product manipulation makes Byzantinetolerant SGD vulnerable. We conduct case studies on coordinatewise median (Yin et al., 2018) and Krum (Blanchard et al., 2017). Figure 1 gives a glimpse of how bad the effect of the attack can be. In a nutshell, creating gradients in the opposite direction with large magnitude crashes coordinatewise median, while creating gradients in the opposite direction with small magnitude crashes Krum. We provide theoretical analysis as well as empirical results to validate these findings.
Based on these results, we argue that there is a need to revise the definition of Byzantine tolerance in distributed SGD. We provide a new definition, and study its satisfaction on two prevailing robust distributed SGD algorithms, theoretically and empirically. In summary, our contributions are:

We break two prevailing Byzantine tolerant SGD algorithms – coordinatewise median (Yin et al., 2018) and Krum (Blanchard et al., 2017) – using a new class of attacks called inner production manipulation attacks. We theoretically prove that under certain conditions, we can backdoor these two algorithms, even when the assumptions and theorems presented in these papers are valid.

We show how to design Byzantine gradients to compromise the robust aggregation rules. We conduct experiments to validate further.

Following our theoretical and empirical analysis, we propose a revised definition of Byzantine tolerance for distributed SGD.
2 Related Work
Robust estimators such as the median are well studied, and can naturally be applied to Byzantine tolerance. Coordinatewise median is one approach that generalizes the median to highdimensional vectors. In Yin et al. (2018), statistical error rates are studied for using coordinatewise median in distributed SGD.
Blanchard et al. (2017) propose Krum, which is not based on robust statistics. For each candidate gradient, Krum computes the local sum of squared Euclidean distances to the other candidates, and outputs the one with minimal sum.
In this paper, we focus on coordinatewise median and Krum. There are other Byzantinetolerant SGD algorithms. For example, Bulyan (Guerraoui et al., 2018) is built based on Krum, which potentially shares the same flaws. DRACO (Chen et al., 2018) uses coding theory to ensure robustness, and is different from the other Byzantinetolerant SGD algorithms.
Recently, an increasing number of papers propose attack mechanisms to break the defense of machine learning in various scenarios. For example, Athalye et al. (2018) propose attack techniques using adversarial training data. Bhagoji et al. (2018); Bagdasaryan et al. (2018) break the defense of federated learning (McMahan et al., 2016). In this paper, we focus on attacking distributed synchronous SGD using adversarial gradients sent by Byzantine workers.
3 Preliminaries
In this paper, we focus on distributed synchronous Stochastic Gradient Descent (SGD) with Parameter Server (PS). In this section, we formally introduce distributed synchronous SGD and the threat model of Byzantine failures.
3.1 Stochastic Gradient Descent
We consider the following optimization problem:
where is a differentiable function, is sampled from some unknown distribution , is the number of dimensions. We assume that there exists at least one minimizer of , which is denoted by , where .
This problem is solved in a distributed manner with workers. In each iteration, each worker will sample independent and identically distributed (i.i.d.) data points from the distribution , and compute the gradient of the local empirical loss , where is the th sampled data on the th worker. The servers will collect and aggregate the gradients sent by the workers, and update the model as follows:
where is an aggregation rule (e.g., averaging), and is the gradient sent by the th worker, is the learning rate in the iteration. For an honest worker,
is an unbiased estimator such that
. When all the workers are honest, the most common choice of the aggregation rule is averaging:The detailed algorithm of distributed synchronous SGD with aggregation rule is shown in Algorithm 1.
3.2 Threat Model
In the Byzantine failure model, the gradients sent by malicious workers can take an arbitrary value:
(1) 
where “” represents arbitrary values.
Formally, we define the threat model of Byzantine failure as follows.
Definition 1.
(Threat Model (Blanchard et al., 2017; Chen et al., 2017; Yin et al., 2018)). In the iteration, let be i.i.d. random vectors in , where . The set of correct vectors is partially replaced by arbitrary vectors, which results in , as defined in Equation (1). In other words, a correct gradient is , while a Byzantine gradient, marked as “”, is assigned arbitrary value. We assume that out of vectors are Byzantine, where . Furthermore, the indices of faulty workers can change across different iterations. If the failures are caused by attackers, the threat model includes the case where the attackers can collude.
The notations used in this paper is summarized in Table 1.
Notation  Description 

Number of workers  
Minibatch size on each worker  
Number of iterations  
Set of integers  
Number of Byzantine workers  
Learning rate  
Model parameters  
Gradient sent by the th worker  
in the iteration, potentially Byzantine  
Correct gradient produced by the th worker  
in the iteration  
All the norms in this paper are norms  
Inner product between and 
4 Defense Techniques
In this section, we introduce two prevailing robust aggregation rules against Byzantine failures in distributed synchronous SGD: coordinatewise median and Krum. For the remainder of this paper, we ignore the iteration superscript in and for convenience.
4.1 CoordinateWise Median
Definition 2.
(Coordinatewise Median (Yin et al., 2018)) We define the coordinatewise median aggregation rule as
where for any , the th dimension of is , is the th dimension of the vector , is the onedimensional median.
4.2 Krum
Definition 3.
(Krum (Blanchard et al., 2017))
where are the indices of the nearest neighbours of in as measured by squared Euclidean distance.
For convenience, we refer to the coordinatewise median and Krum as Median and Krum.
5 Attack Techniques
In this section, we revise the definition of Byzantine tolerance in distributed synchronous SGD. Then, we theoretically analyze the Byzantine tolerance of coordinatewise median and Krum, and show that under certain conditions, these two robust aggregation rules are no longer Byzantinetolerant.
5.1 Inner Product Manipulation
In the previous work on Byzantinetolerant SGD algorithms, most of the robust aggregation rules only guarantee that the robust estimator is not arbitrarily far away from the mean of the correct gradients. In other words, the distance between the robust estimator and the correct mean is upperbounded. However, for gradient descent algorithms, to guarantee the descent of the loss, the inner product between the true gradient and the robust estimator must be nonnegative:
so that at least the loss will not increase in expectation. In particular, bounded distance is not enough to guarantee robustness, if the attackers manipulate the Byzantine gradients and make the inner product negative.
The intuition underlying the inner product manipulation attack is that, when gradient descent algorithm converges, the gradient approaches . Thus, even if the distance between the robust estimator and the correct mean is bounded, it is still possible to manipulate their inner product to be negative, especially when the upper bound of such distance is large.
We formally define a revised version of Byzantine tolerance for distributed synchronous SGD (DSSGDByzantine tolerance):
Definition 4.
(DSSGDByzantine Tolerance) Without loss of generality, suppose that in a specific iteration, the server receives correct gradients and Byzantine gradients . We assume that the correct gradients have the same expectation . An aggregation rule is said to be DSSGDByzantinetolerant if
With the revised definition, now we theoretically analyze the DSSGDByzantine tolerance of coordinatewise median and Krum.
Remark 1.
Note that we do not argue that the theoretical guarantees in the previous work are wrong. Instead, our claim is that the theoretical guarantees on the bounded distances are not enough to secure distributed synchronous SGD. In particular, DSSGDByzantine tolerance is different from the Byzantine tolerance proposed in previous work.
5.2 CoordinateWise Median
The following theorem shows that under certain conditions, Median is not DSSGDByzantinetolerant.
Theorem 1.
We consider the worst case where . The server receives correct gradients and Byzantine gradients . We assume that the stochastic gradients have identical expectation , and nonzero coordinatewise variance , where is the th coordinate of , and is the th coordinate of . When , there exist Byzantine gradients such that
Proof.
(sketch) Since median is independently taken in each coordinate, it is sufficient to prove Byzantine vulnerability for one coordinate or scalars. Thus, for convenience, with a little bit abuse of notation, we suppose that the correct gradients and Byzantine gradients are all scalars. We only need to show that under certain attacks, the aggregated value has a different sign than .
Without loss of generality, we assume that (the mirror case can be easily proved with a similar procedure). The Byzantine gradients are all assigned negative value: . Furthermore, we make the Byzantine gradients small enough such that .
By sorting the correct gradients, we can define the sequence , where is the th smallest element in :
We also define the expectation of the th smallest element: .
Then, it is easy to check that , and .
Thus, when , is negative.
∎
Remark 2.
When gradient descent converges, the expectation of the gradient approaches . Furthermore, since the gradient produced by the correct workers are stochastic, the variance always exists. Thus, eventually, the condition will be satisfied. To make things worse, the closer SGD approaches a critical point, the less likely the coordinatewise median is DSSGDByzantinetolerant.
Remark 3.
The proof of Theorem 1 provides the intuition of constructing adversarial gradients for the attackers. In practice, in each coordinate, the attackers only need to guarantee that all the Byzantine values are much smaller than the smallest correct value if the correct expectation is positive, or much larger than the largest correct value if the correct expectation is negative. Then, hopefully, if the variance is large enough, the smallest/largest value has the opposite sign to the correct expectation. Then, the attackers can successfully manipulate the aggregated value into the opposite direction to the correct expectation.
5.2.1 Toy Example
We provide an 1dimensional toy example to illustrate how easily Median can fail. Suppose there are correct gradients with the mean , and Byzantine gradient with the negative mean . According to Definition 2, it is easy to check that , which means that Median produces a value with the opposite sign of the mean of the correct gradients.
5.3 Krum
The following theorem proves that under certain conditions, Krum is not DSSGDByzantinetolerant. Note that Krum requires that .
Theorem 2.
We consider the worst case where . The server receives correct gradients and Byzantine gradients . We assume that the stochastic gradients have identical expectation . We define the mean of the correct gradients . We assume that the correct gradients are bounded by . Furthermore, we assume that , and such that . We take , where is a small positive constant value such that . When is large enough: , we have
Proof.
(sketch) For , , where .
Since any is identical, the nearest neighbours of must belong to . The remaining nearest neighbours must belong to the set of correct gradients . Thus, we have
For the correct gradients , there are two cases:

Case 1: There are some which belong to the nearest neighbours of .
Suppose there are nearest neighbours in and nearest neighbours in , where . Since the correct gradients are bounded by , it is easy to check that . Thus, we have

Case 2: There are no which belong to the nearest neighbours of . Thus, we have
In both cases, we have Thus, when is large enough: , we have
As a result, . Thus, .
∎
Remark 4.
In the theorem above, we assume that all the correct gradients are inside a Euclidean ball centered at their mean: . Such assumption can not always be satisfied, but it is reasonable that the random samples are sometimes inside such a Euclidean ball, if the variance is not too large. On the other hand, we assume that the pairwise distances between the correct gradients are lowerbounded by . Almost surely, such exists, no matter how small it is. Note that the Byzantine attackers are supposed to be omniscient. Thus, the attackers can spy on the honest workers, and obtain and . Then, the attackers can choose an such that . Finally, we only need the number of workers to be large enough, so that .
Remark 5.
The proof of Theorem 2 provides the intuition of constructing adversarial gradients for the attackers. In practice, the attackers only need to assign to all the Byzantine gradients, with an small enough.
Remark 6.
Note that in Blanchard et al. (2017), Krum requires the assumption that for convergence, where is a general constant, is the maximal variance of the gradients, and is the gradient in expectation. Note that when SGD converges to a critical point. Thus, such an assumption is never guaranteed to be satisfied, if the variance is nonzero. Furthermore, the better SGD converges, the less likely such an assumption can be satisfied.
5.3.1 Toy Example
Note that the assumptions made in Theorem 2 are sufficient but not necessary conditions of the DSSGDByzantine vulnerability of Krum. In practice, it can be easier to find an that crashes Krum, especially for 1dimensional cases.
We provide an 1dimensional toy example to show how easily Krum can fail. Suppose there are correct gradients with the mean , and Byzantine gradient with the negative mean . According to Definition 3, the corresponding function values of are . Thus, , which means that Krum chooses the Byzantine gradient with the opposite sign of the mean of the correct gradients.
6 Case Study
In this section, we implement special attack strategies for Median and Krum, and evaluate our attack strategies on a realworld application. The attack strategies are designed by using the intuitions underlying Theorem 1 and Theorem 2, which are mentioned in Remark 3 and Remark 5.
6.1 Datasets and Evaluation Metrics
We conduct experiments on the benchmark CIFAR10 image classification dataset (Krizhevsky & Hinton, 2009)
, which is composed of 50k images for training and 10k images for testing. We use a convolutional neural network (CNN) with 4 convolutional layers followed by 1 fully connected layer. The detailed network architecture can be found in the appendix. For any worker, the minibatch size for SGD is
.In each experiment, we launch 25 worker processes. We repeat each experiment 10 times and take the average. We use top1 accuracy on the testing set and the crossentropy loss function on the training set as the evaluation metrics.
We use the averaging, Median, and Krum without attacks as the gold standard, which are referred to as Mean without attack, Median without attack, and Krum without attack. We start the attack at different epochs, so that SGD can warm up and make some progress first. We include some additional experiments in the appendix.
6.2 Median
In each iteration, the server receives gradients. A randomly selected subset of correct gradients are replaced by Byzantine gradients. We define the set of Byzantine gradients as , and the set of the remaining correct gradients as . Our attack strategy is as follows:
According to Theorem 1 and Remark 3, Median is vulnerable to positive with large magnitude .
We test the above attack strategy with different . The results are shown in Figure 4. Median fails when . When , Median gets stuck and stops making progress. When , Median successfully defends against the attack.
6.3 Krum
In each iteration, the server receives gradients. A randomly selected subset of correct gradients are replaced by Byzantine gradients. We define the set of Byzantine gradients as , and the set of the remaining correct gradients as . Our attack strategy is as follows:
According to Theorem 2 and Remark 5, Krum is vulnerable to positive with small magnitude .
We test the above attack strategy with different . The results are shown in Figure 5. Krum fails when is small. When is large enough, Krum successfully defends against the attack.
6.4 Discussion
Surprisingly, both Median and Krum are more vulnerable than we expected. Note that our theorems only analyze the worst cases. There are other cases where Median and Krum can fail.
For Median, even if we take , SGD still performs badly. Theoretically, even if we do not use positive , small can still enlarge the variance of SGD, which can be potentially harmful to the convergence. We can see that with large negative , the defense of Median is successful. In our experiment, we reveal certain new vulnerabilities of Median in distributed synchronous SGD. The experiments conducted by Yin et al. (2018) do not fail because the attacker only changes the labels of the poisoned training data by flipping a to . It is very likely that such an attack produces Byzantine gradients surrounding the correct gradients coordinatewisely on both sides. However, according to Theorem 1 and Remark 3, an effective attack should place the Byzantine gradient on one and only one side of the correct gradients, which is the side opposite to the mean of the correct gradients, coordinatewisely.
For Krum, small positive makes SGD vulnerable. Furthermore, even if we take , Krum still fails. In our experiment, we reveal certain new vulnerability of Krum in distributed synchronous SGD. The experiments conducted by Blanchard et al. (2017) do not fail even though a similar attack strategy called “omniscient” is conducted. The reason is that, in the paper of Blanchard et al. (2017), the attacker “proposes the opposite vector, scaled to a large length”, which is similar to our attack strategy with a large .
Guided by our theoretical analysis, we designed efficient attack strategies for both Median and Krum. Our results show that the definition of Byzantine tolerance for distributed synchronous SGD should be revised. Using our definition of DSSGDByzantine tolerance, research can be conducted to design better defense techniques.
7 Conclusion
We propose a revised definition of Byzantine tolerance for distributed synchronous SGD. With the new definition, we theoretically and empirically examine the Byzantine tolerance of two prevailing robust aggregation rules. Guided by our theoretical analysis, attack techniques can be designed to fail the aggregation rules. In the future, we hope new defense techniques can be designed using our revised definition of Byzantine tolerance.
References
 Alistarh et al. (2018) Alistarh, D., AllenZhu, Z., and Li, J. Byzantine stochastic gradient descent. arXiv preprint arXiv:1803.08917, 2018.
 Arnold et al. (1979) Arnold, B. C., Groeneveld, R. A., et al. Bounds on expectations of linear systematic statistics based on dependent samples. The Annals of Statistics, 7(1):220–223, 1979.
 Athalye et al. (2018) Athalye, A., Carlini, N., and Wagner, D. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274–283, 2018.
 Avizienis et al. (2004) Avizienis, A., Laprie, J.C., Randell, B., and Landwehr, C. Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing, 1(1):11–33, 2004.
 Bagdasaryan et al. (2018) Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., and Shmatikov, V. How to backdoor federated learning. arXiv preprint arXiv:1807.00459, 2018.
 Bhagoji et al. (2018) Bhagoji, A. N., Chakraborty, S., Mittal, P., and Calo, S. Analyzing federated learning through an adversarial lens. arXiv preprint arXiv:1811.12470, 2018.
 Blanchard et al. (2017) Blanchard, P., Guerraoui, R., Stainer, J., et al. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems, pp. 118–128, 2017.
 Chen et al. (2018) Chen, L., Wang, H., Charles, Z., and Papailiopoulos, D. Draco: Byzantineresilient distributed training via redundant gradients. In International Conference on Machine Learning, pp. 902–911, 2018.
 Chen et al. (2017) Chen, Y., Su, L., and Xu, J. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. POMACS, 1:44:1–44:25, 2017.
 Feng et al. (2014) Feng, J., Xu, H., and Mannor, S. Distributed robust learning. arXiv preprint arXiv:1409.5937, 2014.
 Fischer et al. (1982) Fischer, M. J., Lynch, N. A., and Paterson, M. S. Impossibility of distributed consensus with one faulty process. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE LAB FOR COMPUTER SCIENCE, 1982.
 Guerraoui et al. (2018) Guerraoui, R., Rouault, S., et al. The hidden vulnerability of distributed learning in byzantium. In International Conference on Machine Learning, pp. 3518–3527, 2018.
 Hawkins (1971) Hawkins, D. M. On the bounds of the range of order statistics. Journal of the American Statistical Association, 66(335):644–645, 1971.
 Krizhevsky & Hinton (2009) Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. 2009.
 Lamport et al. (1982) Lamport, L., Shostak, R. E., and Pease, M. C. The byzantine generals problem. ACM Trans. Program. Lang. Syst., 4:382–401, 1982.
 Li et al. (2014a) Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J., and Su, B.Y. Scaling distributed machine learning with the parameter server. In OSDI, volume 14, pp. 583–598, 2014a.
 Li et al. (2014b) Li, M., Andersen, D. G., Smola, A. J., and Yu, K. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems, pp. 19–27, 2014b.
 Lynch (1996) Lynch, N. A. Distributed algorithms. Elsevier, 1996.
 McMahan et al. (2016) McMahan, H. B., Moore, E., Ramage, D., Hampson, S., et al. Communicationefficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.
 Su & Vaidya (2016a) Su, L. and Vaidya, N. H. Faulttolerant multiagent optimization: Optimal iterative distributed algorithms. In PODC, 2016a.
 Su & Vaidya (2016b) Su, L. and Vaidya, N. H. Defending nonbayesian learning against adversarial attacks. arXiv preprint arXiv:1606.08883, 2016b.
 Tanenbaum & Van Steen (2007) Tanenbaum, A. S. and Van Steen, M. Distributed systems: principles and paradigms. PrenticeHall, 2007.
 Xing et al. (2016) Xing, E. P., Ho, Q., Xie, P., and Wei, D. Strategies and principles of distributed machine learning on big data. Engineering, 2(2):179–195, 2016.
 Yin et al. (2018) Yin, D., Chen, Y., Ramchandran, K., and Bartlett, P. Byzantinerobust distributed learning: Towards optimal statistical rates. arXiv preprint arXiv:1803.01498, 2018.
Comments
There are no comments yet.