Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation

03/10/2019 ∙ by Cong Xie, et al. ∙ 0

Recently, new defense techniques have been developed to tolerate Byzantine failures for distributed machine learning. The Byzantine model captures workers that behave arbitrarily, including malicious and compromised workers. In this paper, we break two prevailing Byzantine-tolerant techniques. Specifically we show robust aggregation methods for synchronous SGD -- coordinate-wise median and Krum -- can be broken using new attack strategies based on inner product manipulation. We prove our results theoretically, as well as show empirical validation.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The security of distributed machine learning has drawn increasing attention in recent years. Among the threat models, Byzantine failures (Lamport et al., 1982) are perhaps the most well-studied. In the Byzantine model, workers can behave arbitrarily and maliciously. In addition, Byzantine workers are omniscient and can conspire. Most of the existing Byzantine-tolerant machine-learning algorithms (Blanchard et al., 2017; Chen et al., 2017; Yin et al., 2018; Feng et al., 2014; Su & Vaidya, 2016a; b; Alistarh et al., 2018)

focus on the protection of distributed Stochastic Gradient Descent (SGD).

In this paper, we consider Byzantine-tolerant SGD in a server-worker architecture (also known as the parameter server architecture (Li et al., 2014a; b)), depicted in Figure 3

. The system is composed of server nodes and worker nodes. In each epoch, the workers pull the latest model from the servers, estimate the gradients using the locally sampled training data, and then push the gradient estimators to the servers. The servers aggregate the gradient estimators, and update the model by using the aggregated gradients.

Figure 1: Illustration of failed Byzantine-tolerant SGD. We execute distributed synchronous SGD on CIFAR-10 image classification, with 25 workers. Beginning from the 100th epoch, we attack the a system by replacing some workers with Byzantine workers. During the attack, 12 workers are Byzantine in the case of coordinate-wise median, and 11 workers are Byzantine in the case of Krum. The Byzantine workers push to the server, where is the true gradient.

We consider Byzantine failures at a subset of the worker nodes. Byzantine workers send arbitrary values instead of the gradient estimators to the server. Such Byzantine gradients are potentially adversarial, and this can result in convergence to sub-optimum models, or even lead to divergence. To make things worse, the Byzantine workers can spy on the information at any server or at any honest worker (omniscience). Byzantine gradients can thus be tailored to have similar variance and magnitude as the correct gradients, which makes them hard to distinguish. Additionally, in different iterations, different subsets of workers can behave in a Byzantine manner, evading detection. Existing literature assumes that less than half of the workers are Byzantine in any iteration.

Compared to traditional Byzantine tolerance in distributed systems (Lynch, 1996; Avizienis et al., 2004; Tanenbaum & Van Steen, 2007; Fischer et al., 1982), Byzantine tolerance in distributed machine learning has unique properties and challenges. Traditional Byzantine tolerance attempts to reach consensus on correct values. However, machine learning algorithms do not need to reach consensus. Further, even non-Byzantine-tolerant machine learning algorithms can naturally tolerate some noise in the input and during execution (Xing et al., 2016). Thus for distributed SGD, existing techniques for Byzantine-tolerant execution guarantee an upper-bound on the distance between the aggregated approximate gradient (under Byzantine workers) and the true gradient (Blanchard et al., 2017; Yin et al., 2018).

A deeper introspection reveals, however, that what really matters for gradient descent algorithms is the direction of the descent. As shown in Figure 2

, to let the gradient descent algorithm make progress, we need to guarantee that the direction of the aggregated vector agrees with the true gradient, i.e., the inner product between the aggregated vector and the true gradient must be non-negative. This can be violated by an attack that makes the inner product negative. We call this class of new attacks “inner product manipulation attacks”.

Figure 2: Descent Direction. The blue arrows represent the directions which agree with the steepest descent direction (negative gradient). The red arrows represent directions which agree with the steepest ascent direction (gradient).

We observe that the bounded distance between the aggregated value and the true gradient guaranteed by existing techniques is not enough to defend distributed synchronous SGD against inner product manipulation attacks. For example, for the coordinate-wise median, if we put all the Byzantine values on the opposite side of the true gradient, the inner product between the aggregated vector and the true gradient can be manipulated to be negative.

In this paper, we study how inner product manipulation makes Byzantine-tolerant SGD vulnerable. We conduct case studies on coordinate-wise median (Yin et al., 2018) and Krum (Blanchard et al., 2017). Figure 1 gives a glimpse of how bad the effect of the attack can be. In a nutshell, creating gradients in the opposite direction with large magnitude crashes coordinate-wise median, while creating gradients in the opposite direction with small magnitude crashes Krum. We provide theoretical analysis as well as empirical results to validate these findings.

Based on these results, we argue that there is a need to revise the definition of Byzantine tolerance in distributed SGD. We provide a new definition, and study its satisfaction on two prevailing robust distributed SGD algorithms, theoretically and empirically. In summary, our contributions are:

  • We break two prevailing Byzantine tolerant SGD algorithms – coordinate-wise median (Yin et al., 2018) and Krum (Blanchard et al., 2017) – using a new class of attacks called inner production manipulation attacks. We theoretically prove that under certain conditions, we can backdoor these two algorithms, even when the assumptions and theorems presented in these papers are valid.

  • We show how to design Byzantine gradients to compromise the robust aggregation rules. We conduct experiments to validate further.

  • Following our theoretical and empirical analysis, we propose a revised definition of Byzantine tolerance for distributed SGD.

2 Related Work

Robust estimators such as the median are well studied, and can naturally be applied to Byzantine tolerance. Coordinate-wise median is one approach that generalizes the median to high-dimensional vectors. In Yin et al. (2018), statistical error rates are studied for using coordinate-wise median in distributed SGD.

Blanchard et al. (2017) propose Krum, which is not based on robust statistics. For each candidate gradient, Krum computes the local sum of squared Euclidean distances to the other candidates, and outputs the one with minimal sum.

In this paper, we focus on coordinate-wise median and Krum. There are other Byzantine-tolerant SGD algorithms. For example, Bulyan (Guerraoui et al., 2018) is built based on Krum, which potentially shares the same flaws. DRACO (Chen et al., 2018) uses coding theory to ensure robustness, and is different from the other Byzantine-tolerant SGD algorithms.

Recently, an increasing number of papers propose attack mechanisms to break the defense of machine learning in various scenarios. For example, Athalye et al. (2018) propose attack techniques using adversarial training data. Bhagoji et al. (2018); Bagdasaryan et al. (2018) break the defense of federated learning (McMahan et al., 2016). In this paper, we focus on attacking distributed synchronous SGD using adversarial gradients sent by Byzantine workers.

3 Preliminaries

In this paper, we focus on distributed synchronous Stochastic Gradient Descent (SGD) with Parameter Server (PS). In this section, we formally introduce distributed synchronous SGD and the threat model of Byzantine failures.

Figure 3: Worker-Server Architecture

3.1 Stochastic Gradient Descent

We consider the following optimization problem:

where is a differentiable function, is sampled from some unknown distribution , is the number of dimensions. We assume that there exists at least one minimizer of , which is denoted by , where .

This problem is solved in a distributed manner with workers. In each iteration, each worker will sample independent and identically distributed (i.i.d.) data points from the distribution , and compute the gradient of the local empirical loss , where is the th sampled data on the th worker. The servers will collect and aggregate the gradients sent by the workers, and update the model as follows:

where is an aggregation rule (e.g., averaging), and is the gradient sent by the th worker, is the learning rate in the iteration. For an honest worker,

is an unbiased estimator such that

. When all the workers are honest, the most common choice of the aggregation rule is averaging:

The detailed algorithm of distributed synchronous SGD with aggregation rule is shown in Algorithm 1.

  Server
   {Initialization}
  for  do
     Broadcast to all the workers
     Wait until all the gradients arrive
     Compute
     Update the parameter
  end for
  Worker
  for  do
     Receive from the server
     Draw the samples, compute, and send the gradient to the server
  end for
Algorithm 1 Distributed Synchronous SGD with Robust Aggregation

3.2 Threat Model

In the Byzantine failure model, the gradients sent by malicious workers can take an arbitrary value:

(1)

where “” represents arbitrary values.

Formally, we define the threat model of Byzantine failure as follows.

Definition 1.

(Threat Model (Blanchard et al., 2017; Chen et al., 2017; Yin et al., 2018)). In the iteration, let be i.i.d. random vectors in , where . The set of correct vectors is partially replaced by arbitrary vectors, which results in , as defined in Equation (1). In other words, a correct gradient is , while a Byzantine gradient, marked as “”, is assigned arbitrary value. We assume that out of vectors are Byzantine, where . Furthermore, the indices of faulty workers can change across different iterations. If the failures are caused by attackers, the threat model includes the case where the attackers can collude.

The notations used in this paper is summarized in Table 1.

Notation Description
Number of workers
Minibatch size on each worker
Number of iterations
Set of integers
Number of Byzantine workers
Learning rate
Model parameters
Gradient sent by the th worker
in the iteration, potentially Byzantine
Correct gradient produced by the th worker
in the iteration
All the norms in this paper are -norms
Inner product between and
Table 1: Notations

4 Defense Techniques

In this section, we introduce two prevailing robust aggregation rules against Byzantine failures in distributed synchronous SGD: coordinate-wise median and Krum. For the remainder of this paper, we ignore the iteration superscript in and for convenience.

4.1 Coordinate-Wise Median

Definition 2.

(Coordinate-wise Median (Yin et al., 2018)) We define the coordinate-wise median aggregation rule as

where for any , the th dimension of is , is the th dimension of the vector , is the one-dimensional median.

4.2 Krum

Definition 3.

(Krum (Blanchard et al., 2017))

where are the indices of the nearest neighbours of in as measured by squared Euclidean distance.

For convenience, we refer to the coordinate-wise median and Krum as Median and Krum.

5 Attack Techniques

In this section, we revise the definition of Byzantine tolerance in distributed synchronous SGD. Then, we theoretically analyze the Byzantine tolerance of coordinate-wise median and Krum, and show that under certain conditions, these two robust aggregation rules are no longer Byzantine-tolerant.

5.1 Inner Product Manipulation

In the previous work on Byzantine-tolerant SGD algorithms, most of the robust aggregation rules only guarantee that the robust estimator is not arbitrarily far away from the mean of the correct gradients. In other words, the distance between the robust estimator and the correct mean is upper-bounded. However, for gradient descent algorithms, to guarantee the descent of the loss, the inner product between the true gradient and the robust estimator must be non-negative:

so that at least the loss will not increase in expectation. In particular, bounded distance is not enough to guarantee robustness, if the attackers manipulate the Byzantine gradients and make the inner product negative.

The intuition underlying the inner product manipulation attack is that, when gradient descent algorithm converges, the gradient approaches . Thus, even if the distance between the robust estimator and the correct mean is bounded, it is still possible to manipulate their inner product to be negative, especially when the upper bound of such distance is large.

We formally define a revised version of Byzantine tolerance for distributed synchronous SGD (DSSGD-Byzantine tolerance):

Definition 4.

(DSSGD-Byzantine Tolerance) Without loss of generality, suppose that in a specific iteration, the server receives correct gradients and Byzantine gradients . We assume that the correct gradients have the same expectation . An aggregation rule is said to be DSSGD-Byzantine-tolerant if

With the revised definition, now we theoretically analyze the DSSGD-Byzantine tolerance of coordinate-wise median and Krum.

Remark 1.

Note that we do not argue that the theoretical guarantees in the previous work are wrong. Instead, our claim is that the theoretical guarantees on the bounded distances are not enough to secure distributed synchronous SGD. In particular, DSSGD-Byzantine tolerance is different from the Byzantine tolerance proposed in previous work.

5.2 Coordinate-Wise Median

The following theorem shows that under certain conditions, Median is not DSSGD-Byzantine-tolerant.

Theorem 1.

We consider the worst case where . The server receives correct gradients and Byzantine gradients . We assume that the stochastic gradients have identical expectation , and non-zero coordinate-wise variance , where is the th coordinate of , and is the th coordinate of . When , there exist Byzantine gradients such that

Proof.

(sketch) Since median is independently taken in each coordinate, it is sufficient to prove Byzantine vulnerability for one coordinate or scalars. Thus, for convenience, with a little bit abuse of notation, we suppose that the correct gradients and Byzantine gradients are all scalars. We only need to show that under certain attacks, the aggregated value has a different sign than .

Without loss of generality, we assume that (the mirror case can be easily proved with a similar procedure). The Byzantine gradients are all assigned negative value: . Furthermore, we make the Byzantine gradients small enough such that .

By sorting the correct gradients, we can define the sequence , where is the th smallest element in :

We also define the expectation of the th smallest element: .

Then, it is easy to check that , and .

Using Theorem 1(b) from Hawkins (1971) (equiv. 9(a) from Arnold et al. (1979)), we have

Thus, when , is negative.

Remark 2.

When gradient descent converges, the expectation of the gradient approaches . Furthermore, since the gradient produced by the correct workers are stochastic, the variance always exists. Thus, eventually, the condition will be satisfied. To make things worse, the closer SGD approaches a critical point, the less likely the coordinate-wise median is DSSGD-Byzantine-tolerant.

Remark 3.

The proof of Theorem 1 provides the intuition of constructing adversarial gradients for the attackers. In practice, in each coordinate, the attackers only need to guarantee that all the Byzantine values are much smaller than the smallest correct value if the correct expectation is positive, or much larger than the largest correct value if the correct expectation is negative. Then, hopefully, if the variance is large enough, the smallest/largest value has the opposite sign to the correct expectation. Then, the attackers can successfully manipulate the aggregated value into the opposite direction to the correct expectation.

5.2.1 Toy Example

We provide an 1-dimensional toy example to illustrate how easily Median can fail. Suppose there are correct gradients with the mean , and Byzantine gradient with the negative mean . According to Definition 2, it is easy to check that , which means that Median produces a value with the opposite sign of the mean of the correct gradients.

5.3 Krum

The following theorem proves that under certain conditions, Krum is not DSSGD-Byzantine-tolerant. Note that Krum requires that .

Theorem 2.

We consider the worst case where . The server receives correct gradients and Byzantine gradients . We assume that the stochastic gradients have identical expectation . We define the mean of the correct gradients . We assume that the correct gradients are bounded by . Furthermore, we assume that , and such that . We take , where is a small positive constant value such that . When is large enough: , we have

Proof.

(sketch) For , , where .

Since any is identical, the nearest neighbours of must belong to . The remaining nearest neighbours must belong to the set of correct gradients . Thus, we have

For the correct gradients , there are two cases:

  • Case 1: There are some which belong to the nearest neighbours of .

    Suppose there are nearest neighbours in and nearest neighbours in , where . Since the correct gradients are bounded by , it is easy to check that . Thus, we have

  • Case 2: There are no which belong to the nearest neighbours of . Thus, we have

In both cases, we have Thus, when is large enough: , we have

As a result, . Thus, .

Remark 4.

In the theorem above, we assume that all the correct gradients are inside a Euclidean ball centered at their mean: . Such assumption can not always be satisfied, but it is reasonable that the random samples are sometimes inside such a Euclidean ball, if the variance is not too large. On the other hand, we assume that the pair-wise distances between the correct gradients are lower-bounded by . Almost surely, such exists, no matter how small it is. Note that the Byzantine attackers are supposed to be omniscient. Thus, the attackers can spy on the honest workers, and obtain and . Then, the attackers can choose an such that . Finally, we only need the number of workers to be large enough, so that .

Remark 5.

The proof of Theorem 2 provides the intuition of constructing adversarial gradients for the attackers. In practice, the attackers only need to assign to all the Byzantine gradients, with an small enough.

Remark 6.

Note that in Blanchard et al. (2017), Krum requires the assumption that for convergence, where is a general constant, is the maximal variance of the gradients, and is the gradient in expectation. Note that when SGD converges to a critical point. Thus, such an assumption is never guaranteed to be satisfied, if the variance is non-zero. Furthermore, the better SGD converges, the less likely such an assumption can be satisfied.

5.3.1 Toy Example

Note that the assumptions made in Theorem 2 are sufficient but not necessary conditions of the DSSGD-Byzantine vulnerability of Krum. In practice, it can be easier to find an that crashes Krum, especially for 1-dimensional cases.

We provide an 1-dimensional toy example to show how easily Krum can fail. Suppose there are correct gradients with the mean , and Byzantine gradient with the negative mean . According to Definition 3, the corresponding function values of are . Thus, , which means that Krum chooses the Byzantine gradient with the opposite sign of the mean of the correct gradients.

(a) Top-1 Accuracy on Testing Set,
(b) Cross Entropy on Training Set,
(c) Top-1 Accuracy on Testing Set,
(d) Cross Entropy on Training Set,
(e) Top-1 Accuracy on Testing Set,
(f) Cross Entropy on Training Set,
(g) Top-1 Accuracy on Testing Set,
(h) Cross Entropy on Training Set,
Figure 4: Convergence on training set, using Median as aggregation rule. .
(a) Top-1 Accuracy on Testing Set,
(b) Cross Entropy on Training Set,
(c) Top-1 Accuracy on Testing Set,
(d) Cross Entropy on Training Set,
(e) Top-1 Accuracy on Testing Set,
(f) Cross Entropy on Training Set,
(g) Top-1 Accuracy on Testing Set,
(h) Cross Entropy on Training Set,
Figure 5: Convergence on training set, using Krum as aggregation rule. .

6 Case Study

In this section, we implement special attack strategies for Median and Krum, and evaluate our attack strategies on a real-world application. The attack strategies are designed by using the intuitions underlying Theorem 1 and Theorem 2, which are mentioned in Remark 3 and Remark 5.

6.1 Datasets and Evaluation Metrics

We conduct experiments on the benchmark CIFAR-10 image classification dataset (Krizhevsky & Hinton, 2009)

, which is composed of 50k images for training and 10k images for testing. We use a convolutional neural network (CNN) with 4 convolutional layers followed by 1 fully connected layer. The detailed network architecture can be found in the appendix. For any worker, the minibatch size for SGD is

.

In each experiment, we launch 25 worker processes. We repeat each experiment 10 times and take the average. We use top-1 accuracy on the testing set and the cross-entropy loss function on the training set as the evaluation metrics.

We use the averaging, Median, and Krum without attacks as the gold standard, which are referred to as Mean without attack, Median without attack, and Krum without attack. We start the attack at different epochs, so that SGD can warm up and make some progress first. We include some additional experiments in the appendix.

6.2 Median

In each iteration, the server receives gradients. A randomly selected subset of correct gradients are replaced by Byzantine gradients. We define the set of Byzantine gradients as , and the set of the remaining correct gradients as . Our attack strategy is as follows:

According to Theorem 1 and Remark 3, Median is vulnerable to positive with large magnitude .

We test the above attack strategy with different . The results are shown in Figure 4. Median fails when . When , Median gets stuck and stops making progress. When , Median successfully defends against the attack.

6.3 Krum

In each iteration, the server receives gradients. A randomly selected subset of correct gradients are replaced by Byzantine gradients. We define the set of Byzantine gradients as , and the set of the remaining correct gradients as . Our attack strategy is as follows:

According to Theorem 2 and Remark 5, Krum is vulnerable to positive with small magnitude .

We test the above attack strategy with different . The results are shown in Figure 5. Krum fails when is small. When is large enough, Krum successfully defends against the attack.

6.4 Discussion

Surprisingly, both Median and Krum are more vulnerable than we expected. Note that our theorems only analyze the worst cases. There are other cases where Median and Krum can fail.

For Median, even if we take , SGD still performs badly. Theoretically, even if we do not use positive , small can still enlarge the variance of SGD, which can be potentially harmful to the convergence. We can see that with large negative , the defense of Median is successful. In our experiment, we reveal certain new vulnerabilities of Median in distributed synchronous SGD. The experiments conducted by Yin et al. (2018) do not fail because the attacker only changes the labels of the poisoned training data by flipping a to . It is very likely that such an attack produces Byzantine gradients surrounding the correct gradients coordinate-wisely on both sides. However, according to Theorem 1 and Remark 3, an effective attack should place the Byzantine gradient on one and only one side of the correct gradients, which is the side opposite to the mean of the correct gradients, coordinate-wisely.

For Krum, small positive makes SGD vulnerable. Furthermore, even if we take , Krum still fails. In our experiment, we reveal certain new vulnerability of Krum in distributed synchronous SGD. The experiments conducted by Blanchard et al. (2017) do not fail even though a similar attack strategy called “omniscient” is conducted. The reason is that, in the paper of Blanchard et al. (2017), the attacker “proposes the opposite vector, scaled to a large length”, which is similar to our attack strategy with a large .

Guided by our theoretical analysis, we designed efficient attack strategies for both Median and Krum. Our results show that the definition of Byzantine tolerance for distributed synchronous SGD should be revised. Using our definition of DSSGD-Byzantine tolerance, research can be conducted to design better defense techniques.

7 Conclusion

We propose a revised definition of Byzantine tolerance for distributed synchronous SGD. With the new definition, we theoretically and empirically examine the Byzantine tolerance of two prevailing robust aggregation rules. Guided by our theoretical analysis, attack techniques can be designed to fail the aggregation rules. In the future, we hope new defense techniques can be designed using our revised definition of Byzantine tolerance.

References