1. Introduction
Deep learning models have achieved impressive results in many areas of application. However, deep learning models remain vulnerable to adversarial attacks [SZS13]: small changes (imperceptible to the human eye) in the model input may lead to vastly different model predictions. In securitybased applications, this vulnerability is of utmost concern. For example, traffic signs may be modified with small stickers to cause misclassification, causing say a stop sign to be treated as speed limit sign [EEF18]
. Facial recognition systems can be easily spoofed using colourful glasses
[SBBR16].This security flaw has led to an arms race in the research community, between those who develop defences to adversarial attacks, and those working to overcome these defences with stronger adversarial attack methods [WK18, RSL18]. Notably, as the community develops stronger adversarial attack methods, claims of model robustness to adversarial attack are often proved to be premature [CW17, ACW18].
There are two approaches to demonstrating a model is resistant to adversarial attacks. The first is theoretical, via a provable lower bound on the minimum adversarial distance necessary to cause misclassification [WK18, RSL18, KBD17]. Theoretical lower bounds are often pessimistic: the gap between the theoretical lower bound and adversarial examples generated by stateoftheart attack algorithms can be large. Therefore, a second empirical approach is also used: an upper bound on the minimum adversarial distance is demonstrated through adversarial examples created by adversarial attacks [KGB16, MMS17, CW17, BRB18]. The motivation to design strong adversarial attacks is therefore twofold: on the one hand, to validate theoretical lower bounds on robustness; and on the other, to construct empirical upper bounds on the minimum adversarial distance. Ideally, the gap between the theoretical lower bound and the empirical upper bound should be small. As adversarial attacks become stronger, the gap narrows from above.
The process of finding an adversarial example with an adversarial attack is an optimization problem: find a small perturbation of the model input which causes misclassification. This optimization problem has been recast in various ways. Rather than directly enforcing misclassification, many adversarial attacks instead attempt to maximize the loss function. The Fast Gradient Signed Method (FGSM) was one of the first adversarial attacks to do so
[SZS13], measuring perturbation size in the norm. Iterative versions of FGSM were soon developed, where perturbations were measured in either the and norms [KGB16, MMS17, ZCR18]. These iterative methods perform Projected Gradient Descent (PGD), maximizing the loss function subject to a constraint enforcing small perturbation in the appropriate norm. Other works have studied sparse adversarial attacks, as in [PMJ16]. Rather than maximizing the loss, Carlini and Wagner [CW17] developed a strong adversarial attack by forcing misclassification to a predetermined target class. If only the decision of the model is available (but not the loss or the model gradients), adversarial examples can still be found using gradientfree optimization techniques [BRB18].In this paper, rather than using the loss as a proxy for misclassification, we design an adversarial attack that solves the adversarial optimization problem directly: minimize the size of the input perturbation subject to a misclassification constraint. Our method is gradientbased, but does not use the training loss function. The method is based on a sound, welldeveloped optimization technique, namely the logarithmic barrier method [NW06]. The logarithmic barrier is a simple and intuitive method designed specifically to enforce inequality constraints, which we leverage to enforce misclassification. We compare the LogBarrier attack against current benchmark adversarial attacks (using the Foolbox attack library [RBB17]), on several common datasets (MNIST [LC], CIFAR10 [Kri09], ImageNet1K [DDS09]) and models. On average, we show that the LogBarrier attack is comparable to current stateoftheart adversarial attacks. Moreover, we show that on challenging images (those that require larger perturbations for misclassification), the LogBarrier attack consistently outperforms other adversarial attacks. Indeed, we illustrate this point by attacking models trained to be adversarially robust, and show that the LogBarrier attack perturbs all images more efficiently than other attack methods: the LogBarrier attack is able to perturb all images using a much smaller perturbation size than that of other methods.
2. Background material
Adversarial examples arise in classification problems across multiple domains. The literature to date has been concerned primarily with adversarial examples in image classification: adversarial images appears no different (or only slightly so) from an image correctly classified by a model, but despite this similarity, are misclassified.
We let be the space of images. Typically, pixel values are scaled to be between 0 and 1, so that is the unit box . We let be the space of labels. If the images can be one of classes, is usually a subset of . Often
is the probability simplex, but not always. In this case, each element
of a label correspond to the probability an image is of class. Groundtruth labels are then onehot vectors.
A trained model, with fixed model weights , is a map . For brevity, in what follows we drop dependence on . For an input image , the model’s predicted classification is the of the model outputs. Given imagelabel pair , let be the index of the correct label (the of ). A model is correct if .
An adversarial image is a perturbation of the original image, , such that the model misclassifies:
(1) 
The perturbation must be small in a certain sense: it must be small enough that a human can still correctly classify the perturbed image. There are various metrics for measuring the size of the perturbation. A common choice is the norm (the maxnorm); others use the (Euclidean) norm. If perturbations must be sparse – for example, if the attacker can only modify a small portion of the total image – then the count of nonzero elements in may be used. Throughout this paper we let be a generic metric on the size of the perturbation . Typically is a specific norm, , such as the Euclidean or max norms. Thus the problem of finding an adversarial image may be cast as an optimization problem, minimize the size of the perturbation subject to model misclassification.
The misclassification constraint is difficult to enforce, so a popular alternative is to introduce a loss function . For example, could be the loss function used during model training. In this case the loss measures the ‘correctness’ of the model at an image . If the loss is large at a perturbed image , then it is hoped that the image is also misclassified. The loss function is then used as a proxy for misclassification, which gives rise to the following indirect method for finding adversarial examples:
(2)  
subject to 
maximize the loss subject to perturbations being smaller than a certain threshold. The optimization approach taken by (2) is by far the most popular method for finding adversarial examples. In one of the first papers on this topic, Szegedy et al [SZS13] proposed the Fast Signed Gradient Method (FGSM), where is the norm of the perturbation , and the solution to (2) is approximated by taking one step in the signed gradient direction. An iterative version with multiple steps, Iterative FGSM (IFGSM) was proposed in [KGB16], and remains the method of choice for adversarial attacks measured in . When perturbations are measured in , (2) is solved with Projected Gradient Descent (PGD) [MMS17].
Fewer works have studied the adversarial optimization problem directly, i.e., without a loss function. In a seminal work, Carlini and Wagner [CW17], developed a targeted attack, in which the adversarial distance is minimized subject to a targeted misclassification. In a targeted attack, not just any misclassification will do: the adversarial perturbation must induce misclassification to a prespecified target class. The CarliniWagner attack (CW) incorporates the targeted misclassification constraint as a penalty term into the objective function. The CW attack was able to overcome many adversarial defence methods that had been thought to be effective, and was an impetus for the adversarial research community’s search for rigorous, theoretical guarantees of adversarial robustness.
There is interest in gradientfree methods for finding adversarial examples. In this scenario, the attacker only has access to the classification of the model, but not the model itself (nor the model’s gradients). In [BRB18], Brendal et al directly minimize the adversarial distance while enforcing misclassification using a gradientfree method. Their Boundary attack iteratively alternates between minimizing the perturbation size, and projecting the perturbation onto the classification boundary. The projection step is approximated by locally sampling the model decision near the classification boundary.
3. The LogBarrier Attack
We tackle the problem of finding (untargeted) adversarial examples by directly solving the following optimization problem,
(3)  
subject to 
that is, minimize the adversarial distance subject to misclassification. We use the logarithmic barrier method [NW06] to enforce misclassification, as follows. We are given an imagelabel pair and correct label . Misclassification at an image occurs if there is at least one index of the model’s prediction with greater value than the prediction of the correct index:
(4) 
This is a necessary and sufficient condition for misclassification. Thus, we rewrite (3):
(5)  
subject to 
The barrier method is a standard tool in optimization for solving problems such as (5) with inequality constraints. A complete discussion of the method can be found in [NW06]. In the barrier method, inequality constraints are incorporated into the objective function via a penalty term, which is infinite if a constraint is violated. If a constraint is far from being active, then the penalty term should be small. The negative logarithm is an ideal choice:
(6) 
where we denote and . If the gap between is much larger than , the logarithmic barrier term is small. However, as this gap shrinks, the penalty term approaches infinity. Thus the penalty acts as a barrier, forcing an optimization algorithm to search for solutions where the constraint is inactive. If (6) is solved iteratively with smaller and smaller values of , in the limit as , the solution to the original problem (5) is recovered. (This argument can be made formal if desired, using convergence [Bra02].) See Figure 1, where the barrier function is plotted with decreasing values of . In the limit as , the barrier becomes 0 if the constraint is satisfied, and otherwise.
3.1. Algorithm description
We now give a precise description of our implementation of the log barrier method for generating adversarial images. The constraint can be viewed as a feasible set. Thus, the algorithm begins by finding an initial feasible image: the original image must be perturbed so that it is misclassified (not necessarily close to the original). There are several ways to find a misclassified image. A simple approach would be to take another natural image with a different label. However, we have found in practice that closer initial images are generated by randomly perturbing the original image with increasing levels of noise (e.g. Standard Normal or Bernoulli) until it is misclassified. After each random perturbation, the image is projected back onto the set of images in the box, via the projection . This process is briefly described in Algorithm 1. Note that if the original image is already misclassified, no random perturbation is performed, since the original image is already adversarial.
After an initial perturbation is found, we solve (6) for a fixed . Various optimization methods algorithms may be used to solve (6). For small to mediumscale problems, variants of Newton’s method are typically preferred. However, due to computational constraints, we chose to use gradient descent. After each gradient descent step, we check to ensure that the updated adversarial image remains in the box. If not, it is projected back into the set of images with the projection .
It is possible that a gradient descent step moves the adversarial image so that the image is correctly classified by the model. If this occurs, we simply backtrack along the line between the current iterate and the previous iterate, until we regain feasibility. To illustrate the backtracking procedure, let be the previous iterate, and be a candidate adversarial image which is now correctly classified. We continue backtracking the next iterate via
(7) 
until the iterate is misclassified. The hyperparameter is a backtracking parameter. The accumulation point of the above sequence is . As a result, this process is guaranteed to terminate, since the previous iterate is itself misclassified. This backtracking procedure is sometimes necessary when iterates are very close to the decision boundary. If the iterate is very close to the decision boundary, then the gradient of the log barrier term is very large, and dominates the update step. Since the constraint set is not necessarily convex or even fully connected, it is possible that the iterate could be sent far from the previous iterate without maintaining misclassification. We rarely experience this phenomenon in practice, but include the backtracking step as a safety. An alternate approach (which we did not implement), more aligned with traditional optimization techniques, would be instead to use a dynamic step size rule such as the ArmijoGoldstein condition [Arm66].
The gradient descent algorithm comprises a series of iterates in an inner loop. Recall that as , the log barrier problem approaches the original problem (5). Thus, we shrink by a certain factor and repeat the procedure again, iterating in a series of outer loops (of course, initializing now with previous iterate). As shrinks, the solutions to (6) approach the decision boundary. In each inner loop, if the iterates fail to move less than some threshold value , we move onto the next outer loop. The path taken by the iterates of the outer loop is called the central path, illustrated in Figure 2.
The LogBarrier attack pseudocode is presented in Algorithm 2. For brevity we write the log barrier . We remark that the LogBarrier attack can be improved by running the method several times, with different random initializations (although we do not implement this here).
The literature on adversarial perturbations primarily focuses on perturbations measured in the and norms. For perturbations measured in the norm, we set the distance measure to be the squared Euclidean norm, . When perturbations are measured in the norm, we do not use the maxnorm directly as a measure, due to the fact that the norm is nonsmooth with sparse subgradients. Instead, we use the following approximation of the norm [LZH14],
where . As the norm is recovered.
Algorithm hyperparameters
Like many optimization routines, the logarithmic barrier method has several hyperparameters. However, because our implementation is parallelized, we have found that the tuning process is relatively quick. For the attack, our default parameters are , and . For , we set with and ; the rest are the same as in the case.
For the initialization procedure, we have and . If attacking in
, we initialize using the Standard Normal distribution. Else, for
, we use the Bernoulli initialization with .Top5 misclassification
The LogBarrier attack may be generalized to enforce Top5 misclassification as well. In this case, the misclassification constraint is that , where now is the index of sorted model outputs. (In other words, , and is the secondlargest model output, and so forth.) We then set the barrier function to be . In this scenario, the LogBarrier attack is initialized with an image that is not classified in the Top5.
4. Experimental results
We compare the LogBarrier attack with current stateoftheart adversarial attacks on three benchmark datasets: MNIST [LC], CIFAR10 [Kri09], and ImageNet1K [DDS09]. On MNIST and CIFAR10, we attack 1000 randomly chosen images; on ImageNet1K we attack 500 randomly selected images, due to computational constraints. On ImageNet1K, we use the Top5 version of the LogBarrier attack.
All other attack methods are implemented using the adversarial attack library Foolbox [RBB17]. For adversarial attacks measured in , we compare the LogBarrier attack against Projected Gradient Descent (PGD) [MMS17], the CarliniWagner attack (CW) [CW17], and the Boundary attack (BA) [BRB18]. These three attacks all very strong, and consistently perform well in adversarial attack competitions. When measured in , we compare against IFGSM [KGB16], the current stateoftheart. We leave Foolbox hyperparameters to their defaults, except the number of iterations in the Boundary attack, which we set to a maximum of 5000 iterations.
4.1. Undefended networks
MNIST  CIFAR10  ImageNet1K  

AllCNN  ResNeXt34  
LogBarrier  1.29  
CW  1.27  1.59  
PGD  2.54  2.53  1.15  
BA  1.41  1.55  3.31 
norm. We report the mean and variance of the adversarial distance on a subsample of the test dataset. Lower values are better.
MNIST  CIFAR10  ImageNet1K  
AllCNN  ResNeXt34  
LogBarrier  
IFGSM 
We first study the LogBarrier attack on networks that have not been trained to be adversarially robust. For MNIST, we use the network described in [CW17, PMJ16]. On CIFAR10, we consider two networks: AllCNN [SDBR14], a shallow convolutional network; and a ResNeXt34 (2x32) [XGD17], a much deeper network residual network. Finally, for ImageNet1K, we use a pretrained ResNet50 [HZRS16]
available for download on the PyTorch website.
Tables 1 and 2 report the percentage misclassified, for each attack at a fixed perturbation size. A strong attack should have a high misclassification rate. In the tables, the perturbation size is chosen to agree with attack thresholds commonly reported in the adversarial literature. Measured in Euclidean norm, we see that the LogBarrier attack is the strongest on all datasets and models. Measured in the maxnorm, the LogBarrier outperforms IFGSM on all datasets and models, except on ImageNet1K where the difference is slight.
We also report the mean and variance of the adversarial attack distances, measured in and , in Tables 3 and 4 respectively. A strong adversarial attack should have a small mean adversarial distance, and a small variance. Small variance is necessary to ensure precision of the attack method. A strong attack method should be able to consistently find close adversarial examples. Table 3 demonstrates that, measured in , the LogBarrier attack is either the first ranked attack, or a close second. When measured in , the LogBarrier attack significantly outperforms IFGSM on all datasets and models, except ImageNet1K.
For illustration, we show examples of adversarial images from the IFGSM and LogBarrier attacks in Figure 3. On images where IFGSM requires a large distance to adversarially perturb, the LogBarrier attack produces visibly less distorted images.
4.2. Defended networks
In this section we turn to attacking adversarially defended networks. We first consider two defence strategies: gradient obfuscation [ACW18], and multistep adversarial training as described in Madry et al [MMS17]. We study these two strategies on the MNIST and ResNeXt34 networks used in Section 4.1. We limit ourselves to studying defence methods for attacks in the norm. Attacks are performed on the same 1000 randomly selected images as the previous section. Finally, we also test our attack on a MNIST model trained with Convex Adversarial Polytope [WK18] training, the current stateoftheart defence method on MNIST.
Gradient Obfuscation
Although discredited as a defence method [ACW18]
, gradient obfuscation is a hurdle any newly proposed adversarial attack method must be able to surmount. We implement gradient obfuscation by increasing the temperature on the softmax function computing model probabilities from model logits. As the softmax temperature increases, the size of the gradients of the model probabilities approaches zero, because the model probabilities approach onehot vectors. Although the decision boundary of the model does not change, many adversarial attack algorithms have difficulty generating adversarial examples when model gradients are small.
In Tables 5 and 6 we show that the LogBarrier attack easily overcomes gradient obfuscation, on both CIFAR10 and MNIST models. The reason that the LogBarrier method is able to overcome gradient obfuscation is simple: away from the decision boundary, the logarithmic barrier term is not active (indeed, it is nearly zero). Thus the LogBarrier algorithm focuses on minimizing the adversarial distance, until it is very close to the decision boundary, at which point the barrier term activates. In contrast, because IFGSM is a local method, if model gradients are small, it has a difficult time climbing the loss landscape, and is not able to generate adversarial images.
Adversarial Training
Adversarial training is a popular method for defending against adversarial attacks. We test the LogBarrier attack on networks trained with multistep adversarial training in the norm, as presented in Madry et al [MMS17]. Our results are shown in Tables 5 and 6. We also plot defence curves of the LogBarrier and IFGSM attacks on defended and undefended models in Figures 3(a) and 3(b), for respectively MNIST and CIFAR10.
On MNIST, we did not observe a reduction in test accuracy on clean images with adversarially trained models compared to undefended models. As expected, adversarial training hinders both LogBarrier and IFGSM from finding adversarial images at very small distances. However, we see that the LogBarrier attack is able to attack all images with nearly the same distance in both the defended and undefended models. In contrast, IFGSM requires a very large adversarial distance to attack all images on the defended model, as shown in Figure 3(a). That is, adversarial training does not significantly reduce the empirical distance required to perturb all images, when the LogBarrier attack is used. The point is illustrated in Table 5, where we report the distance required to perturb 90% of all images. The LogBarrier attack requires an adversarial distance of 0.22 on the undefended MNIST model, and 0.29 on the defended MNIST model, to perturb 90% of all images. In contrast, IFGSM requires a distance of 0.46 on the undefended model, but 0.65 on the defended model.
On CIFAR10, we observe the same behaviour, although the phenomenon is less pronounced. As shown in Table 6 and Figure 3(b), the LogBarrier attack requires a smaller adversarial distance to perturb all images than IFGSM. Notably, the LogBarrier attack on the defended network is able to attack all images with a smaller adversarial distance than even IFGSM on the undefended network.
Against the Convex Adversarial Polytope
Finally, we use the LogBarrier attack on a provable defence strategy, the Convex Adversarial Polytope [WK18]. The Convex Adversarial Polytope is a method for training a model to guarantee that no more than a certain percentage of images may be attacked at a given adversarial distance. We chose to attack the defended MNIST network in [WK18], which is guaranteed to have no more than 5.82% misclassification at perturbation size . We validated this theoretical guarantee with both the LogBarrier attack and IFGSM, and found that both methods were unable to perturb more than 3% of test images at distance 0.1.
5. Discussion
We have presented a new adversarial attack that uses a traditional method from the optimization literature, namely the logarithmic barrier method. The LogBarrier attack is effective in both the and norms. The LogBarrier attack directly solves the optimization problem posed by the very definition of adversarial images; i.e., find an image close to an original image, while being misclassified by a network. This is in contrast to many other adversarial attack problems (such as PGD or IFGSM), which attempt to maximize a loss function as a proxy to the true adversarial optimization problem. Whereas lossbased adversarial attacks start locally at or near the original image, the LogBarrier attack begins far from the original image. In this sense, the LogBarrier attack is similar in spirit to the Boundary attack [BRB18]: both the LogBarrier attack and the Boundary attack begin with a misclassified image, and iteratively move the image closer to the original image, while maintaining misclassification. The LogBarrier attack is a gradientbased attack: to enforce misclassification, gradients of the logarithmic barrier are required. In contrast, the Boundary attack is gradientfree, and uses rejection sampling to enforce misclassification. Although the LogBarrier attack uses gradients, we have shown that it is not impeded by gradient obfuscation, a common drawback to other gradientbased attacks. Because the LogBarrier attack is able to use gradients, it is typically faster than the Boundary attack.
The LogBarrier attack may be used as an effective tool to validate claims of adversarial robustness. We have shown that one strength of the LogBarrier attack is its ability to attack all images in a test set, using a fairly small maximum adversarial distance compared to other attacks. In other words, the LogBarrier attack estimates the mean adversarial distance with high precision. Using the LogBarrier attack, we have raised questions about the robustness of multistep adversarial training
[MMS17]. For instance, on MNIST, we showed that multistep adversarial training did not significantly improve the necessary distance required to perturb all test images, relative to an undefended model. For adversarially trained models on CIFAR10, we showed that the necessary distance to perturb all images is significantly smaller than the estimate provided by IFGSM. This is further motivation for the development of rigorous, theoretical guarantees of model robustness.References

[ACW18]
Anish Athalye, Nicholas Carlini, and David A. Wagner.
Obfuscated gradients give a false sense of security: Circumventing
defenses to adversarial examples.
In
Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 1015, 2018
, pages 274–283, 2018.  [Arm66] Larry Armijo. Minimization of functions having Lipschitz continuous first partial derivatives. Pacific Journal of Mathematics, 16(1):1–3, 1966.
 [Bra02] Andrea Braides. convergence for Beginners. Oxford University Press, 2002.
 [BRB18] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decisionbased adversarial attacks: Reliable attacks against blackbox machine learning models. In International Conference on Learning Representations, 2018.

[CW17]
Nicholas Carlini and David A. Wagner.
Towards evaluating the robustness of neural networks.
In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 2226, 2017, pages 39–57, 2017. 
[DDS09]
Jia Deng, Wei Dong, Richard Socher, LiJia Li, Kai Li, and FeiFei Li.
Imagenet: A largescale hierarchical image database.
In
2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 2025 June 2009, Miami, Florida, USA
, pages 248–255, 2009.  [EEF18] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physicalworld attacks on deep learning visual classification. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 1822, 2018, pages 1625–1634, 2018.
 [HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 2730, 2016, pages 770–778, 2016.
 [KBD17] Guy Katz, Clark W. Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In Computer Aided Verification  29th International Conference, CAV 2017, Heidelberg, Germany, July 2428, 2017, Proceedings, Part I, pages 97–117, 2017.
 [KGB16] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, abs/1607.02533, 2016.
 [Kri09] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

[LC]
Yann LeCun and Corinna Cortes.
The MNIST database of handwritten digits.
http://yann.lecun.com/exdb/mnist/.  [LZH14] Mandy Lange, Dietlind Zühlke, Olaf Holz, Thomas Villmann, and SaxoniaGermany Mittweida. Applications of lpnorms and their smooth approximations for gradient based learning vector quantization. In ESANN, 2014.
 [MMS17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. CoRR, abs/1706.06083, 2017.
 [NW06] Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
 [PMJ16] Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy, EuroS&P 2016, Saarbrücken, Germany, March 2124, 2016, pages 372–387, 2016.
 [RBB17] Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox v0.8.0: A python toolbox to benchmark the robustness of machine learning models. CoRR, abs/1707.04131, 2017.
 [RSL18] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. CoRR, abs/1801.09344, 2018.
 [SBBR16] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to a crime: Real and stealthy attacks on stateoftheart face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 2428, 2016, pages 1528–1540, 2016.
 [SDBR14] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014.
 [SZS13] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013.
 [WK18] Eric Wong and J. Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 1015, 2018, pages 5283–5292, 2018.
 [XGD17] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 2126, 2017, pages 5987–5995, 2017.
 [ZCR18] Tianhang Zheng, Changyou Chen, and Kui Ren. Distributionally adversarial attack. CoRR, abs/1808.05537, 2018.