1 Introduction
The key difference between machine learning and pure optimization is that, machine learning algorithm is expected to have the ability to generalize from the given (training) dataset to the unseen (test) dataset. Though there can be a variety of the ways how the data are collected and/or annotated, one underly assumption is shared by most of the scenarios, that there is always a stationary stochastic process that generates each and every sample point in both of the dataset. In other word, both samples in either datasets follow the independent and identical distribution (i.i.d) promise. But in many realworld settings, this assumption maybe oversimplified.
Early researches in the machine learning community realize this issue when they try to measure and improve the robustness of the classifiers, in the sense that it should ties it best to be immune to the anomaly points in the training data or the test data. Studies on this topic can trace back to Wald’s robustness of statistical decision theory [1]. Researches fall in this category commonly use a term “minimax” robustness with somehow similar intuition to the modern “adversarial” description of the task. For instance, a research conducted by Lanckriet et al. [2]
defines a task of finding the optimal worstcase (thus minimax) misclassification probability on a binary classification problem on all possible choices of classconditional densities, but with a fixed known the mean and covariance matrix, and is then reduced to a convex second order cone optimization problem.
Another storyline starts from the information security area, which the minimax games are ubiquitous. For instance, when a security controller designed to prevent the online service from malicious requests is ensured by a binary classifier, the requests generated by the malicious users will mostly likely not follow the “stationary” assumption. Anomalies of this type may occur only in the test dataset, commonly not due to the “rare events”, but attribute to the intentional attacks on the system launched by an “adversarial” party. Besides the adversarial during test time, research by Biggio et al. also show that a few wellselected training instances may cause severe poisoning attacks against Support Vector Machines (SVM), together with a new principled defense method highly resilient against all poisoning attacks
[3]. More recently, Jagielski et al. studies exclusively on the poisoning attack on linear regression, and propose a theoreticallygrounded optimization framework
[4]. In sensitive scenarios such as malware classification [5], the existence of adversarial examples has put forward great challenges on the applicability of machine learning based approaches [6, 5]: The system security can be severely compromised, if an adversary is able to make slight modifications on a malware file to make it remain a malware, but is thus misclassified as benign by the classifier. These potential security breaches may become even more sensitive to safetycritical tasks such as an airborne collision avoidance system for unmanned aircraft [7] or selfdriving car [8].The specialness within the term “adversarial machine learning” is the involvement of two “adversarial” parties within an minimax game, in which the attacker side learns to produce the adversarial examples to trick the defender (i.e., the classifier induced by a machine learning algorithm), while the classifier tries to defend attacks of this kind. Lowd and Meek [9] were among the first who employ the term “adversarial”, and introduce the concept of “adversarial learning” to refer to a learning problem of Adversarial Classifier Reverse Engineering (ACRE), in which the algorithms aims to collect sufficient information about a classifier to construct adversarial attacks.
In contrary to Lowd and Meek’s task definition, Huang et al., [10] use the term “adversarial machine learning”, which refers to the study of effective machine learning techniques against an adversarial opponent. Although the task definition of the two seem to be rather opposite, their ultimate goals are however coincident. From the point view of the attack side, adversarial
means one needs to base the attack activity on the knowledge or heuristics of the defense (classifier) side instead of random moves, while the defense side also need to improve the robustness on the “worst cases” instead of i.i.d inputs. Hereby, this paper serves to present a comprehensive review on the literatures over the adversarial problem in machine learning, mainly focuses on the attacking and defending strategies in deep neural networks.
2 Backgrounds
2.1 The “Imperceivable” Perturbation
In Section 1, we introduced the concept of “adversarial learning” as a minimax game[11], in which the attacker aims to maximize its likelihood to trick the classifier, while the classifier aims to minimize its risk in such worst case. There are many ways to launch attacks on a machine learning based system. One could inject malicious data points into the training data, to manipulate the training procedure, or one can use the model information (e.g., the gradient w.r.t the input) to construct the adversarial. Attackers of any types above, together with its counter party naturally involves a pair of “adversarial” parties, and thus can be categorized into the “adversarial machine learning”.
However, the recent researches in the deep learning community impose more restriction on the definition of “adversarial”. It commonly refers to a process of producing “small and imperceptible perturbations” (or distortions in some literature) to human to the normal example, which however, can fool the machine learning algorithms. Formally speaking, a classification function
maps the input realvalued vectors to a discrete label set. For deep neural networks, the classifier’s output is commonly based on a probability distribution produced by
that gives a probability distribution on the class labels by a final softmax layer. So, for a given input instance
, the output class of a neural network can be written as . Thus, the goal of the attacker is to find the adversarial input that is close to such that .The parameters of the network
is then optimized by minimizing the expectation of a loss function
, where outputs the vector form of a categorical distribution , such that , and is the expected output class label for the input , commonly represented as a vector form of the discrete Dirac distribution , such that(1) 
For classification tasks, the loss function usually appears in the form of cross entropy
, which is equivalent to the KullbackLeibler divergence of
from for a fixed .The origin of the study on adversarial in deep learning may trace back to the profound discovery made by Szegedy et al. [12]
, pointing out that in many of the deep learning based computer vision systems, a imperceptibly small perturbation on the input image may cause a total and seemingly uninterpretable failure in the classification result. This discovery enforces the researchers to rethink the question of “What went wrong?”, despite the great success of the “superhuman” performance in many of the computer vision tasks using deep Convolutional Neural Networks (CNNs). As such, there are several followup questions:

Why these perturbations on the input are imperceptible to human but have large impact on systems with “superhuman” performance?

How can adversarial examples of this type can be constructed?

How do we prevent from the classifiers to make mistakes on these adversarial examples?
2.2 Measuring the Perturbation
2.2.1 The Metric
To have a more solid foundation to study this problem, we need a more precise way to measure the perturbation. Although the ways to define “imperceptible” are task dependent, most of which are based on the norm distance. The norm distance between two points and in a highdimension space can be written as , where the operator pnorm on an dimensional vector is defined as
(2) 
The value of is parameter that may vary due to the need of a specific task. When , the distance between two points and (written as ) is simply measures the Euclidean distance between and , which turns out to be a default choice in many early researches. When , the distance is equivalent to the sum of the absolute value of each dimension, which is also known as the Manhattan distance. Two special value of is and : The distance between two points measures the number of dimensions that have different values , and the distance is a conventional way to denote the maximum absolute difference among all dimensions , where is the indicator function, and denotes the value of the ith dimension of .
When the norm is used, the distance between two points will accumulate along the each of the dimensions, making the number of dimensions of the input data can lead to significant difference concerning the “imperceptible”. A simple way to remove the bias due to the dimension number of the input (e.g., the number of pixels of the input image) is by considering the average distortion on a perdimension basis by , where
is the number of input dimensions. However, this measurement may underestimate the distortion if
is large. Therefore, in the computer vision scenarios, experiment may often conducted on images of different sizes or resolutions, therefore, is often a preferred choice, which simply measure the maximum change of any of the pixel w.r.t the original input, thus invariable to the image resolution. That is, the distortion for any of the pixel has to be under a fixed bound.The actual choice of the distortion measurement can be taskdependent. For example, naive Bayesian classifier is a common choice by many of the natural language processing tasks, which uses boolean features (i.e., if a certain word occurs or not, e.g., as appeared in Lowd and Meek
[13]), making other measurements may seem more plausible. An alternative choice that show interesting result is the “one pixel attack” proposed by Su et al. [14], or a few pixels proposed by Narodytska and Kasiviswanathan [15] , in which the constrains is imposed on the number of pixels that is modified and is equivalent to the norm. One major reason for choosing the measurement by Goodfellow et al. [16] is because they consider the success of the adversarial attacks is due to the accumulated perturbation in a large number of dimensions. But since the metric has constraint the number of dimensions to be modified, their proposed explanation is then proved to be unnecessary.2.2.2 Other Metrics
Ideally, the reported amount of perturbation due to some measurement should reflect the perceptual impact it has on human’s cognition. So in Lowd and Meek’s framework [9], they introduce cost function as a more general form to measure the perturbation, by considering its negative impact on its attacking goal. Opposite to today’s common setting, the construction of the adversarial input in their work starts from the malicious sample, which is then modified to evade being detected by the classifier. Thus, two spam emails may both trick a classifier to be a normal mail though, one of them may be effective for its goal, while the other may lose too much of its original. A more recent version is given by Madry et al.[17], which formalize it as a minimax game, in which an unified cost is defined, and the attacker’s goal is to maximize the cost, while the classifier would need to minimize the cost. This point of view can be traced back to the early theoretical work on statistical optimization problem by Wald [1].
Although these more generic definitions on the perturbation metrics, the norm distance provides an simple and elegant way to calculate this quantity, making it more practical to be embedded inside an optimization problem,. However, one potential issue with the norm based perturbation measurement is that it is commonly applied directly on the input space , the distance in which may not reflect the affect at the perceptual level. This lead to the idea of measuring the distance between the two points after mapped into the latent space from both the attacking side and the defending side. We will elaborate strategies fall inside this type in Section 3.3.4 and 4.3.4.
2.3 Nondeeplearning Adversarial
Early research that is similar to the modern concept of “adversarial” can be traced back to the work by Dalvi et al. [18]
. There have been attempts to build attacks on security systems that built on machine learning algorithms though, they notice that previously approaches to address this problem is by repeated, manual, ad hoc reconstruction of the classifier, which motivates them they develop a formal framework and algorithms for this problem. Similar to the study on the robustness of statistic, they model the classification scene in a security system as a “minimax game” between the classifier and the adversary, in which the classifier should be optimized in a way that has the best performance given the adversary’s optimal strategy. Their experiment is exclusively focused on building adversaryaware Naive Bayes classifier
[19] applied on to be made spam detection system. Soon after, a similar idea is presented by Lowd and Meek, who propose the good word attack [13] that a spammer can transform spam message into legitimate email by simply inserting or appending “good words”. Wang et al. [20] recently propose a new type of attacking task, namely the hyperparameter stealing attack task, which aims to identify the hyperparameter , which balances the training data optimization and the overfitting penalty.One important class of approaches to solving this problem is suggested by Globerson et al. [21], who argues that the key factor to prevent classifier from such plight is to avoid assigning too much weight to any single input feature. The same idea is shared by many of the followup researches. For instance, Zhang et al. [22]
argue that the common practice of featurereduction process adopted by most machine learning systems turns out to be hazardous for classifier’s robustness, and proposed an adversaryaware feature selection method to improve classifier security against evasion attacks. Following the same intuition, Ning et al.
[23]suggest that using balanced decision tree to build random forests can significantly improve the robustness of the ensemble classifier.
A recent study by Papernot et al. show that the transferability of adversarial example may even exist among different learning algorithms [24], despite the fact that they have different assumptions and intuitions in nature (e.g., decision tree, support vector machine, or deep neural networks, etc.), which gives rise to one type of blackbox attack, and will be discussed in Section 3.3.
2.4 Derived Topics
2.4.1 Generative adversarial networks
So far, our discussion on the adversarial learning focuses on a class of machine learning algorithms called discriminative model, whereas the other side of it is what known as the generative model. A discriminative model aims to predict the label of some latent variable based on the observed variable , also known as the classification task, commonly approached by modeling the conditional probability of . Comparatively, a generative model is a model that can generate new sample points from the distribution . Though originated from the discriminative model, the research on the minimax game between the two adversarial parties lead to one of the most important advancement in learning generative models, namely the Generative Adversarial Networks (GANs) [25].
In the setting of adversarial machine learning, the two adversarial parties happen to fit the two classes of the machine learning algorithms: the defending side aims to correctly classify all input examples, regardless of whether it is a naturally occurred or adversarially manipulated; on the other hand, the attacking side aims to generate new examples that can lead to the mistakes made by the classifier. In adversarial learning, the ultimate goal is to improve the robustness of the classifier by putting it under the strongest adversarial generator. Contrarily, the ultimate goal of the GAN is to find the optimal generator, whose output can not be distinguished from the normal data even by the strongest discriminator.
In GAN, a generator parameterized by , aiming to fake realistic samples that cannot be distinguished from real ones, and a discriminator parameterized by , aiming to detect if an input sample is a genuine one or faked by the generator , form a pair of adversarial parties. The basic version of this minimax game is a zerosum game, in which the objective function of the generator and the objective function of the discriminator add up to . Under this setting, the game can be expressed in a single value function for the discriminator:
(3) 
in which the generator produces faked samples from the latent codes , and the discriminator gives the probability of an observed sample being genuine. Since the goal of the minimax game is to find the optimal generator which can draw new example from the distribution of , the optimization objective can be written as
(4) 
The original proposed objective function for the discriminator is as follows:
(5) 
Following Equation 4, we can derive the objective function for the generator as
(6) 
by dropping the constant term in the with respective to .
The huge success of GAN in synthesizing realistic images from a low dimension input draws exponential growth of research focus. Soon, the original form the GAN is found to suffer several problems, most importantly:

The training process can be extremely unstable, and often fail to converge at all;

The quality of the update gradient received by the generator drops as the discriminator improves.
Many of the extended researches are devoted to how to stabilize the training process by studying different possibilities of the objective functions. One of the most important improvements over the GAN since its invention is the Wasserstein GAN [26, 27], which uses the Wasserstein distance (a.k.a., the earth moving distance) to replace the original KullbackLeibler divergence based distance. Currently, a large portion of the stateofart research on deep learning is moving towards the GAN based approaches [28].
Although discriminative model and generative model are the two faces of machine learning, they are not parallel to each other. For instance in GAN, a discriminative model is introduced to find the highquality update gradient for optimizing the generative model. It is also true for adversarial machine learning, in which a generator that captures the data distribution well can be useful to improve the robustness of the classifier, as we will discuss in Section 4.3. Moreover, the GAN model can be directly involved in both the attacking side and the defending side.
2.4.2 Adversarial Reprogramming
Deep learning is a type of representation learning, whose goal is to find the best representation transformations via a hierarchy of levels, such that the data in a certain domain that best fit the task goal. Previous researches show that many of the lowlevel features (e.g., edge detection) can be shared among a wide variety of tasks, and can considerably leverage the data from a task with rich data supply to a task with few, leading to the a special type of transfer learning
. When employing the transfer learning method, one would reuse the representation transformation learnt from another task, and then
repurposing the model by retraining the parameters (possibly with extra layers).A recent research work by F. Elsayed et al. [29] combine the idea of transfer learning with the adversarial behavior of the deep neural networks, seeking to achieve the “repurposing” not by retraining the model, but by add specially designed perturbations to the inputs. They propose a new reprogramming task, in which the target model is reprogrammed to perform a new task chosen by the attacker, by using a single unrestricted magnitude adversarial perturbation across all natural inputs to the machine learning model, even if the model was not originally trained for this task. These perturbations can be considered as a special type of program, and feeding inputs with these them are identical to reprogramming the neural network.
In adversarial reprogramming, it is important to distinguish between the two tasks:

the original task , which consists of the input (suppose the input is an image of size with 3 channels) and the output label ,

and the adversarial (or more precisely, the reprogramming) task , which consists of the input with , such that the input for to be smaller in size than that for , and the desired output .
As such, the perturbation in this scenario is designed for the input of the adversarial task
, which is firstly padded into
to fit the input shape of the original network. The extra space of the input allows us to create an optional fixed mask to make the perturbed input(7) 
such that the original input may stay visually intact for easier inspection. Here, ensures the perturbation is valid in range, and denotes the elementwise product. In addition, the number of possible output labels for the adversarial task may be fewer than that for the original task, which requires us to create a mapping function
, possibly by hardcoding manually. For instance, if we reprogram the network originally trained for ImageNet task, which has 1,000 output labels, for a MNIST task, which only has 10 output labels, then we need to map averagely 100 values of
into one value of . Hence, the adversarial reprogramming task can be formulated as follows:(8) 
where is the adversarial program parameters, and is a hyperparameter to scale the weight penalty to reduce overfitting.
Their empirical study show that trained neural networks are more susceptible to adversarial reprogramming than random networks, which is a strong evidence to support the hypothesis that adversarial program serves as a special type of transfer learning to repurpose a welltrained network. Their findings also lead to a wide range of potential applications, and raise even more questions about the network security to be explored. For instance, it is possible for a malicious user to abuse the image classifier hosted on cloud to solve image captchas.
3 Attacking Strategies
3.1 A Taxonomy of Attacks
There are more than one way to attack a system. Defining the threat model, a concept brought from the information security community, which specifies what the attacker wants to achieve, what kind of knowledge she possesses, and what steps she would follow, allows us to analyze the possible attacks systematically and theoretically. Barreno et al. [30] propose a classification scheme of the strategies, which consists of three dimensions.

Influence In this dimension, one would need to consider how much control the attack has on the training process of the machine learning algorithm. In some occasions, the attacker has the ability to influence the training data during the training process, which is called causative attack. One typical class of of the causative attack is known as the poisoning attack, in which poisonous inputs are injected into the training data to alter the fitted decision boundary [3, 4]. The trojan attack proposed by Liu et al. [31] is another example of causative attack, which instead of tampering with the original training process, produces a new model and an attack trigger or trojan trigger by adding extra training data to retrain the original model. In most reallife scenarios, the attacker may only be able to probe and explore the behavior of the classifier, and infer the underly structure of it, but have no control over the training process. Attacks of this kind are called exploratory attack.

Specificity This dimension determines how much specificity the attacker’s goal is set. At one end of the dimension is the targeted attack, in which the one specific input point or a small set of input points is considered valuable to the attacker. At the other end of the dimension is the indiscriminate attacks, in which the attackers’ goal is more flexible, such that misleading the classifier to any false positive categorization would be a success.

Security violation This dimension distinguish the attacking strategy by the type of error that classifier is expected to make by it. At one end, the socalled integrity attack aims to cause false negatives on the classifier, which is similar to the goal of the evasion attack; At the other end, the socalled availability attack aims to would simply aims to cause the misclassifications of both false negatives and false positives, so that the system becomes effectively unusable.
These dimensions, along with their possible values, have considered many important points when working with adversarial attacks from the perspective of information security. In addition, more recent researches in deep learning community often consider another set of variables when discussing creating adversarial examples, which is wellsummarized by Papernot et al. [32]. This section will give more detailed discussions about some important aspects within this taxonomy.
3.1.1 Whitebox and Blackbox Attack
The difference between whitebox and blackbox attacks lies in whether the attacker has the knowledge of the classifier. In a whitebox attack, the attacker would be able to have as mush knowledge as it need to make the adversarial perturbation, which is totally invisible blackbox attack. Such difference is partially captured by the influence dimension proposed by Barreno et al. [30]. However, the causative attack specified by Barreno et al. launches the attack by influencing the classifier’s behavior during the training process, while the whitebox attack, though behave rather passively, would often require the gradient of the cost function with respect to some input .
The blackbox attack, which often requires the attacker to explore the behaviors of the classifier, is similar to the exploratory attack specified by Barreno et al. On this side, different researches may have different assumption on the type of information that the probe input will return. For instance, in Papernot et al. setting, the probe will only returns the class label produced by the classifier [33], while in Narodytska and Kasiviswanathan setting, partial knowledge of the confidence score produced by the classifier can be obtained [15]. In this paper, we treat the Whitebox v.s. blackbox as the most significant difference when presenting the selected strategy examples.
3.1.2 Targeted and Untargeted Attack
Unlike the interpretation of targeted attack in Barreno’s taxonomy, in modern terms, however, a targeted attack commonly means that there exists a particular targeted class (the target class) which the classifier is expected to (mistakenly) output. This central confusion between the two settings is whether the attack targets
on a particular input or output. Since in network security area, scenarios like intrusion detection is mostly binary classification problem, the targeted output is thus already implied. On the other hand, targeting at a specific output class becomes more valuable when tasks like image classification may involves thousands of possible classes. Another type of attack target is to move the correct class label outside the top
ranked output of the classifier as is proposed by Narodytska and Kasiviswanathan [15]. One extreme example of the untargeted attack task is proposed by MoosaviDezfooli et al. [34], who show that it is possible to find a single small image perturbation to fool a deep neural network classifier on all natural images.3.1.3 Leastcost and Constrained Attack
Constructing the adversarial example requires a tradeoff between the amount of perturbation to be added, and its utility (or effectiveness) of the attack that can be measured either by the likelihood of success or the desired impact on the probability distribution of the classifier’s confidence. The distance defines a way to measure the perturbation between the original input and the distorted version, but does not address how the tradeoff problem should be formalized. Broadly speaking, the attacker may either make sure the attack succeed when tries his best to minimize the amount of perturbation that attack requires on the input, or make sure the amount of perturbation is in a fixed range and tries his best to trick the classifier. Such difference leads to two broad types of attacking strategies, which are named as besteffort attack and bounded attack by this paper.
Leastcost attacks are of the type that would require the attack attempt must succeed, when minimize the amount of perturbation that is required. Since this formalization is used in the original paper on adversarial problem in deep learning by Szegedy et al. [12], it is also the earliest and most widely used type of formalization. They use the norm metric, the adversarial attack problem for a given input on a target class can be described as Equation 9.
Minimize  (9a)  
subject to:  
(9b)  
(9c) 
Notice that the constrains expressed in Equation 9c is often known as the boxconstraint to the optimization theory, and is always neccessary to make sure any pixel of the distorted input is still within the valid range of intensity, but sometimes omitted for simplicity.
In some scenarios, the leastcost attack may not suffice the requirement of the attacker, who may impose more restrictions on the amount of perturbation allowed to be added to the input. The effectiveness of the perturbation is thus measured by how much confidence the classifier will place on the distorted version comparing to the original input, when the maximum allowable distortion in the optimal direction is added. Tasks of this type can be formalized as
Maximize  (10a)  
subject to:  
(10b)  
(10c) 
in which denotes the probability score the neural network assigns to the class label for any given input. The hyperparameter specifies the upperbound of the perturbation that can be added to any input instance .
Notice that when comparing to Equation 9a, the optimization objective in Equation 10a becomes the attack utility rather than the perturbation, while the amount of perturbation now appears as a constraint term in Equation 10b. Within the attacking bound , one may still attempt to find the optimized solution for all feasible points of . For attacking strategies designed for leastcost attacks that uses iterative update method, it is simple to employ earlystop mechanism such that the iteration will end whenever the amount of distortion is above the bound.
3.2 Whitebox Attacks
Whitebox attack assumes that the attacker has the complete knowledge of the classifier. In most of the strategies, the attack strategy would need to be able to find the derivative (or gradient) of the model at certain input point. Although such conditions may seem impractical in most realworld scenarios, this type of attacks does reveal some underly weakness of the classifiers, and also forms the basis for the more realistic blackbox attacks.
In this part of the paper, we introduce several representative adversarial attack strategies that have been proposed by previous researches. Since it is unlikely to enumerate all possible strategies that have been seen in the literature, this survey tries to capture the basic ideas in the most cited works within the deep learning community, and their relationships.
3.2.1 The Optimization Attacks
The LBFGS attack is the firstly proposed by Szegedy et al. along with their discovery of the adversarial examples for the deep learning algorithms [12]. They formalize the problem as finding the optimal perturbation input , in terms of the norm, to make the classification function output a wrong class, which can be described in Equation 9. They replace the attack goal by introducing the loss function that originally used to optimize the parameters , and tackle this problem by using the boxconstrained LBFGS, and rewrite the optimization problem into the form of
(11) 
By using a line search to find the constant , this method yields an approximation to the optimal solution due to the nonconvexity nature of deep neural networks.
The optimizationbased attack strategy proposed by Carlini and Wagner [35], referred to as the CW attack, reuse the task definition as is described in Equation 9, and can be regarded as an extension of the LBFGS attack [12] in several ways. First, in addition to the norm metric, they define a more general distance function , which can has any of the , , or form. Second, instead of using the loss function of the classifier in substitution of the target directly, they study up to seven different objectives. Third, instead of using the LBFGSB optimization algorithm, they investigate three different methods for the optimization problem that make using optimizers don’t natively support box constraints possible.
3.2.2 The Fast Gradient Sign Method
One important drawback of the original LBFGS method is its huge computation cost, which requires estimating the Hessian matrix. Therefore, many followup researches start to use only the first derivative information, and lead to a wide class of the firstorder method. Fast Gradient Sign Attack (FGSM) [16] is among the first a few proposed adversarial attacks against deep learning algorithms, which aims to optimize the distance metric, and is designed primarily to be fast by using only the gradient information, instead of producing very close adversarial examples.
Given the input , the fast gradient sign method sets the adversarial example as
(12) 
where is the true label for the input , is chosen to be sufficiently small so as to be undetectable. Intuitively, the distorted input is obtained by moving away from the original input point of to increase the loss function alongside the direction of the gradient. This method can be easily used to generate targeted attack by making the distorted input move towards the target class at the gradient’s direction, as is described in Equation 13, where is the targeted output class label.
(13) 
Tramèr et al. extend this attacking strategy by adding a random noise that follows Gaussian distribution before the standard FGSM
[36]. Kurakin et al. [37] extend the FGSM by replacing the single update with step size in the direction of with multiple updates of step size and clipped by the same . Their proposed basic iterative method can be expressed as(14) 
where , i.e. the value of each pixel changes only by 1 on each step. They then further propose an iterative leastlikely class method, which turns the original indiscriminate attack strategy to an targeted strategy. The leastlikely class according to the predication of the trained network on image is
(15) 
Analogous to the change in the onestep version from untargeted FGSM to targeted FGSM, the iterative version of targeted FGSM can be described as:
(16) 
3.2.3 The JSMA Attack
Another type of firstorder method called Jacobianbased Saliency Map Attack (JSMA) is proposed by Papernot et al. [32]. The idea of saliency map originates from the visualization method for convolutional networks proposed by Simonyan et al. [38]. In JSMA, the saliency map identifies most promising input feature on which perturbation should be applied to cause the desired changes in the classifier’s output. Adversarial examples are then generated by performing greedily featurewise modification on the input instance.
The saliency map is based on the forward derivative, which refers to the gradient of the forwardpass function with respect to the input . The “positive” salience score of the feature is given by
(17) 
That is, the saliency score of feature remains 0, if increasing its value will either decrease the confidence of being classified as , or increasing its value will cause, averagely, the increase of the confidence score of all other output classes that . Otherwise, its saliency score is the forward derivative
scaled by the factor of
It is also easy to create a negative version of the saliency map by considering the impact of decrease the value of feature :
(18) 
The overall adversarial generation algorithm is implemented by iteratively choose the currentoptimal feature to modify until either the attack succeed or meet the distortion bound .
3.2.4 The DeepFool Attack
The DeepFool attacking strategy proposed by MoosaviDezfooli et al. [39] has a lot in common with the iterative FGSM, in the sense that it also moves towards the gradient’s directions iteratively, and updates the gradient at each iteration. However, the Deepfool strategy uses the original gradient direction instead of the sign of gradient. In addition, instead of setting the change rate to a fixed value , which is set to 1 at iterative FGSM’s experiment[37], the Deepfool strategy will always move the attacking point to the estimated decision boundary directly, by following the current gradient’s direction with the step size of the estimated distance between its current position and the decision boundary. So the Deepfool strategy for a binary classifier in the criterion setting can be described in Equation 19:
(19) 
where denotes the distorted input at iteration , is the estimated distance between the point of and the decision boundary of the function , while is the normalized gradient vector pointing towards the decision boundary. The same method can be easily extended to a multiclass classifier.
3.3 Blackbox Attacks
3.3.1 Intuitions
Crafting adversarial attack does not make random moves, but requires heuristics derived from the knowledge of the targeted classifier. Therefore, the main problem for making the blackbox attacks is how to obtain the information needed. The two basic premises making it possible are the transferability of the adversarial examples and probing the behavior of the classifier.
Transferability
Early researches on the adversarial problem in deep learning has shown that an adversarial example for one model will often transfer to be an adversarial on a different model, even if they are trained on different sets of training data [12, 16]. A more recent research by Papernot et al. [24] shows a more astonishing property of the adversarial examples for the machine learning algorithms, that they can transfer between the models even if they use entirely different algorithms (i.e., adversarial examples on neural networks transfer to random forests). They argue that for each pair of classifiers : the transferability can be broken down into two types: the intratechnique transferability, meaning both and are of the same machine learning technique (e.g., both are neural networks) but are trained on different parameter initializations or datasets, and the crosstechnique transferability, meaning and are two different techniques in natural (e.g., is based on a neural network and a decision tree). Then they prove that both types of the transferabilities are consistently strong across the space of machine learning techniques.
Probing
Another important premise of the blackbox attack is that the attacker is usually assumed to be able to have probe the classifier. To avoid ambiguity, we would refer to the targeted classifier as the oracle. A model receives dimensional input, each with possible values has the input space of cardinality of . Exploring the behavior of the oracle by testing all possible input is intractable. Therefore, approaches that require probing commonly focuses how to find the minimal set of input points to optimally probe the oracle classifier. More details on the probing strategies will be discussed in Section 3.3.3.
3.3.2 Substitute Network
Based on the transferability of the adversarial examples, Papernot et al. proposed a substitute network approach [33] which trains a local substitute deep neural networks with only limited number of label queries and can be scaled to different machine learning classifier types.
For an oracle network that outputs the probability distribution to indicate the final class label , the attack process begins with the construction of a substitute network , which should comply with the oracle’s expected input and output.^{1}^{1}1As before, the class label output of is given by . To make the substitute network best imitate the behavior of the oracle with the only limited label queries, they generate synthetic training data from an initial set of seed points, based on the Jacobian matrix to identify the directions in which the model’s output varies most significantly. For each of the input point , a new instance can be generated as . The overall working process is illustrated in Figure 1, in which is the initial seed training data, is the created substitute network architecture.
At iteration , the algorithm query the oracle to obtain the output label , together with the inputs , to train a new version of the substitute network . At the end of the iteration , a new set of training instances can be synthesized by appending distorted version of each instance to . To prevent the exponential growth of the query candidates, the reservoir sampling technique can be adopted here [33, 24]. For some iteration that above a threshold , only number of new inputs are selected from the original Jacobianbased dataset augmentation, such that each input in has an equal probability to be augmented in .
3.3.3 Local Search
Narodytska and Kasiviswanathan [15] point out that the substitute network method [33] tend to result in a degradation in the effectiveness of the attacks. Meanwhile, their proposed local search method does not require the assumption of the transferability, and can also save the overhead of retraining the substitute network. However, their local search method would require partial information of the confidence score produced by the neural network, when the substitute network method by Papernot et al. needs only the class label output of the oracle classifier.
They target on a socalled misclassification task. If we reuse the notations in Section , that returns the confidence score for each of the class by the oracle, and returns the final output label of the oracle, then is a function that returns confidence ranking of upto top classes in a vector form, which is also called the vector. Then the misclassification can be described as
(20) 
In general, the proposed local search strategy relies on the numerical approximations of the gradient, and iteratively searches a local neighborhood of pixels positions, to refine the current image. Each iteration of the local search procedure consists of two steps: first, evaluate the objective function for each of the points that form a local neighborhood, which is defined as the confidence score of the original correct class label of the on the distorted version of the input that produced by the oracle; and then, select a new solution based on the previous solution and points in .
3.3.4 GANbased Attack
The interpretation of the “imperceptible” provides a mathematically convenient way to measure the distortion to be added to the input. However, the simple norm measures only the distortion in the superficial highdimensional space, which may not reflect the impact on the human’s perception level, which can be crucial in complex domains. For instance, simple errors in the gramma or the improper wording in the adversarial example in natural language can be outstanding, as has been noticed by Li et al. [40] and Jia et al. [41]. Zhao et al. considers creating the natural adversarial example as their major focus, which can be tackled by using the GAN model [42]. One nice feature of their approach is that it requires no information of the gradient of the classifier, and thus can be applied in a blackbox scenario.
As has been introduced in Section 2.4.1, generative adversarial networks is family of generative models that are capable of generating realistic samples of various types of input like image or text from a latent space input . Zhao et al. first train a generator parameterized by by using a WGAN on the training dataset, which maps a random dense vector to an instance in the superficial space . Then they separately train a inverse function parameterized by which is capable of mapping the superficial space to the latent space by minimizing the reconstruction error:
(21) 
Based on the learnt generator and its inverse function , the original input can be mapped into the latent space , on which the perturbation is performed, resulting the distorted version in the latent space , and then mapped back to the input space . Thus, the natural adversarial example is derived by:
(22) 
and denotes the original classifier the attacker is targeting at. Although the “naturalness” is hard to measure directly, sample cases produced from several standard dataset do suggest their works as expected on both image and text domain.
A similar idea to this GAN based adversarial generation approach is proposed by [43], in which a deep generative model is involved, and the attacker aims to manipulate the latent representation instead of the superficial input . However, their attacking scenario of [43] is quite different, in that instead of attacking a single classifier to produce a wrong label, the adversarial input is constructed for an encoderdecoder pair, such that the decoding output becomes dramatically different from the input, as illustrated in Figure 2.
4 Defending Strategies
4.1 Intuitions
As their counter part of the attacking strategy taxonomy, Barreno et al. also propose a categorization for the defense strategies [30], which shares much overlapping with the ideas of the latest defending method for contemporary deep learning algorithms:

Robustness Whereas the entire problem of adversarial training can be boiled down to the robustness, Barreno et al. uses this term here to refer to a more specific meaning of model regularization. Regularization extends the basic optimization objective by adding a term to penalize the model complex, and has now become a standard practice in machine learning.

Detecting Attacks The defender may devise an attack detector to filter out the abnormal input before sending to the classifier.

Disinformation The defender may try to confuse the adversary’s estimate of the learner’s state. For instance, a sophisticated learner could trick the adversary into believing that a particular intrusion was not included in the training set.

Randomization The defender can randomize the decision boundary of the classifier, which is especially effective to defend against targeted attack that focuses on a specific input instance.
More recent researches on adversarial in deep learning community have proposed various methods to enhance the robustness of the model, most of which can be classified into one or more approach categories listed above. Defending strategies, once published, are exposed to the public, and can be compromised at any moment. In fact, many of the strategies to be reviewed in this section have already been proved to be unsecured by some follow up studies, and even the existence of universally robust defending strategy is by far an open question. Thus, some researchers try to provide intuitive interpretations to the cause of adversarial inputs, which may hopefully lead to some theoretical guides to the design of new and more effective defending strategies.
4.1.1 The Capacity Matters
When adversarial were firstly spotted in the deep neural networks, Szegedy et al. suggest that such phenomenon was caused by the extreme nonlinearity of the function. However, followup researches prove that compared to the deep learning models, simple models with low model capacity are in fact more vulnerable to such attacks [16, 44]. This argument is also soon supported by Madry et al. [17], who show that the model capacity alone is a major impactor on improving the robustness of the classifier, even training on the natural dataset only. In a word, “capacity alone helps”.
Theoretically speaking, the universal approximator theorem [45] guarantees that a neural network with at least one hidden layer can represent any function to an arbitrary degree of accuracy, provided that hidden layer has enough units. This gives the deep neural networks the possibility to be able to resist the adversarial perturbation given proper training data and procedure. On the other hand, it is impossible to have a shallow linear model to keep constant near training points while also assigning different outputs to different training points, making the models of this kind to have the capability to resist adversarial input rather dim. Therefore, it has been so far widely accepted that, shallow models are however, even more vulnerable to adversarial attacks. The concerns of the risk in deep models may partly due to its complexity — it may hard to fix a known problem even if it has been spotted.
4.1.2 Flaws in the Training Procedure
Although the root cause of the existence of adversarial examples for deep learning models is still unclear, it has been widely accepted that it has some direct connection with the standard supervised training procedure that does not consider the potential adversarial attacks. The objective defined by deep learning algorithms is usually set to minimize the expected error (or cost) that the classifier would output. This objective is sensible only under certain assumptions:

The training objective is usually the empirical risk minimization (ERM) that averages over all known training instances, in which each of instance is considered equally important. Though one may specify the weight of each training instance or the output labels, it is still a problem how these weights can be derived.

More importantly, the training data is wellsampled in a way that can unbiasedly reflect the data distribution which is assumed to be stationary. This does not reflect the dynamics of the minimax game that the attacker and defender should continuously adjust their strategies.
In the scenarios when such assumptions no longer hold, extreme caution must be paid to a betterformulated objective. Previous researches on alternative views of the optimization objectives usually root in the need to balance the ability to generalize from the training data, which is considered to be insufficient to problem of the adversarial learning [16].
4.2 Strengthening the Training Process
As is discussed above, one paradigm to understand the existence of adversarial attack is due to the indiscretion within the training process. Consequently an intuitive way to figuring out the solution is to improving the training process somehow to address the potential adversarial inputs.
4.2.1 Data Augment
Szegedy et al. [12] showed that by training on a mixture of adversarial and clean examples, a neural network could be regularized somewhat. Training on adversarial examples is different from other data augmentation schemes. For example, common approach to augment data for training a computer vision like translations or crops are expected to actually occur in the test set. However, for adversarial training, data augmentation requires inputs that tend to expose flaws in the ways that the model conceptualizes its decision function, which are unlikely to occur naturally.
Goodfellow et al. recommend that the training process needs to continually generating new adversarial examples at every step, and inject these adversarial examples into the training set [16]
. They also suggest to use batch normalization
[46] to scale adversarial training to large models for tasks like ImageNet, in which it is critical for each batch of examples to contain both normal and adversarial examples.The effectiveness of the data augment method is largely decided by the way that the augmented data are generated. Madry et al. [17] argue that adversarial training may benefit the most, when the adversarial examples for training closely maximize the model’s loss. They advocate to use the multistep PGD (projected gradient descent) as the best firstorder method to find the perturbation value to generate these adversarial examples, as well as to increase the model capacity simultaneously. This approach has been considered one of the most effective defending strategies that has been developed by far.
Data augment is a flexible method that can be easily plugged into other defending mechanisms. For instance, it is possible to inject the autoconstructed adversarial examples into the training data dynamically and iteratively, while employing other counteradversarial methods such as the distillation or testtime detection techniques.
4.2.2 Extrapenalty Term
The simplest form of regularization for neural networks, commonly known as the weightdecay is by penalizing the weights between layer pairs. Goodfellow et al. show that the original weight decay regularizer can overestimate the impact of adversary examples when compared to the dataaugment approach, making it hard to balance the training error rate and the regularization benefit. [16] As such, they suggest an alternative way to define the adversarial learning objective, based on the fast gradient sign method, which can be expressed by Equation 23:
(23) 
in which the hyperparameter is set to 0.5 by default. This approach can be interpreted in a way that it continually injects the newly generated adversarial examples based on the current version of the model.
4.2.3 Distillation
The transferability of the knowledge between models may not only exist for adversarial examples, but also allow us to compress models [47]. The idea of distillation is first proposed by Hinton et al. [48], in order to “distill” the knowledge from a cumbersome model to a small model that is more suitable for deployment.
For a classification task, the class probabilities produced by the neural networks are typically from a
layer that converts the logit,
into a normalized probability :(24) 
where the temperature parameter is commonly set to 1 by default, and a higher value for always produces a softer probability distribution over classes. The groundtruth labels in the training data commonly follow a distribution that puts all mass one a single correct value (i.e., the Dirac distribution of Equation 2.1). The training loss is commonly defined by the cross entropy between and the probability , which is sometimes improper (e.g., a digit “7” may sometimes does look like “1”) and may also cause numerical issues. Therefore, the distilling training process trains a cumbersome model (also known as the teacher model) first as usual, which outputs a softer version of the target distribution to train a distill model. Both the training process will use a higher temperature value, and restore the temperature to 1 for the test cases.
The intuition behind using distill training as a defense strategy by Papernot et al. [49] is to overcome the possible overfitting which may cause the neural networks to have “blind spots” [12]
. Comparing to the original distill training, defensive distillation employs the same network architecture for the distill model as the teacher model, meanwhile use a large temperature parameter
during training.4.3 Test Time Perturbation Recovery
Another class of approaches tries to tackle the problem at test time by removing the adversarial distortion from the input, usually follow the general idea to map the input into a latent space, and then reproduce a similar input via a deep generative model. In this part of the paper, we will cover several representative works that follow this intuition.
4.3.1 Deep Contractive Network
Comparatively speaking, signal denoising is a wellstudied research topic in machine learning area. Suppose we regard the adversarial perturbation on the input image as a special type of “noise”, then is it possible to remove such noise by standard denoising techniques as a preprocessing stage before classification?
This question is firstly answered by Gu and Rigazio [50], who show that a Deep Contractive Network (DCN) is a suitable choice for this task. As baseline approaches, they consider three simple methods for comparison:

by adding additive Gaussian noise and Gaussian blurring to the input;

by constructing a standalone autoencoder, in order to map the adversarial examples back to the original data samples; and

by using a standard denoising autoencoder (DAE), which can be regarded as the combination of the previous two methods
Their empirical experiments show that, the first baseline method can recover a large portion of the adversarial examples, but still falls drastically behind the performance on the clean input. In contrast, the second method can recover at least 90% of the original adversarial errors, but new adversarial examples can again generated from the stacked network that consists of both the encoding neural network and the original classifier. The same issue is also shared by the third method, making these methods not appropriate choice for the task. In fact, the newly derived models can be even more vulnerable to adversarial attacks.
As such, they appeals for an alternative type type of autoencoder, namely the Contractive autoencoder (CAE), which employs an extra Frobenius norm of the Jacobian matrix
in addition to the reconstruction error appears in the loss function of autoencoder [51], where is the code in latent space produced by the encoder, and is the tradeoff hyperparameter. With this penalty term, a contractive autoencoder tries to minimize the reconstruction error while keep the model invariantto any small change on the input. Deep Contractive Network (DCN) generalizes the (shallow) Contractive Autoencoder (CAE) by using a deep feedforward neural network for both the encoder and decoder, allows the model to have a much powerful transformation capability. The major difference is that the Jacobian matrix based penalty term is given in a layerwise manner instead of endtoend to reduce the computation complexity.
4.3.2 Testtime Adversarial Detection
A similar idea to the denoising approach (therefore may have overlap with the denoising approach) is by creating a detector in addition to the original classifier. Nguyen et al. [52] proposes that the detector can be simply implemented by adding a new adversarial class in the last layer of the original neural network. This idea is explored by Grosse et al. [53], who show that adversarial examples are not drawn from the same distribution as the normal training data, and can be detected by statistical tests. Instead of adding an extra adversarial class as is implemented by [53], Metzen et al. [54] proposed to create a subnetwork which is trained on the binary classification task of distinguishing normal data from data adversarial inputs. Another benefits of this approach is that the detector is trained entirely independent of the original DNN, without affecting the classification accuracy on the legitimate inputs.
One interesting idea of adversarial detection is based on the intuition that unnecessarily large feature input spaces that are unimportant to many of the classification tasks, is one of the main reasons that leaves a backdoor to the possibility of adversarial construction. Carlini et al. [55] notice that lowering the sampling rate can help to defend against the adversarial voice commands. Later, Xu et al. [56]
propose a strategy to reduce the degrees of freedom available by “squeezing” out unnecessary input features, namely the “feature squeezing”. More concretely, they explore two types of squeezing, including reducing the color depth of images, and using local or nonlocal smoothing (or blurring) to reduce the variation among pixels. Their empirical study prove that these squeezing method affect very little on the classifier accuracy, but can largely remove the impact of the adversarial perturbation. Thus, the testtime adversarial detector can be built by comparing the model’s prediction on the received input with its “squeezed version” to see if the outputs of the two versions are significantly different.
After studying a number of publications, Carlini and Wagner [57] conclude with a number of common techniques involved in the detection methods, including: 1) the use of a second neural network for dedicated to classify if an input is adversarial; 2) the use of PCA to detect statistical properties of the input image or network parameters; 3) statistical tests to detect the difference of distribution between the natural images and adversarial inputs, and 4) performing input normalization with randomization and blurring.
4.3.3 MagNet
MagNet is a popular defending mechanism that combines the idea of adversarial detection and distortion removal. From the perspective of representation learning, an autoencoder is capable of learning a lowdimensional manifold (e.g., 10 or 50 dimensions) that embeds in the highdimensional input space (e.g., dimensions for MNIST dataset), such that only points lie on the manifold are sensible. Therefore, the latent codes of the data points that follow the same generation process of the unperturbed training data should lie on the same manifold of that of the training data, and thus have low reconstruction errors. Based on this intuition, the MagNet method proposed by Meng and Chen [58] separate the defending strategy into two components: detector defense and reformer defense, both based on the autoencoders.
A detector is designed to ensure that the reconstruction error for an input can never be above a certain threshold derived from the validation set, where denotes the reconstruction function of the autoencoder. Otherwise, this input is considered to be an adversarial input that will be rejected by the defender. Inputs that have low reconstruction error are not necessarily close to the legitimate manifold. Therefore, one can also use the JensenShannon divergence between the distributions that generated by the softmax function of the classifier on both the original input and the reconstructed input as an extension, to estimate how likely the input is an adversarial example.
A reformer behave much similar to the DCN approach by Gu and Rigazio [50], which tries to move the input point towards the manifold of normal examples by using autoencoder, such that the adversarial perturbation can be removed in the newly reconstructed input. MagNet further strengthen the robustness of the reformer defense by building not just one, but a collection of autoencoders, and one reformer network is chosen randomly for a new input example at test time.
4.3.4 DefenseGAN
In Section 3.3.4, we introduced an attacking method based on the generative adversarial network, which constructs the input perturbation based on the latent space . Conversely, an analogous GAN based defending method, namely the DefenseGAN, is proposed by Samangouei et al. [59], which leverages the expressive capability of generative models to defend deep neural networks against such attacks.
A properly trained generator from GAN (especially the WGAN [26]) is capable of modeling the distribution of unperturbed training images . Formally speaking, for input data that follow the distribution of unperturbed image, the reconstruction error of the generator converges to zeros:
(25) 
where is the generator after steps of the training [60].
At test time, the algorithms first first map the new input to the latent space by minimizing the reconstruction error:
(26) 
by using common gradient descent optimization. Then, instead of the original input (with potential adversarial perturbation), the classifier is with a newly generated example . The whole process can be described by Figure 3.
Since the generator is expected to perfectly model the distribution of the unperturbed inputs, the reconstruction process can effectively reduce any potential adversarial noise.
4.4 The Fragility of Defenses
Despite the great efforts made by the community, the security guarantee concerning the adversarial attacks on general deep learning models is still inconclusive. Designing a “universal” defense approach seems to be impractical so far. A large number of existing defending techniques, though were claimed to be robust when proposed, have already been proven vulnerable. For instance, Carlini and Wagner [57] studied a number of the detectionbased adversarial learning techniques, and come with the conclusion that the detectionbased methods are not sufficiently strong to defend well design attacks, for both whitebox or blackbox attacks. In addition, they also show that the onceimportant Defensive Distillation and MagNet defenses are either completely or mostly broken [61, 62]. What makes it worse is that, even combining multiple weak defense mechanisms do not promise a stronger one [63].
Athalye et al. [64] studies this problem from the perspective of the model’s gradient, which plays a central role in enhancing the training process. They prove that a wide range of defending strategies are based on the so called obfuscated gradients, which do not guarantee security against adversarial attacks as expected. They classify the obfuscated gradients into three subcategories, and provide an effective attacking strategy for each:

Shattered Gradients either due to the defense is nondifferentiable, or causes the true gradient signal incorrect, and can be compromised by Backward Pass Differentiable Approximation;

Stochastic Gradients caused by randomized defenses, either the network itself is randomized or the input is randomized, and can be compromised by Differentiating over Randomness;

Exploding or Vanishing Gradients often caused by defenses that consist of multiple iterations of neural network evaluation, and can be compromised by Reparameterization.
They empirically show that 7 out of 8 defending strategies appear in ICLR 2018 conferences fall into these categories, all of which can be compromised by their attacking strategies.
The current empirical results force us to give more careful considerations on this question: Does the absolute security really exist? Our pursue of more powerful attacking and defending strategies may form an infinite loop, which can be endless after all.
5 Extended Studies
5.1 Verifying the Robustness
Absolute security is not possible by the stateoftheart though, it is often desired to have a clear view of the measurable risk, when a system is deployed, and leads to a new research topic of neural network verification. The key challenge of verifying the robustness of a system lies in the need to generalize the known valid inputs to a new dataset, whose size is often much larger than the test set for most benchmarks. However, the asymmetry between the two sides is that when the attacking side needs only to demonstrate one way to succeed, the defending side must prove the inexistence of such possibility [64], when it is not possible to exhaustively enumerate all valid input points near which the classifier is expected to give a approximately constant output. Therefore, the verification problem can be reduced to finding or disprove the existence of adversarial examples.
Carlini et al. suggests that there are two general approaches to evaluate the robustness of a learning algorithm [35]:

When a theoretical proof is attainable, a lowerbound of the robustness can be obtained, which tells the maximum risk that the system may face;

Or, an upperbound by constructing a successful attack can be obtained, to indicate the minimum risk of the system.
Intuitively, the theoretical lowerbound is more interesting to the application, but turns out much more difficult to obtain. In addition, making assumptions or approximations are always necessary when studying the theoretical proofs for deep neural networks, thus the claimed guarantee may not be the actual guarantee when these assumptions or approximations no longer holds.
Research works on this topic starts much earlier than adversarial learning in today’s deep learning sense, which can be traced back to Pulina et al. [65, 66]
, who was looking for a method to ensure the body position would stay in the safe region when controlled by a neural network. They developed one of the earliest verification system for MultiLayer Perceptrons (MLPs) by approximating the sigmoid activation function thus is reducible to constrained SAT problem, and can be solved by Satisfiability Modulo Theories (SMT) solvers
[67]. Though they considers only a very small dimension input space with very limited size of training data, the idea of employing SMT solvers in this research topic has considerable influence on future works.In the deep learning area, the CleverHans project by Papernot et al. [68, 69] is one of the most important projects of this type^{2}^{2}2https://github.com/tensorflow/cleverhans, which provides a software library to standardized reference implementations of both adversarial construction and adversarial training methods. They advocate the concept of verification to replace the common testing method to study the robustness of deep neural networks in adversarial environments. Similarly, Huang et al. [70] propose a Deep Learning Verification framework (DLV) that enables exhaustive search of the region partially based on But they feature their work by restricting the verification task on a set of small regions, and obtain the strength to guarantee that adversarial examples must be found if any of them exist for that given region and family of manipulations.
Some researches on the other side, impose some restrictions on the type of classifiers to get more interesting results. For instance, Fawzi et al. [71] study specifically the linear and quadratic family of classifiers, and proved theoretically the fundamental upper and lower bounds of the robustness against adversarial perturbation. Katz et al. [7]
suggest that the nonlinear activation function is the main reason for the difficulty of studying the properties of deep neural networks. As such, they focus specifically on the Rectified Linear Unit (ReLU) activation function
[72, 73, 74], and proposed the Reluplex (ReLU with Simplex), another verification system that works much more efficient by using an extended version of the simplex method.5.2 Adversarial in Reinforcement Learning
Reinforcement learning (RL) is a special type of machine learning tasks, in which an intelligent agent are required to learn how best to interact with the environment to maximize the accumulated or expected rewards it is expected to receive. At each time step , the agent is able to observe the environment at some state s, and decide its action . When the environment receives the action , its state is transferred into a new state with a probability conditioned on and
. Reinforcement learning differs from the usual supervised learning task in that the training data are not prepared in advance, but requires the agent to collect through its experience with the environment. However, both class of machine learning algorithms are fundamentally based on function fitting. From the RL’s perspective,
Huang et al. [75] firstly study the problem of adversarial attacks in reinforcement learning settings, and conclude that such attacks are also effective when targeting neural network policies. They experiment with three major class of RL learning algorithms: A3C (Asynchronous Advance ActorCriticor) [76], TRPO (Trusted Region Policy Optimization) [77], and DQN (Deep QNetworks) [78], and show that just like in the supervised case, even in the presence of perturbations perceivable to human may trick the agent to choose wrong actions and reduce the overall return.
Lin et al. [79] extend the research by Huang et al. by introducing the strategicallytimed attack and enchanting attack. In strategicallytimed attack, the adversarial choose to only attack the critical decision timepoint by strategically selects a subset of time steps instead of attacking at every time step. The intuition behind the criterion of being critical is that when a welltrained policy shows a strong preference of choosing a certain action, it is often means choosing other options may cause a drastic reduction of the future rewards. This can be easily captured by a relative action preference function with e threshold parameter . Instead of simply reducing the overall return, enchanting attack’s goal however, is to trick the agent to enter a targeted state from current state of step after steps, by creating a sequence of adversarial examples . This requires a combined effort of a generative model to predict the future states, and a planning model to generate a preferred sequence of actions.
5.3 Attacks in Physical World
Verifying the robustness of the models became firstly interested to physical control systems as they are machine learning algorithms due to their huge stack of potential loss [65, 7]. However, as deep learning models are more involved in physicalworld systems like autodriving, this issue began to draw attention of both communities. Manipulating images in their digital format is comparatively simple. But when image inputs are obtained directly from a digital camera sensor, when alternating the inputs from the physical world may face various restrictions. As remarked by [80], that creating adversarial attacks in the physical world can have a lot difference from that in the digital world. Many of the known methods become less effective due to reasons like the change of distance or the lighting condition etc.
Despite these difficulties, researchers have demonstrated such attacks in realistic scenarios [24, 33, 37]. Such attacks may also exists for inputs other than the images. For instance, Carlini et al.[55] show that audio signals can be synthesized in a way that is meaningless to human but may be recognized by smart phones as voice commands. Papernot et al. [81] point out that, attackers could target autonomous vehicles by using stickers or paint to create an adversarial stop sign, that the vehicle would interpret as some other sign like yield. There have been works to develop verification platforms focus exclusively on the potential adversarial risks of deep learning models exclusively to simulate physical world environment. For instance, by employing the idea of adversarial learning, Pei et al. [8] create a selfdriving test platform called DeepXplore to enhance the robustness of the selfdriving systems.
6 Conclusions
Adversarial learning is a newly emerged topic in the deep learning community, which however has a relatively long history in the information security area. The interpretation of the term “adversarial” in the deep learning community implies a restriction over the attacking method, that is adding an imperceivable amount of perturbation to the original input to influence the final output of the classifier.
In this paper, we present a selective review on attacking and defending strategies, which only cover a tiny fractions of what have been proposed so far. Whereas the minimax game seems to be endless loop, in which new methods and approaches of both sides emerge constantly. Studying such phenomenon does not only provide us with a tool of verifying the robustness and security of deep learning based system, but also an insight of the behavior of deep neural networks. Besides, it also leads to several new research topics, among which the Generative Adversarial Networks (GAN), a generative model that takes advantage of a minimax game with a discriminator to find the update gradients of its parameters, may be the most influential one. This paper also talks briefly about the adversarial attacks in more complex scenarios, such as the reinforcement learning, or even the physical world attack.
One obvious feature of the current adversarial deep learning is that most of the researches now focus exclusively on the computer vision problem, especially the image classification problem, which is heavily dependent on the convolution network architecture. There are drawbacks that known to be inherent in the convolution network architecture, for instance, by using the subsampling to keep translation invariance, the precise spatial relationships between higherlevel parts are lost. Therefore, we expect more studies on the nonconvolution architecture to appear in future researches.
References
 [1] Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Mathematics, pages 265–280, 1945.
 [2] Gert R G Lanckriet, Laurent El Ghaoui, Chiranjib Bhattacharyya, and Michael I Jordan. A robust minimax approach to classification. Journal of Machine Learning Research, 3(3):555–582, 2003.
 [3] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012.
 [4] Matthew Jagielski, Alina Oprea, Chang Liu, Cristina Nitarotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. ieee symposium on security and privacy, 2018.
 [5] Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. Adversarial perturbations against deep neural networks for malware classification. arXiv preprint arXiv:1606.04435, 2016.
 [6] George E Dahl, Jack W Stokes, Li Deng, and Dong Yu. Largescale malware classification using random projections and neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 3422–3426. IEEE, 2013.
 [7] Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International Conference on Computer Aided Verification, pages 97–117. Springer, 2017.
 [8] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, October 2831, 2017, pages 1–18, 2017.
 [9] Daniel Lowd and Christopher Meek. Adversarial learning. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 641–647. ACM, 2005.

[10]
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and
JD Tygar.
Adversarial machine learning.
In
Proceedings of the 4th ACM workshop on Security and artificial intelligence
, pages 43–58. ACM, 2011.  [11] Michael Brückner and Tobias Scheffer. Nash equilibria of static prediction games. In Advances in neural information processing systems, pages 171–179, 2009.
 [12] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
 [13] Daniel Lowd and Christopher Meek. Good word attacks on statistical spam filters. In CEAS, volume 2005, 2005.
 [14] Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi. One pixel attack for fooling deep neural networks. arXiv preprint arXiv:1710.08864, 2017.

[15]
Nina Narodytska and Shiva Kasiviswanathan.
Simple blackbox adversarial attacks on deep neural networks.
In
2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
, pages 1310–1318. IEEE, 2017.  [16] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
 [17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
 [18] Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, Deepak Verma, et al. Adversarial classification. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 99–108. ACM, 2004.
 [19] Pedro Domingos and Michael Pazzani. On the optimality of the simple bayesian classifier under zeroone loss. Machine learning, 29(23):103–130, 1997.
 [20] Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning. ieee symposium on security and privacy, 2018.
 [21] Amir Globerson and Sam Roweis. Nightmare at test time: robust learning by feature deletion. In Proceedings of the 23rd international conference on Machine learning, pages 353–360. ACM, 2006.
 [22] Fei Zhang, Patrick PK Chan, Battista Biggio, Daniel S Yeung, and Fabio Roli. Adversarial feature selection against evasion attacks. IEEE transactions on cybernetics, 46(3):766–777, 2016.
 [23] Ning Cao, Guofu Li, Pengjia Zhu, Qian Sun, Yingying Wang, Jing Li, Maoling Yan, and Yongbin Zhao. Handling the adversarial attacks. Journal of Ambient Intelligence and Humanized Computing, pages 1–15, 2018.
 [24] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to blackbox attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016.
 [25] Ian J Goodfellow, Jean Pougetabadie, Mehdi Mirza, Bing Xu, David Wardefarley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. pages 2672–2680, 2014.
 [26] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.
 [27] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5767–5777. Curran Associates, Inc., 2017.
 [28] Antonia Creswell, Tom White, Vincent Dumoulin, Arulkumaran Kai, Biswa Sengupta, and Anil A. Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1):53–65, 2017.
 [29] Gamaleldin F. Elsayed, Ian Goodfellow, and Jascha SohlDickstein. Adversarial examples for generative models. arXiv: Machine Learning, 2018.
 [30] Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug Tygar. Can machine learning be secure? In Proceedings of the 2006 ACM Symposium on Information, computer and communications security, pages 16–25. ACM, 2006.
 [31] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wenchuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. network and distributed system security symposium, 2018.
 [32] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In Security and Privacy (EuroS&P), 2016 IEEE European Symposium on, pages 372–387. IEEE, 2016.
 [33] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical blackbox attacks against deep learning systems using adversarial examples. arXiv preprint, 2016.
 [34] SeyedMohsen MoosaviDezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. arXiv preprint, 2017.
 [35] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pages 39–57. IEEE, 2017.
 [36] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble Adversarial Training: Attacks and Defenses. ArXiv eprints, May 2017.
 [37] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv: Computer Vision and Pattern Recognition, 2016.
 [38] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013.
 [39] Seyed Mohsen Moosavi Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), number EPFLCONF218057, 2016.
 [40] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016.
 [41] Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328, 2017.
 [42] Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv preprint arXiv:1710.11342, 2017.
 [43] Jernej Kos, Ian Fischer, and Dawn Song. Adversarial examples for generative models. arXiv: Machine Learning, 2017.
 [44] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016.
 [45] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
 [46] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. international conference on machine learning, pages 448–456, 2015.
 [47] Cristian Buciluǎ, Rich Caruana, and Alexandru NiculescuMizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, pages 535–541, New York, NY, USA, 2006. ACM.
 [48] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
 [49] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Security and Privacy (SP), 2016 IEEE Symposium on, pages 582–597. IEEE, 2016.
 [50] Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068, 2014.

[51]
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio.
Contractive autoencoders: Explicit invariance during feature extraction.
In ICML, 2011.  [52] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. computer vision and pattern recognition, pages 427–436, 2015.
 [53] Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick D Mcdaniel. On the (statistical) detection of adversarial examples. arXiv: Cryptography and Security, 2017.
 [54] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. international conference on learning representations, 2017.
 [55] Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields, David A Wagner, and Wenchao Zhou. Hidden voice commands. pages 513–530, 2016.
 [56] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. network and distributed system security symposium, 2018.
 [57] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3–14. ACM, 2017.
 [58] Dongyu Meng and Hao Chen. Magnet: A twopronged defense against adversarial examples. computer and communications security, pages 135–147, 2017.
 [59] Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defensegan: Protecting classifiers against adversarial attacks using generative models. In International Conference on Learning Representations, volume 9, 2018.
 [60] Maya Kabkab, Pouya Samangouei, and Rama Chellappa. Taskaware compressed sensing with generative adversarial networks. arXiv preprint arXiv:1802.01284, 2018.
 [61] Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv:1607.04311, 2016.
 [62] Nicholas Carlini and David Wagner. Magnet and ”efficient defenses against adversarial attacks” are not robust to adversarial examples. arXiv preprint arXiv:1711.08478, 2017.
 [63] Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defenses: Ensembles of weak defenses are not strong. arXiv: Learning, 2017.
 [64] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420, 2018.
 [65] Luca Pulina and Armando Tacchella. An abstractionrefinement approach to verification of artificial neural networks. In International Conference on Computer Aided Verification, pages 243–257. Springer, 2010.
 [66] Luca Pulina and Armando Tacchella. Challenging smt solvers to verify neural networks. Ai Communications, 25(2):117–135, 2012.
 [67] Clark W Barrett, Roberto Sebastiani, Sanjit A Seshia, Cesare Tinelli, et al. Satisfiability modulo theories. Handbook of satisfiability, 185:825–885, 2009.
 [68] Nicolas Papernot, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Fartash Faghri, Alexander Matyasko, Karen Hambardzumyan, YiLin Juang, Alexey Kurakin, Ryan Sheatsley, et al. cleverhans v2. 0.0: an adversarial machine learning library. arXiv preprint arXiv:1610.00768, 2016.
 [69] Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, YiLin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. cleverhans v2.1.0: an adversarial machine learning library. arXiv preprint arXiv:1610.00768, 2018.
 [70] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neural networks. In International Conference on Computer Aided Verification, pages 3–29. Springer, 2017.
 [71] Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Analysis of classifiers’ robustness to adversarial perturbations. Machine Learning, 107(3):481–508, 2018.
 [72] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 315–323, 2011.
 [73] Kevin Jarrett, Koray Kavukcuoglu, Yann LeCun, et al. What is the best multistage architecture for object recognition? In Computer Vision, 2009 IEEE 12th International Conference on, pages 2146–2153. IEEE, 2009.

[74]
Vinod Nair and Geoffrey E Hinton.
Rectified linear units improve restricted boltzmann machines.
In Proceedings of the 27th international conference on machine learning (ICML10), pages 807–814, 2010.  [75] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017.
 [76] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. international conference on machine learning, pages 1928–1937, 2016.
 [77] John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. international conference on machine learning, pages 1889–1897, 2015.
 [78] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A Riedmiller. Playing atari with deep reinforcement learning. arXiv: Learning, 2013.
 [79] YenChen Lin, ZhangWei Hong, YuanHong Liao, MengLi Shih, MingYu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06748, 2017.
 [80] Jiajun Lu, Hussein Sibai, Evan Fabry, and David A Forsyth. No need to worry about adversarial examples in object detection in autonomous vehicles. arXiv: Computer Vision and Pattern Recognition, 2017.
 [81] Nicolas Papernot, Patrick D Mcdaniel, Ian J Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical blackbox attacks against machine learning. computer and communications security, pages 506–519, 2017.
Comments
There are no comments yet.