I Introduction
Deep Neural Networks (DNN) have been widely used in many applications, e.g., autopilot [1], speech recognition [2]
, and face recognition
[3]. For certain tasks, its capability is even better than humans, making it an indispensable part of some critical systems, such as selfdriving cars [1], access control systems [4], and radio systems [5]. However, the safety of DNN has been widely concerned, i.e., it is vulnerable to adversarial samples, which were first discovered by Szegedy et al. [6] and means a kind of samples formed by adding a small perturbation to natural samples. The tiny changes of samples usually do not affect human judgment, but it can indeed make the trained model return a different output from the original sample with high confidence.The researchers of the Machine Learning (ML) community have been studying the relationship between adversarial samples and DNN trustworthiness
[7]. From the perspective of algorithm, it is widely considered that the existence of adversarial samples is caused by the extremely high dimension of input feature space and the linear nature of the DNN model [8]; while other researchers may also argue that adversarial samples are not the problem of model structure, but rather the features of datasets [9]. The Software Engineering (SE) community, on the other hand, considers the DNN model as a kind of software and adversarial samples as the bugs hidden inside the program. To detect potential defects of a DNN model, various testing methods were proposed to perform quantitative analysis on the quality of DNN models. Neural coverage [10] is the most used and the first criteria for testing. After that, more and more testing methods are emerging, e.g., surprise adequacy [11], mutation testing [12], neuron boundary coverage
[13], DeepXplore [14], DeepInspect [15], fuzzing testing [16], and the integrated testing framework [17].Several DNN testing methods were validated to be effective in detecting adversarial samples as a kind of bug in DNN systems. For instance, Wang et al. [18] used Model Mutation Testing (MMT) to detect adversarial samples; Kim et al. [11] showed that adversarial samples and normal samples have different surprise adequacy; Wang et al. [19] found that adversarial samples can be distinguished with the model hidden layer output. However, adversarial sample detection mechanisms may be still vulnerable to adaptive attacks, i.e., the attacker grabs the model information and defense strategy. Tramer et al. evaluated [20] the most recent works on adversarial defense and detection proposed by the ML community. They found that almost all of these methods can be circumvented with their accuracy substantially reduced from what was originally claimed. One possible reason is that most of the evaluated methods are based on a single model, and thus are very easy to be targeted.
MMT is one of the stateoftheart multiplemodel method on detecting adversarial samples [18]. The idea is that adversarial samples are more sensitive to the slight change of the decision boundary. It first creates a group of mutated models and then detects adversarial samples based on the consistency of the outputs generated by the mutated models and the original model. Though MMT achieved great success in adversarial sample detection, there are two main disadvantages that may hinder its wide application in reality. First, it may need over 100 mutated DNN models (with a similar computational cost to the original model) to detect adversarial samples, leading to an unacceptable cost in practical use [19], especially on the embedded systems with limited computational and storage resources. Second, since the mutated models have very similar decision boundaries to the original model, such lack of diversity could make MMT fail to detect highconfidence adversarial samples relatively far from the decision boundary.
To overcome the above shortcomings, in this paper, we propose GraphGuided Testing (GGT) for adversarial detection, which is also a multiplemodel method. We argue that DNN of a certain structure can be beneficial to image classification, as well as adversarial detection, while the structure of DNN can be designed or optimized by the theory in network science. The relational graph is the most recently proposed technology to facilitate the design of DNN structure. According to the characteristics of the relational graph, we selectively remove some of the edges in the graph to achieve the pruning of DNN and obtain diverse decision boundaries. We also exploit the average number of edges connected to a node, i.e., Average Degree (AD), and the average distance between any two nodes in the network, i.e., Average Shortest Path Length (ASPL), to obtain more competitive detection results. By comparing with MMT, the generated DNN models in our GGT have much less floating point operations (FLOPs), about only 5% of the original model, and meanwhile, the high diversity of DNN structure leads to significantly better performance on detecting adversarial samples with an even smaller number of models required. In particular, we make the following contributions.

We propose a graphguided pruning method for DNN models to significantly reduce the model computational cost with low accuracy loss, where the Average Degree (AD) of the graph is used to control the number of FLOPs in the generated DNN model.

We propose GraphGuided Testing (GGT) for adversarial detection, which can detect adversarial samples more efficiently, i.e., using significantly fewer additional models with much less computational cost, by comparing with MMT.

We find that the Average Shortest Path Length (ASPL) is correlated with both the accuracy of individual models and the adversarial detection performance of GGT. The graph with shorter ASPL can naturally generate a DNN model with a higher accuracy, which can be further used to better distinguish adversarial samples and normal samples, and thus is more suitable to establish our GGT.

We compare our GGT with MMT on CIFAR10 and SVHN, and find that GGT indeed outperforms MMT both on accuracy and efficiency. MMT achieves the average accuracy of 74.74% and 44.79% on CIFAR10 and SVHN by using 51.12 and 77.16 models on average, respectively; while GGT detects 93.61% of adversarial samples on CIFAR10 and 94.46% on SVHN with only 30.04 and 28.50 pruned models on average, respectively. Moreover, the FLOPs of a single model adopted in GGT is only 5.22% of the model adopted in MMT.
The rest of the paper is organized as follows. Next, we summarize the related works. In Section III, we discuss our research questions. In Section IV, we introduce our method, followed by the results in Section V. Finally, we give the threats to validation in Section VI and the paper is concluded in Section VII.
Ii Related Works
In this part, we give the related works on graph based DNN analysis, adversarial samples for DNN, and DNN testing for adversarial sample detection.
Iia Graph Structure and DNN
In the real world, a lot of complex systems in biology, society, and technology can be described by graphs composed of a node set and an edge set [21]. Typical examples are brain network [22], communication network [23], and worldwide web [24], where each node represents a neuron, a person, or a web page, respectively, and the edges are the connections between each kind of these objects, which facilitate their information exchange.
DNN consists of layers of neurons and the connections between them, with its structure naturally captured by a graph, where each neuron is connected with those in the former and the next layers, with the extracted feature transmitted through edges. DNN architecture and its performance are highly correlated, which is widely recognized in the ML community [25, 26]. Recently, You et al. [27] established the relationship between DNN architecture and its performance using a relational graph. They argue that the directed data flow is too complex to model, and thus it is more reasonable to focus on the tractable message exchange.
As shown in Fig. 1
, a 3layer MultiLayer Perceptron (MLP) with fixed layer width can be represented by a 4node relational graph. Assume that each node
has a node feature , the edges between node 1 on the previous layer and nodes 1 and 2 on the next layer mean there are message transmissions from the previous layer to the next layer, which is defined by a message function . The input of a message function is the previous node feature and the output is the updated node feature. At the inference stage, MLP computes the input feature layer by layer. The corresponding process in the relational graph is that message functions transform node features, then their outputs are aggregated at each node by an aggregation function . For an layer MLP, the th round message exchange can be described as(1) 
where and reppresent two nodes in the relational graph, is the neighborhood of , is the input node feature of , and is the output node feature of .
The relational graph can also be applied to more complex DNN architectures, including Convolution Neural Networks
[28](CNNs) and residual connections
[26].IiB Adversarial Samples for DNN
Adversarial attacks exist in a variety of deep learning application scenarios, e.g., image classification [29], link prediction in social networks [30]
, and natural language processing
[31]. This work focuses on adversarial samples in image classification. Adversarial attacks can be divided into blackbox attacks and whitebox attacks, with respect to their knowledge of the DNN model information. Whitebox attacks grab the whole information of the target model, including model structure and parameters. Blackbox attacks only know the input and output of the DNN model, and thus usually need greater perturbation to make a successful attack. We will introduce several most commonly used adversarial attack methods, including 4 whitebox attacks (FGSM, JSMA, CW, and Deepfool) and 2 blackbox attacks (Local Search Attack and One Pixel Attack).IiB1 Fgsm
Fast Gradient Sign Method (FGSM) is proposed by Goodfellow et al. [8]. They found the derivative of the model to the input, then used a sign function to get the gradient direction. The perturbation is obtained by multiplying by one step in the sign direction and the final adversarial sample is
(2) 
where is the benign sample, is the step,
is the loss function of the trained model, and
is the ground truth label of . FGSM is “fast” because it only updates the perturbation once, and the result is not guaranteed to be minimal.IiB2 Jsma
Jacobianbased Saliency Map Attack [32] (JSMA) is a kind of targeted attack that utilizes the adversarial saliency between the input feature and the output only by modifying a small number of input pixels. There are 3 steps in JSMA, calculating the forward derivative, calculating the adversarial saliency map, and adding perturbation. The forward derivative is obtained by deriving the output of the model’s last layer for a given input. Then the attacker calculates a saliency map based on the Jacobian matrix that reflects the impact of different input features on the output. The final perturbation is added on the top2 dominating features (2 pixels) of the input.
IiB3 Cw
Carlini & Wagner Attack [33] (CW) is a kind of optimization based attack method. The idea is treating the input as a variable, train the input (add perturbation) with fixed model parameters, maximize the distance between the ground truth label and the model output, and minimize the distance between the target label and model output.
IiB4 Deepfool
Deepfool Attack [29]
aims to find the shortest distance from the normal sample to the classification hyperplane and cross the boundary to generate adversarial samples. For twoclassification problem, the object of perturbation
is(3)  
where is the model output on the classification hyperplane and is the normal sample. This objective function can be solved iteratively to obtain the smallest perturbation. For the multiclassification problem and the detailed derivation process, we refer readers to [29].
IiB5 Local Search Attack
Narodytska et al. [34] proposed a simple black box attack with no internal knowledge of the target network. Their attack uses local search to construct an approximation of the model gradient, then uses it to guide the generation of perturbation. The object is to minimize the output confidence of benign sample on label . There are mainly two steps in the local search. First, obtain the confidence with the perturbation added on the last iteration, where is the perturbed image, and sorts the confidence scores in descending order; Second, the perturbation is added depending on whether the attack is successful. This process will end after the specified number of iterations or the attack successes.
IiB6 One Pixel Attack
One Pixel Attack achieves misclassification by only modifying one pixel of the input image [35]. The object is
(4)  
where is the normal sample, is the perturbation, and is the model confidence on the target adversarial label. To obtain the best pixel location and perturbation amplitude, Su et al. [35]
used differential evolution. The perturbation is represented by a 5element vector, which is the xy axis coordinates and the disturbance amplitude of each RGB channel. We refer readers to
[35] for details.IiC Testing for Adversarial Sample Detection
Testing for DNN models aims to find bugs (adversarial samples) or evaluate the quality of a test set, based on which we can fix the model and avoid vulnerabilities being exploited. Quite recently, DNN testing technologies are continuously developed for adversarial sample detection. Guo et al. [36] proposed a testing framework to detect logical bugs, crashes, and NotaNumber (NaN) errors in deep learning models. However, their work only involves softwarelevel bugs, but cannot detect bugs (adversarial samples) hidden in the model structure and weights. Ma et al. [37] found that adversarial samples have different activation patterns from normal ones, and thus introduced invariant checking for adversarial sample detection. Yin et al. [38] thought that adversarial and normal samples can be distinguished in the input subspace, i.e., adversarial samples are closer to the decision boundary, based on which they trained a number of detectors using asymmetrical adversarial training. Wang et al. [19] proposed that for a normal input, a deep learning model should process it with increasing confidence. They dissected the middle layers of DNN to generate a set of submodels whose prediction profile for the input can be used to determine whether the corresponding input is within the processing capacity of the model, then a sample is considered as normal if it is within the handling capabilities of the submodels, otherwise, it is adversarial. Zhang et al. [39] investigated the uncertainty patterns between normal samples and adversarial samples, raised an automated testing technique to generate testing samples with diverse uncertainty patterns. Their work mainly focuses on providing a high quality testing dataset and evaluating existing defense techniques.
However, detecting DNN bugs is not easy, Carlini and Wagner [40] evaluated ten detection methods proposed in the past years on different threat models and found that six of them are significantly less effective under generic attacks. Moreover, with a whitebox attack that designed for penetrating a given defense mechanism, half of the methods fail to provide robustness, three increase robustness slightly, and two only works on simple datasets. All of the evaluated methods are based on a single model, and their criteria can be easily added to the penalty of adversarial sample generation to bypass the detection mechanism. We argue that the testing methods based on multiple models, such as MMT, can be more robust in adversarial sample detection. In fact, MMT indeed achieves SOTA performance in detecting various adversarial samples, when it integrates enough mutated models. However, each coin has two sides, and the huge resource cost may hinder its wide application. This is the reason why we propose GGT in this paper, which is based on multiple pruned models, and much simpler than MMT, making it more applicable in reality.
Iii Research Questions
When we use multiple models to detect adversarial samples, it is crucial to ensure that each individual model has competitive accuracy with the original one. Therefore, in MMT, the mutation rate is set to a very small value to make sure that the mutated models are valid, i.e., their performance is not significantly worse than the original model. Such a setting may result in a huge redundancy, leading to unnecessary resource consumption. On the other hand, recently, a lot of pruning methods are proposed to simplify DNN models without losing their effectiveness [41, 42]. An interesting approach is to use a relational graph to capture the structure of DNN [27], and then prune DNN models under the guidance of sparse graph design. Traditionally, graph sparsity is always represented by the small AD, then we give the first research question, RQ1: Can we significantly reduce the complexity of DNN models with low accuracy loss, guided by the AD of the relational graph, so that we can use smaller pruned models to design our GGT for efficient adversarial sample detection?
Generally, in network science, ASPL, defined as the minimum number of links required to connect any pair of nodes, is an important structural property, which is strongly correlated with a series of network dynamics [43], e.g., small ASPL can reduce the communication cost and increase the ability of synchronization [44]. As DNN structure is described by a relational graph, the training and testing of DNN can be considered as certain dynamics on the relational graph. Therefore, it is expected that the ASPL of a relational graph could also be correlated with the performance of DNN model, which leads to our second question, RQ2: What is the relationship between the performance of the pruned DNN and the ASPL of the relational graph? Can we adopt the DNN models with longer (or shorter) ASPL to design better GGT so as to improve the adversarial sample detection?
Based on information theory, a sparse graph can provide more structural information than a fully connected graph, i.e., a large number of graphs with different structure could be generated for a particular AD satisfying , while only one fully connected graph can be generated when it is satisfied , supposing the number of nodes in the graph is fixed. Such diversity can indeed benefit graph algorithms, such as node matching between networks [45]. Therefore, it is expected that the integration of DNN models with sparse structure can benefit our GGT, i.e., we may need a smaller number of pruned models in our GGT to get better adversarial sample detection accuracy, compared with MMT. On the other hand, there is no such thing as a free lunch, i.e., it may be argued that since a single pruned model in GGT has significantly smaller computational cost than the mutated model adopted in MMT, we may need more pruned models to detect adversarial samples with comparable accuracy. This leads to our third question, RQ3: Does the diversity of pruned DNN models, introduced by the sparsity of relational graphs, benefit our GGT for better detecting adversarial samples? Are smaller number of pruned models required in GGT to get comparable detection accuracy, compared with MMT?
In MMT, various mutation operators are adopted to make the decision boundaries of the mutated models slightly different from the decision boundary of the original model. Therefore, MMT metrics favor identifying the adversarial samples that locate near the decision boundary, which have relatively low confidence in the model output [46]. In other words, MMT may not be able to detect the highconfidence adversarial samples that are relatively far from the decision boundary. Our GGT, on the other hand, is based on the pruned models which may be significantly smaller than the original model, i.e., their decision boundaries could be more different. As a result, GGT may be more effective than MMT in detecting highconfidence adversarial samples. Thus it is interesting to give our last question, RQ4: Is the superiority of GGT over MMT even more significant on detecting highconfidence adversarial samples that are relatively far from the decision boundary?
Iv GraphGuided Testing
Our GraphGuided Testing (GGT) for adversarial sample detection consists of three steps, graph generation, pruned model generation, and label change statistics, as shown in Fig. 2. Given a DNN model, we first generate diverse relational graphs with a certain number of nodes and edges according to the predefined AD. We then generate pruned DNN models based on the relational graphs, which are further integrated to detect adversarial samples.
The detection is based on the sensitivity of the pruned models to different kinds of samples. For a normal sample, pruned models tend to give consistent outputs; for an adversarial sample, there will be a variety of labels in the outputs. We use statistical hypothesis testing to capture such difference, and the threshold is determined by normal samples. Thus our method can be used to detect unknown adversarial samples.
Iva Graph Generation
There are a lot of characteristics, such as degree distribution, Average Degree (AD), Average Shortest Path Length (ASPL), average cluster coefficient, etc., to represent the graph structure. In this work, we mainly focus on undirected graphs, and simply set that each node has the same degree since the degree of a normal DNN internal neurons are equal. In this case, AD can be used to control the sparsity of relational graphs, so as to determine the pruning rate of the corresponding DNN model. We explore relational graphs with a wide range of ASPL to better answer RQ2.
According to the definition of relational graph [27], we define an undirected graph by a node set and an edge set . Each node has an associated feature
, which is a tensor representing multichannel data. The number of nodes
must be less equal than the number of channels in the DNN model. Suppose there are links in a relational graph with nodes (selfconnection is not considered here), the AD of the graph is calculated by(5) 
The pruning rate (or sparsity ratio) of the DNN model is then defined as
(6) 
For the generated graph, we further regulate its ASPL to obtain a series of relational graphs with different ASPL. The object of regulating ASPL is given by
(7) 
where is the total number of nodes in the graph, is the distance function that measures the shortest path length between nodes and . The regulation of ASPL follows the rewiring process proposed by Xuan et al. [44]. The details are listed in Algorithm 1. As shown in Fig. 3, we first randomly choose two nodes in the graph, and then exchange their neighbors. Note that the selected four nodes must be different. For example, in Fig. 3, node , its neighbor node , node , and its neighbor are selected. We delete the edges between and , and , and then create new edges to connect and , and and . This ensures the degree of each node fixed during the regulation. After new edges are created, we make the following check: If the graph is connected after rewiring and the current ASPL is closer to the target ASPL interval , we accept the update, otherwise, we reject it, and then return to the node selection step. Once the current ASPL satisfies , we record the relational graph. Once the maximum number of rewiring times is achieved, we stop the rewiring process. Note that, we randomly initialize a large number of relational graphs (all the nodes have equal degrees) and repeat the above process for each of them, in order to get enough relational graphs with their ASPL falling into various predefined intervals.
IvB Pruned Model Generation
Pruned DNN models are created using the adjacency matrix of the above generated relational graph. For a graph with nodes, the elements of its adjacency matrix indicate whether pairs of vertices are adjacent or not in the graph. If there is an edge between nodes and , i.e., there is message exchange between them, we have , otherwise . Considering a DNN with complex components, such as convolution layers and pooling layers, we define the node feature as a tensor and the message exchange as convolutional operator, which is defined as
(8) 
where is an aggregation function, is the convolutional operator, is the neighbors of node , and represents the th round of message exchange. A Pooling can be regarded as a special convolution, a pooling message exchange can be expressed as
(9) 
where
is the max pooling operator and
is the pooling kernel size. As shown in Fig. 4, in a relational graph, a round of message exchange means the extracted feature is passed to the next layer. In a normal convolutional layer, the convolution kernel will traverse the entire feature map, and in the corresponding relational map, it is expressed as one node is linked to all the other nodes. There could be redundant edges that exist in this kind of fully connected relational graph, and reducing the redundancy in the graph may in turn prune the DNN model. Using the generated sparse relational graph, we obtain a pruned model after mapping. The pruning is achieved by using a masking matrix multiplying the original convolution weight, where the reserved weight is multiplied by 1, and the removed weight is multiplied by 0. After pruning, the number of feature maps is constant, but the values of those feature maps corresponding to the masked weights are all equal to 0.Different layers of a DNN model are generally different in size. We allow that the same node in different layers or within the same layer has different dimensions. Specifically, suppose one layer has channels, and the total number of nodes in the relational graph is . Then there are () nodes that have channels. According to the number of channels that a node is assigned, we create a mask matrix as described above, and multiply it by the weights corresponding to the channels. That is, the weights of the corresponding edges in the relational graph are retained.
We will retrain these pruned models to improve their accuracy. Note that the masked weights can be removed in postprocess, thus the actual parameters and computational cost of the pruned models can be significantly reduced. Here, we argue that, when we use the retrained pruned models to detect adversarial samples, it’s testing time, rather than training time, that really matters in the real application. Therefore, it is quite important to simplify DNN models by using our graphguided pruning technology to reduce the memory and computational cost in the online detection phase.
IvC Label Change Statistics
As shown in Fig. 5
, adversarial samples are generally near the original model decision boundary, while pruned models have different boundaries from the original model. On the one hand, pruned models have a small probability to give wrong output for some normal samples (because of pruning and retraining); on the other hand, their boundaries may also be more sensitive to adversarial samples. Because the model decision boundaries are complex, and an adversarial sample that crosses the boundary of the original model may also cross different boundaries of the pruned models, i.e., outputting multiple different labels. We thus can use Label Change Rate (LCR) to measure the diversity
[18] of the pruned models’ outputs, which is defined as(10) 
where is the set of pruned models, is the input, is the output of the original model, is the output of the pruned model, is the size of set , and is defined as
(11) 
which counts the number of times that the classification results in the pruned models are different from that of the original model. We assume that, given a certain number of pruned models, normal samples and adversarial samples will have significantly different scores. According to the experimental results, we then set a threshold to distinguish adversarial and normal samples.
Specifically, we use the Sequential Probability Ratio Testing [47] (SPRT) to determine whether an input is adversarial dynamically. It is usually faster than using a fixed number of pruned models. The SPRT probability ratio is calculated by
(12) 
where is the total number of used pruned models, is the number of pruned models that output different labels, , and . is a threshold, which is determined by the LCR of normal samples. We calculate the Area Under the Receiver Operating Characteristic (AUROC) to determine whether normal and adversarial samples can be distinguished by . That is, we get the LCR values of a set of normal samples and a set of adversarial samples using Eq. (10), and then calculate AUROC for every possible . In the AUROC, the xaxis and yaxis represent the true positive rate and false positive rate obtained by using the threshold , respectively. The closer AUROC is to 1, the better the threshold is. is a relax scale, which means neither hypothesis (normal or adversarial) can be denied in the region . Then, we calculate the deny LCR , and accept LCR using
(13) 
where and denote the probability of false positive (a normal sample is misclassified as an adversarial sample) and false negative (an adversarial sample is misclassified as a normal sample), respectively. During the testing, the input will be sent to the original model as well as the pruned models. Then, we can get the dynamic LCR based on Eq. (12), denoted by , which is compared to the deny LCR and accept LCR. If , the input is considered as an adversarial sample, while if , it is considered as a normal one. The whole process is shown in Algorithm 2.
V Experiments
We implemented our GGT for adversarial sample detection with Pytorch (
version 1.6.0) based on Python (version 3.7).Va Experiment Settings
VA1 Datasets and Models
We evaluate our approach on two image datasets: CIFAR10 and SVHN. They are widely used in the evaluation of DNN testing frameworks [18, 17, 12]
. The former consists of 50,000 images for training and 10,000 images for testing, while these numbers for the latter are 73,257 and 26,032, respectively. Note that here we do not use the MNIST dataset, since the adopted model for this dataset (e.g., LeNet) is too small, and thus the diversity of the relational graph could be largely limited. The image size of both datasets is
. ResNet18 and VGG16 are adopted for CIFAR10 and SVHN, with their accuracy equal to 93.03% and 95.63%, respectively. These models are independently trained in our experiment without fine adjustment of hyperparameters, thus the accuracy will fluctuate within a small range compared with the public models. GGT and MMT both use SPRT to detect adversarial samples, the number of used mutated or pruned models varies for different samples. For the parameters in SPRT, we set both
and equal to 0.05, and the relax scale to 10% of the threshold . And we set the maximum number of models to 100 for both methods.VA2 GraphGuided Pruned Models Generation
For CIFAR10 and SVHN datasets, we set the number of nodes in each relational graph to 64 and use ASPL to guide our relational graph generation. We construct the adjacency matrix using the random_degree_sequence_graph function from the Python networkx package (version 2.4), The parameter tries is set to 1,000,000 and other parameters by default, which is empirically sufficient for successfully generating a valid adjacency matrix of a connected graph, where there must be at least one path connecting each pair of nodes. The AD, which determines the pruning rate of the generated DNN model, will affect the range of ASPL (too high or too low AD can not make ASPL reach the required length). We thus set the AD with the corresponding pruning rate (or sparsity ratio) equal to 95.3% (13/64), under which the accuracy drop does not exceed 5%. The ASPL ranges from 3 to 15 and is divided with a step of 2 (35, 57, 79, 911, 1113, 1315) to explore the relationship between adversarial sample detection accuracy and ASPL. There are 100 graphs for each segment. All the pruned models are retrained with accuracy at least equal to 85.34% and 91.35%, for CIFAR10 and SVHN, respectively.
VA3 Adversarial Sample Generation
We test GGT on detecting adversarial samples generated by six typical adversarial attack methods described in Sec. IIB, including 4 whitebox and 2 blackbox, and meanwhile those wrongly labeled samples are also considered as a kind of adversarial samples. The parameters for each attack are summarized as follows:

FGSM: the scale of perturbation is 0.03;

JSMA: the maximum distortion is 12%;

CW: adopt L2 attack, the scale coefficient is 0.6 and the iteration number is 1,000;

Deepfool (DF): the maximum number of iterations is 50 and the termination criterion is 0.02;

One Pixel Attack (OP): the number of pixels for modification is 3 (in order to ensure that enough successful samples are generated) and the differential algorithm runs with a population size of 400 and a max iteration count of 100;

Local Search Attack (LS): the pixel complexity is 1, the perturbation value is 1.5, the half side length of the neighborhood square is 5, the number of pixels perturbed at each round is 5 and the threshold for kmisclassification is 1;

Wrongly Labeled (WL): the original model accuracy and test set size determine the number of WL samples.
We choose the adversarial samples with confidence higher than 0.9 to perform the highconfidence attacks, regardless of the attack method. In addition to adversarial samples generated by these methods, as usual, we also regard the samples in the test set which are wrongly labeled by the original model as adversarial samples. We randomly select a certain number of samples from each kind of adversarial samples for evaluation, as presented in Table I.
Normal  WL  FGSM  JSMA  CW  DF  OP  LS  
CIFAR10  1000  697  1000  1000  572  1000  714  647 
SVHN  1000  1000  1000  1000  1000  1000  196  309 
VB Evaluation Metrics
We evaluate GGT in the following three ways:

Diversity Score Difference (DSD): the key to our GGT method is that normal sample and adversarial sample have significantly different LCR which is calculated by Eq. (10). Here, we define DSD as
(14) where and are the average LCR of adversarial and normal samples, respectively.

AUROC: GGT works based on the diversity score of normal samples. In order to select the best threshold and see how well it works, we calculate the Area Under the ROC (AUROC) curve to demonstrate whether the diversity score is an appropriate feature or not (the closer AUROC is to 1, the better the threshold is).

Accuracy of detection and the number of pruned models used: The higher the accuracy, the better our GGT is; the fewer pruned models needed for detection, the fewer resources it occupies.
In order to better demonstrate the performance of our method, we compare it with the MMT method proposed by Wang et al. [18] as the SOTA adversarial detection algorithm with a similar mechanism. They use four mutation operators to generate mutated models, i.e., Gaussian Fuzzing (GF), Weight Shuffling (WS), Neuron Switch (NS), and Neuron Activation Inverse (NAI), then distinguish adversarial samples by the statistical label change rate of mutated models. This method randomly mutates a trained model and the mutated models are not guaranteed to be accurate. Therefore, appropriate models should be selected from a bunch of candidates.
VC Results
Answer1: Based on our graphguided mechanism, the DNN model complexity can be significantly reduced without much accuracy loss. Therefore, the much smaller pruned models can be used for subsequent adversarial detection.
To answer the RQ1, we first generate the graphs of different AD varied in ={2,3,4,5,6,7,8,9,16,24,32,40,48,56,64}, with the degree of each node in a graph is set the same, equal to . For each , we randomly create 5 graphs, generate their corresponding DNNs, train these DNNs on CIFAR10 and SVHN, and record the average testing accuracy. The relationship between the average accuracy of DNNs and the AD of the relational graphs is shown in Fig. 6, where we notice that the accuracy of DNN indeed increases as the relational graph gets denser. Such trend is more significant when AD increases from 2 to 8, while the accuracy does not change much as AD further increases from 8 to 64. Moreover, the decrease of the accuracy is not much even when the relational graph gets very sparse, e.g., (with the pruning rate equal to 12/6497%). In this case, about 97% of parameters can be pruned out from the original DNN, with the accuracy loss only equals to 4.35% and 2.00% on CIFAR10 and SVHN, respectively. As suggested in [18], the mutated models of accuracy over 90% of the original model could be chosen for adversarial detection. Therefore we argue that all the graphguided pruned models, with the relational graph keeping connected (no matter what AD is), can be adopted for adversarial detection, due to their relatively low accuracy loss. To improve the detection efficiency, and study the effect of ASPL on the classification accuracy of the pruned model, we set in the rest of the paper, since in this case, it is easier to get relational graphs of a wide range of ASPL.
Answer2: The shorter the ASPL of the relational graph is, the higher the classification accuracy of the pruned model. Meanwhile, the pruned models of shorter ASPL are better candidates to design GGT, since they can better capture the difference between adversarial samples and normal samples.
We record the mean classification accuracy for each set of models with AD , as shown in Fig. 7. As we can see, the accuracy slightly decreases as ASPL increases for both datasets. One possible reason is that the graph with a larger ASPL has a longer information transmission path between neurons, making it more difficult to extract effective features.
To further explore the effect of ASPL on adversarial detection, we calculate the DSD of adversarial samples using 6 groups of pruned models with different ASPL, i.e., the average LCR of adversarial samples divided by that of normal samples, as represented by Eq. (14). Note that, as usual, we treat normal samples that wrongly labeled (WL) by the original model as a kind of adversarial sample [18], and the attacker will not add any perturbation. The results are summarized in Fig. 8. In summary, for all the 6 groups of pruned models, adversarial samples have much higher LCR than normal samples (more than 5 times for CIFAR10 and 10 times for SVHN), indicating that these pruned models are highly sensitive to adversarial samples. Compared with other ASPLs, the pruned models with ASPL 35 show the largest DSD between normal samples and adversarial samples and thus are better candidates to design GGT for adversarial sample detection.
Answer3: GGT outperforms MMT in both efficiency and effectiveness, i.e., GGT achieves significantly higher detection accuracy with much less DNN models, while the single pruned model in GGT is even much smaller than the single mutated model in MMT.
Here, we compare GGT with MMT under the same conditions. The number of the maximum used models is limited to 100 and other parameters of MMT are set to default according to the released code. Note that in the original paper of MMT, the model number limitation is set to 500, which we think is too huge for real applications [18]. The mutation rate we use for MMT is 0.007 for both datasets to obtain the best performance. Our baseline is GF and NAI mutation operators, which are the best performers among their proposed methods. The adversarial sample detection accuracy and the number of used models are shown in Fig. 9. It can be seen that GGT uses only 30.04 and 28.50 models on average considering all the attacks, while MMT needs 51.12 and 77.16 models, on CIFAR10 and SVHN, respectively. Note that the single pruned model in GGT is much smaller than the single mutated model in MMT. Such result suggests that GGT is much more efficient than MMT. Meanwhile, GGT achieves an average detection accuracy of 93.61% and 94.46%, on CIFAR10 and SVHN, while MMT only has 74.74% and 44.79%, respectively, indicating that GGT is also more effective than MMT in detecting adversarial samples.
In particular, we also calculate the AUROC score using LCR as the key feature to distinguish whether the input is an adversarial sample or not. The AUROC results are summarized in Table II, with the best results marked in bold. We find that GGT has a generally higher AUROC than MMT, while the pruned models with ASPL 35 behave the best, in most cases, meaning that this set of pruned models have a good tradeoff between true positives and false positives in detecting adversarial samples. This finding is in line with our expectation since this group of pruned models are of higher classification accuracy and can better capture the difference between adversarial samples and normal samples, as indicated by the results of RQ2.
Attack  MMT  GGT with different ASPL  
GF  NAI  35  57  79  911  1113  1315  
CIFAR10  
FGSM  83.89  92.14  98.35  98.10  98.09  98.15  98.20  98.05 
JSMA  97.23  97.88  99.26  99.21  99.18  99.11  99.18  99.14 
CW  85.85  91.70  97.29  96.97  96.82  96.91  97.04  96.80 
DF  98.34  97.28  99.17  98.82  98.83  98.83  98.85  98.78 
OP  99.95  99.77  99.35  99.03  98.98  99.00  99.02  98.99 
LS  86.76  93.99  96.44  96.18  96.13  96.21  96.19  95.85 
WL  85.92  92.38  93.90  93.35  93.05  93.06  93.32  93.01 
SVHN  
FGSM  68.18  87.61  97.90  97.88  97.97  98.14  98.11  98.07 
JSMA  83.76  96.25  99.31  99.27  99.27  99.28  99.32  99.24 
CW  66.82  89.22  99.46  99.40  99.41  99.44  99.44  99.40 
DF  83.47  96.59  99.74  99.68  99.66  99.69  99.68  99.66 
OP  82.33  94.97  98.90  98.67  98.57  98.67  98.67  98.48 
LS  79.27  93.30  98.42  98.27  98.26  98.28  98.25  98.11 
WL  72.64  89.02  90.81  90.75  90.51  90.77  91.07  90.73 
Answer4: Considering only highconfidence adversarial samples, the adversarial detection accuracy increases 0.69 % and drops 2.76% on CIFAR10 and SVHN, respectively, for GGT; while it drops 16.04% and 12.24% on CIFAR10 and SVHN, respectively, for MMT.
In this experiment, we choose 500 adversarial samples with confidence higher than 0.9 as highconfidence samples, regardless of which attack method is adopted. We find that, initially, MMT can detect 74.74% and 44.79% adversarial samples on CIFAR10 and SVHN, respectively, while these values decrease to 58.7% and 32.55% when detecting highconfidence adversarial samples, i.e., the overall detection accuracy drops almost 14.14%. On the other hand, GGT can still detect 94.30% and 91.70% highconfidence adversarial samples (93.61% and 94.46% initially), i.e., the overall detection accuracy only drops 1.04%. We further decrease the maximum number of models allowed in GGT and MMT, as shown in Fig. 10, and found that the detection accuracy of MMT decreases quickly when only at most 20 models are allowed, i.e., it can barely detect any highconfidence adversarial samples, while GGT still detects 61.50% and 61.40% of them. Such results indicate that our GGT can detect both highconfidence adversarial samples and lowconfidence adversarial samples with similar high accuracy, while MMT may lose its effectiveness in detecting highconfidence adversarial samples.
Vi Threats to Validity
First, the graphDNN mapping only works on CNN currently. Therefore, GGT may not be suitable for other kinds of DNN models. Nevertheless, CNN is still the mainstream deep learning method in computer vision, signal processing, and natural language processing, thus GGT is still valuable in these applications. We also focus on establishing graphDNN mapping for other DNN models to make our GGT suitable for them in the near future.
Second, just like MMT, we only validate GGT on image data, while DNN models are also widely used in processing other types of data, e.g., time series. Considering that both GGT and MMT are based on the diversity of decision boundaries created by different DNN models, it is reasonable to believe that they can also be applied to other kinds of data.
Third, we only explore two graph characteristics, AD and ASPL, to generate pruned DNN models since AD determines the sparsity of a DNN model while ASPL is related to information exchanging efficiency. Indeed, our proposed GGT based on these two characteristics has both higher efficiency and effectiveness than MMT. There are certainly many other characteristics that could be further explored in the future.
Vii Conclusion
In this paper, we establish the mapping between DNN architecture and relational graph and then prune a DNN model guided by the designed relational graph. Using this method, we can prune out more than 95% parameters with only a small loss of accuracy, and find that the accuracy of the pruned model is negatively related to the ASPL of the relational graph. Based on these, we design GraphGuided Testing (GGT) for adversarial sample detection, based on the pruned models with small AD and short ASPL in their corresponding relational graphs. The experimental results show that our GGT detects adversarial samples with 93.61% and 94.46% average accuracy, requiring 30.04 and 28.50 pruned models, on CIFAR10 and SVHN, respectively, much better than the stateoftheart Model Mutation Testing (MMT) in both accuracy and resource consumption. In our future work, we will explore the structural space of DNN and design a better pruned DNN to further improve GGT. we will also extend GGT on more kinds of deep learning models, such as Recurrent Neural Network (RNN) and Graph Neural Network (GNN), and meanwhile adopt it to detect adversarial samples in various areas, such as signal processing and graph data mining.
References
 [1] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang et al., “End to end learning for selfdriving cars,” arXiv preprint arXiv:1604.07316, 2016.
 [2] N. Carlini and D. Wagner, “Audio adversarial examples: Targeted attacks on speechtotext,” in Proceedings of the Security and Privacy Workshops. IEEE, 2018, pp. 1–7.
 [3] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, “Synthesizing robust adversarial examples,” in Proceedings of the International Conference on Machine Learning. PMLR, 2018, pp. 284–293.
 [4] S. Aneja, N. Aneja, and M. S. Islam, “Iot device fingerprint using deep learning,” in Proceedings of the International Conference on Internet of Things and Intelligence System. IEEE, 2018, pp. 174–179.
 [5] S. Riyaz, K. Sankhe, S. Ioannidis, and K. Chowdhury, “Deep learning convolutional neural networks for radio identification,” IEEE Communications Magazine, vol. 56, no. 9, pp. 146–152, 2018.
 [6] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
 [7] K. Sun, Z. Zhu, and Z. Lin, “Towards understanding adversarial examples systematically: Exploring data size, task and model factors,” arXiv preprint arXiv:1902.11019, 2019.
 [8] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
 [9] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, “Adversarial examples are not bugs, they are features,” in Advances in Neural Information Processing Systems, 2019, pp. 125–136.
 [10] Y. Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testing of deepneuralnetworkdriven autonomous cars,” in Proceedings of the International Conference on Software Engineering, 2018, pp. 303–314.
 [11] J. Kim, R. Feldt, and S. Yoo, “Guiding deep learning system testing using surprise adequacy,” in Proceedings of the International Conference on Software Engineering. IEEE, 2019, pp. 1039–1049.
 [12] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. JuefeiXu, C. Xie, L. Li, Y. Liu, J. Zhao et al., “Deepmutation: Mutation testing of deep learning systems,” in Proceedings of the International Symposium on Software Reliability Engineering. IEEE, 2018, pp. 100–111.
 [13] L. Ma, F. JuefeiXu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu et al., “Deepgauge: Multigranularity testing criteria for deep learning systems,” in Proceedings of the International Conference on Automated Software Engineering, 2018, pp. 120–131.
 [14] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in Proceedings of the Symposium on Operating Systems Principles, 2017, pp. 1–18.

[15]
Y. Tian, Z. Zhong, V. Ordonez, G. Kaiser, and B. Ray, “Testing dnn image classifiers for confusion & bias errors,” in
Proceedings of the International Conference on Software Engineering, 2020, pp. 1122–1134.  [16] J. Guo, Y. Jiang, Y. Zhao, Q. Chen, and J. Sun, “Dlfuzz: differential fuzzing testing of deep learning systems,” in Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2018, pp. 739–743.
 [17] X. Xie, L. Ma, F. JuefeiXu, M. Xue, H. Chen, Y. Liu, J. Zhao, B. Li, J. Yin, and S. See, “Deephunter: A coverageguided fuzz testing framework for deep neural networks,” in Proceedings of the International Symposium on Software Testing and Analysis, 2019, pp. 146–157.
 [18] J. Wang, G. Dong, J. Sun, X. Wang, and P. Zhang, “Adversarial sample detection for deep neural network through model mutation testing,” in Proceedings of the International Conference on Software Engineering. IEEE, 2019, pp. 1245–1256.
 [19] H. Wang, J. Xu, C. Xu, X. Ma, and J. Lu, “Dissector: Input validation for deep learning applications by crossinglayer dissection,” in Proceedings of the International Conference on Software Engineering. IEEE, 2020, pp. 727–738.
 [20] F. Tramer, N. Carlini, W. Brendel, and A. Madry, “On adaptive attacks to adversarial example defenses,” arXiv preprint arXiv:2002.08347, 2020.
 [21] F. Chen, Y.C. Wang, B. Wang, and C.C. J. Kuo, “Graph representation learning: a survey,” APSIPA Transactions on Signal and Information Processing, vol. 9, 2020.
 [22] A. Fornito, A. Zalesky, and E. Bullmore, Fundamentals of brain network analysis. Academic Press, 2016.
 [23] N. Akhtar and M. V. Ahamad, “Graph tools for social network analysis,” in Research Anthology on Digital Transformation, Organizational Change, and the Impact of Remote Work. IGI Global, 2021, pp. 485–500.
 [24] C. L. Tan, K. L. Chiew, K. S. Yong, J. Abdullah, Y. Sebastian et al., “A graphtheoretic approach for the detection of phishing webpages,” Computers & Security, vol. 95, p. 101793, 2020.

[25]
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in
Proceedings of the Conference on Computer Vision and Pattern Recognition
, 2015, pp. 1–9.  [26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
 [27] J. You, J. Leskovec, K. He, and S. Xie, “Graph structure of neural networks,” in Proceedings of the International Conference on Machine Learning. PMLR, 2020, pp. 10 881–10 891.

[28]
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,”
Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.  [29] S.M. MoosaviDezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in Proceedings of the Conference on Computer Vision and Pattern Recognition, 2016, pp. 2574–2582.
 [30] S. Yu, M. Zhao, C. Fu, J. Zheng, H. Huang, X. Shu, Q. Xuan, and G. Chen, “Target defense against linkpredictionbased attacks via evolutionary perturbations,” IEEE Transactions on Knowledge and Data Engineering, 2019.
 [31] W. E. Zhang, Q. Z. Sheng, A. Alhazmi, and C. Li, “Adversarial attacks on deeplearning models in natural language processing: A survey,” ACM Transactions on Intelligent Systems and Technology, vol. 11, no. 3, pp. 1–41, 2020.
 [32] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in Proceedings of the European Symposium on Security and Privacy. IEEE, 2016, pp. 372–387.
 [33] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in Proceedings of the Symposium on Security and Privacy. IEEE, 2017, pp. 39–57.
 [34] N. Narodytska and S. Kasiviswanathan, “Simple blackbox adversarial attacks on deep neural networks,” in Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2017, pp. 1310–1318.

[35]
J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural
networks,”
IEEE Transactions on Evolutionary Computation
, vol. 23, no. 5, pp. 828–841, 2019.  [36] Q. Guo, X. Xie, Y. Li, X. Zhang, Y. Liu, X. Li, and C. Shen, “Audee: Automated testing for deep learning frameworks,” in Proceedings of the International Conference on Automated Software Engineering. IEEE, 2020, pp. 486–498.
 [37] S. Ma and Y. Liu, “NIC: Detecting adversarial samples with neural network invariant checking,” in Proceedings of the Network and Distributed System Security Symposium, 2019.
 [38] X. Yin, S. Kolouri, and G. K. Rohde, “Adversarial example detection and classification with asymmetrical adversarial training,” arXiv preprint arXiv:1905.11475, 2019.
 [39] X. Zhang, X. Xie, L. Ma, X. Du, Q. Hu, Y. Liu, J. Zhao, and M. Sun, “Towards characterizing adversarial defects of deep learning software from the lens of uncertainty,” in Proceedings of the International Conference on Software Engineering. IEEE, 2020, pp. 739–751.

[40]
N. Carlini and D. Wagner, “Adversarial examples are not easily detected:
Bypassing ten detection methods,” in
Proceedings of the Workshop on Artificial Intelligence and Security
, 2017, pp. 3–14.  [41] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao, “Hrank: Filter pruning using highrank feature map,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1529–1538.
 [42] N. Liu, X. Ma, Z. Xu, Y. Wang, J. Tang, and J. Ye, “Autocompress: An automatic dnn structured pruning framework for ultrahigh compression rates,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 4876–4883.
 [43] L. d. F. Costa, F. A. Rodrigues, G. Travieso, and P. R. Villas Boas, “Characterization of complex networks: A survey of measurements,” Advances in physics, vol. 56, no. 1, pp. 167–242, 2007.
 [44] Q. Xuan, Y. Li, and T.J. Wu, “Optimal symmetric networks in terms of minimizing average shortest path length and their suboptimal growth model,” Physica A: Statistical Mechanics and its Applications, vol. 388, no. 7, pp. 1257–1267, 2009.
 [45] Q. Xuan and T.J. Wu, “Node matching between complex networks,” Physical Review E, vol. 80, no. 2, p. 026103, 2009.
 [46] Y. Wen, S. Li, and K. Jia, “Towards understanding the regularization of adversarial robustness on neural networks,” in Proceedings of the International Conference on Machine Learning. PMLR, 2020, pp. 10 225–10 235.
 [47] A. Wald, Sequential analysis. Courier Corporation, 2004.