1 Introduction
Convolutional neural networks (CNN) have rapidly accelerated progress in computer vision with many practical applications such as face recognition
[lawrence1997face], image classification [krizhevsky2012imagenet], document analysis [lecun1998gradient] and semantic segmentation. Recently, it has been shown that CNNs are vulnerable to adversarial attacks, where a welltrained CNN can be fooled into producing errant predictions due to tiny changes in their inputs [goodfellow2014explaining]. Many applications such as autonomous driving seek to leverage the power of CNNs. However due the opaque nature of these models there are reservations about using in safetycritical applications. Thus, there is an urgent need for formally evaluating the robustness of a trained CNN.The formal verification of deep neural networks (DNNs) has recently become a hot topic. The majority of the existing approaches focus on verifying safety and robustness properties of feedforward neural networks (FNN) with the Rectified Linear Unit activation function (ReLU). These approaches include: mixedinteger linear programming (MILP)
[lomuscio2017approach, kouvaros2018formal, dutta2017output], satisfiability (SAT) and satisfiability modulo theory (SMT) techniques [katz2017reluplex], optimization [weng2018towards, zhang2018efficient, lin2019robustness, hein2017formal, dvijotham2018dual, wu2019game], and geometric reachability [tran2019parallel, wang2018formal, xiang2018output, xiang2017reachable, xiang2018specification, singh2019abstract, singh2018fast, tran2019fm, wong2017provable]. Adjacent to these methods are property inference techniques for DNNs, which are also an important and interesting research area investigated in [8952519]. In a similar fashion, the problem of verifying the safety of cyberphysical systems (CPS) with learningenabled neural network components with imperfect plant models and sensing information [tran2019emsoft, souradeep2019, xiang2018reachable, sun2018formal, ivanov2018verisig] has recently attracted significant attention due to their real world applications. This research area views the safety verification problem in a more holistic manner by considering the safety of the whole system in which learningenabled components interact with the physical world.Although numerous tools have been proposed for neural network verification, only a handful of methods can deal with CNNs [kouvaros2018formal, singh2019abstract, singh2018fast, katz2019marabou, ruan2018global, anderson2019pldi]. Moreover, in the aforementioned techniques, only one [ruan2018global] can deal with realworld CNNs, such as VGGNet [simonyan2014very]. Their approach makes used of the concept of the distance between two images. Their optimizationbased approach computes a tight bound on the number of pixels that may be changed in an image without affecting the classification result of the network. It can also efficiently generate adversarial examples that can be used to improve the robustness of network. In a similar manner, this paper seeks to verify the robustness of realworld deep CNNs. Thus, we propose a setbased analysis method through the use of the ImageStar, a new set representation that can represent an infinite number of images. As an example, this representation can be used to represent a set of images distorted by an adversarial attack. Using the ImageStar, we propose both exact and overapproximate reachability algorithms to construct reachable sets that contain all the possible outputs of a CNN under an adversarial attack. These reachable sets are then used to reason about the overall robustness of the network. When a CNN violates a robustness property, our exact reachability scheme can construct a set of concrete adversarial examples. Our approach differs from [ruan2018global] in two ways. First, our method does not provide robustness guarantees for a network in terms of the number of pixels that are allowed to be changed (in terms of distance). Instead, we prove the robustness of the network on images that are attacked by disturbances bounded by arbitrary linear constraints. Second, our approach relies on reachable set computation of a network corresponding to a bounded input set, as opposed to a purely optimizationbased approach.
We implement the proposed method in a tool called NNV and compare it with the zonotope method used in DeepZ [singh2018fast] and the polytope method used in DeepPoly [singh2019abstract]. The experimental results show that our method is less conservative and faster than any of these approaches when verifying the robustness of CNNs. The main contributions of the paper are as follows.

The ImageStar set representation, which is an efficient representation for reachability analysis of CNNs.

The provision of exact and overapproximate reachability algorithms for constructing reachable sets and verifying robustness of CNNs.

The implementation of the ImageStar representation and reachability algorithms in NNV [tran2019fm, tran2019emsoft].

Rigorous evaluation and comparison of proposed approaches, such as zonotope [singh2018fast] and polytope [singh2019abstract] methods on different CNNs.
2 Problem formulation
The reachability problem for CNNs is the task of analyzing a trained CNN with respect to some perturbed input set in order to construct a set containing all possible outputs of the network. In this paper, we consider the reachability of a CNN that consists of a series of layers
that may include convolutional layers, fully connected layers, maxpooling layers, average pooling layers, and ReLU activation layers. Mathematically, we define a CNN with
layers as . The reachability of the CNN is defined based on the concept of reachable sets.Definition 1 (Reachable set of a CNN)
An (output) reachable set of a CNN corresponding to a linear input set is defined incrementally as:
where is a function representing the operation of the layer.
The definition shows that the reachable set of the CNN can be constructed layerbylayer. The core computation is constructing the reachable set of each layer defined by a specific operation, i.e., convolution, affine mapping, max pooling, average pooling, or ReLU.
3 ImageStar
Definition 2
An ImageStar is a tuple where is the anchor image, is a set of m images in called generator images, is a predicate, and are the height, width and number of channels of the images respectively. The generator images are arranged to form the ImageStar’s basis array. The set of images represented by the ImageStar is given as:
Sometimes we will refer to both the tuple and the set of states as . In this work, we restrict the predicates to be a conjunction of linear constraints, where, for linear constraints, ,
is the vector of
variables, i.e., , and . A ImageStar is an empty set if and only if is empty.Example 1 (ImageStar)
A gray image with a bounded disturbance applied on the pixel of the position can be described as an ImageStar depicted in Figure 1.
Remark 1
An ImageStar is an extension of the generalized star set recently defined in [bak2017simulation, bak2019numerical, tran2019formats, tran2019fm]. In a generalized star set, the anchor and the generators are vectors, while in an ImageStar, the anchor and generators are images with multiple channels. We will later show that the ImageStar is a very efficient representation for the reachability analysis of convolutional layers, fully connected layers, and average pooling layers.
Proposition 1 (Affine mapping of an ImageStar)
An affine mapping of an ImageStar with a scale factor and an offset image is another ImageStar in which the new anchor, generators and predicate are as follows.
Note that, the scale factor can be a scalar or a vector containing scalar scale factors in which each factor is used to scale one channel in the ImageStar.
4 Reachability of CNN using ImageStars
In this section, we present the reachable set computation for the convolutional layer, average pooling layer, fully connected layer, batch normalization layer, max pooling layer, and the ReLU layer with respect to an input set consisting of an ImageStar.
4.1 Reachability of a convolutional layer
We consider a twodimensional convolutional layer with following parameters: the weights , the bias
, the padding size
, the stride
, and the dilation factor where are the height, width, and the number of channels of the filters in the layer respectively. Additionally, is the number of filters. The reachability of a convolutional layer is given in the following lemma.Lemma 1
The reachable set of a convolutional layer with an ImageStar input set is another ImageStar where is the convolution operation applied to the anchor image, is the convolution operation with zero bias applied to the generator images, i.e., only using the weights of the layer.
Proof
Any image in the ImageStar input set is a linear combination of the center and basis images. For any filter in the layer, the convolution operation applied to the input image performs local elementwise multiplication of a local matrix (of all channels) containing the values of the local pixels of the image and the the weights of the filter and then combine the result with the bias to get the output for that local region. Due to the linearity of the input image, we can perform the convolution operation with the bias on the center and the convolution operation with zero bias on the basis images and then combine the result to get the output image.
Example 2 (Reachable set of a convolutional layer)
4.2 Reachability of an average pooling layer
The reachability of an average pooling layer with pooling size , padding size , and stride is given below, with its proof similar to that of the convolutional layer.
Lemma 2
The reachable set of a average pooling layer with an ImageStar input set is another ImageStar where , , is the average pooling operation applied to the anchor and generator images.
4.3 Reachability of a fully connected layer
The reachability of a fully connected layer is stated in the following lemma.
Lemma 3
Given a twodimensional fully connected layer with weight , bias , and an ImageStar input set , the reachable set of the layer is another ImageStar where , , . Note that it is required for consistency between the ImageStar and the weight matrix that , where are the height, width and number of channels of the ImageStar.
Proof
Similar to the convolutional layer and the average pooling layer, for any image in the ImageStar input set, the fully connected layer performs an affine mapping of the input image which is a linear combination of the center and the basis images of the ImageStar. Due to the linearity, the affine mapping of the input image can be decomposed into the affine mapping of the center image and the affine mapping without the bias of the basis images. The final result is the sum of the individual affine maps.
4.4 Reachability of a batch normalization layer
In the prediction phase, a batch normalization layer normalizes each input channel using the mean
and variance
over the full training set. Then the batch normalization layer further shifts and scales the activations using the offset and the scale factor that are learnable parameters. The formula for normalization is as follows.where is a used to prevent division by zero. The batch normalization layer can be described as a tuple . The reachability of a batch normalization layer with an ImageStar input set is given in the following lemma.
Lemma 4
The reachable set of a batch normalization layer with an ImageStar input set is another ImageStar where:
Proof
The reachable set of a batch normalization layer can be obtained in a straightforward fashion using two affine mappings of the ImageStar input set.
4.5 Reachability of a max pooling layer
Reachability of max pooling layer with an ImageStar input set is challenging because the value of each pixel in an image in the ImageStar depends on the predicate variables . Therefore, the local max point when applying maxpooling operation may change with the values of the predicate variables. In this section, we investigate the exact reachability and overapproximate reachability of a max pooling layer with an ImageStar input set. The first obtains the exact reachable set while the second constructs an overapproximate reachable set.
4.5.1 Exact reachability of a max pooling layer
The central idea in the exact analysis of the maxpooling layer is finding a set of local max point candidates when we apply the max pooling operation on the image. We consider the max pooling operation on the ImageStar in Example 1 with a pool size of , a padding size of , and a stride to clarify the exact analysis stepbystep. First, the maxpooling operation is applied on local regions , as shown in Figure 4. The local regions have only one max point candidate whic is the pixel that has the maximum value in the region. It is interesting to note that region has two max point candidates at the positions and and these candidates correspond to different conditions of the predicate variable . For example, the pixel at the position is the max point if and only if . Note that with , we always have . Since the local region has two max point candidates, and other regions have only one, the exact reachable set of the maxpooling layer is the union of two new ImageStars and . In the first reachable set , the max point of the region is with an additional constraint on the predicate variable . For the second reachable set , the max point of the region is with an additional constraint on the predicate variable . One can see that from a single ImageStar input set, the output reachable set of the maxpooling layer is split into two new ImageStars. Therefore, the number of ImageStars in the reachable set of the maxpooling layer may grow quickly if each local region has more than one max point candidates. The worstcase complexity of the number of ImageStars in the exact reachable set of the maxpooling layer is given below.
Lemma 5
The worstcase complexity of the number of ImageStars in the exact reachability of the max pooling layer is where is the size of the ImageStar output sets, and is the size of the maxpooling layer.
Proof
An image in the ImageStar output set has pixels in each channel. For each pixel, in the worst case, there are candidates. Therefore, the number of ImageStars in the output set in the worst case is .
Finding a set of local max point candidates is the core computation in the exact reachability of maxpooling layer. To optimize this computation, we divide the search for the local max point candidates into two steps. The first one is to estimate the ranges of all pixels in the ImageStar input set. We can solve linear programming optimizations to find the exact ranges of these pixels, where is the size of the input set. However, unfortunately this is a timeconsuming computation. For example, if a single linear optimization can be done in seconds, for an ImageStar of the size , we need about hours to find the ranges of all pixels. To overcome this bottleneck, we quickly estimate the ranges using only the ranges of the predicate variables to get rid of a vast amount of nonmaxpoint candidates. In the second step, we solve a much smaller number of LP optimizations to determine the exact set of the local max point candidates and then construct the ImageStar output set based on these candidates.
Lemma 5 shows that the number of ImageStars in the exact reachability analysis of a maxpooling layer may grow exponentially. To overcome this problem, we propose the following overapproximate reachability method.
4.5.2 Overapproximate reachability of a max pooling layer
The central idea of the overapproximate analysis of the maxpooling layer is that if a local region has more than one max point candidates, we introduce a new predicate variable standing for the max point of that region. We revisit the example introduced earlier in the exact analysis to clarify this idea. Since the first local region has two max point candidates, we introduce new predicate variable to represent the max point of this region by adding three new constraints: 1) , i.e., must be equal or larger than the value of the first candidate ; 2) , i.e., must be equal or larger than the value of the second candidate; 3) , i.e., must be equal or smaller than the upper bound of the pixels values in the region. With the new predicate variable, a single overapproximate reachable set can be constructed in Figure 5.
Lemma 6
The worstcase complexity of the new predicate variables introduced in the overapproximate analysis is where is the size of the ImageStar output set.
4.6 Reachability of a ReLU layer
Similar to maxpooling layer, the reachability analysis of a ReLU layer is also challenging because the value of each pixel in an ImageStar may be smaller than zero or larger than zero depending on the values of the predicate variables (). In this section, we investigate the exact and overapproximate reachability algorithms for a ReLU layer with an ImageStar input set. The techniques we use in this section are adapted from in [tran2019fm].
4.6.1 Exact reachability of a ReLU layer
The central idea of the exact analysis of a ReLU layer with an ImageStar input set is performing a sequence of stepReLU operations over all pixels of the ImageStar input set. Mathematically, the exact reachable set of a ReLU layer can be computed as follows.
where is the total number of pixels in the ImageStar input set . The operation determines whether or not a split occurs at the pixel. If the pixel value is larger than zero, then the output value of that pixel remains the same. If the pixel value is smaller than zero than the output value of that pixel is reset to be zero. The challenge is that the pixel value depends on the predicate variables. Therefore, there is the case that the pixel value may be negative or positive with an extra condition on the predicate variables. In this case, we split the input set into two intermediate ImageStar reachable sets and apply the ReLU law on each intermediate reach set. An example of the stepReLU operation on an ImageStar is illustrated in Figure 6. The value of the first pixel value would be larger than zero if , and in this case we have . If , then . Therefore, the first stepReLU operation produces two intermediate reachable sets and , as shown in the figure. The number of ImageStars in the exact reachable set of a ReLU layer increases quickly along with the number of splits in the analysis, as stated in the following lemma.
Lemma 7
The worstcase complexity of the number of ImageStars in the exact analysis of a ReLU layer is , where is the number of pixels in the ImageStar input set.
Proof
There are local regions in the approximate analysis. In the worst case, we need to introduce a new variable for each region. Therefore, the worst case complexity of new predicate variables introduced is .
Similar to [tran2019fm], to control the explosion in the number of ImageStars in the exact reachable set of a ReLU layer, we propose an overapproximate reachability algorithm in the following.
4.6.2 Overapproximate reachability of a ReLU layer
The idea behind the overapproximate reachability of ReLU layer is replacing the stepReLU operation at each pixel in the ImageStar input set by an approxStepReLU operation. At each pixel where a split occurs, we introduce a new predicate variable to overapproximate the result of the stepReLU operation at that pixel. An example of the overStepReLU operation on an ImageStar is depicted in Figure 7 in which the first pixel of the input set has the ranges of indicating that a split occurs at this pixel. To avoid this split, we introduce a new predicate variable to overapproximate the exact intermediate reachable set (i.e., two blue segments in the figure) by a triangle. This triangle is determined by three constraints: 1) (the for any ); 2) ( for any ); 3) (upper bound of the new predicate variable). Using this overapproximation, a single intermediate reachable set is produced as shown in the figure. After performing a sequence of approxStepReLU operations, we obtain a single overapproximate ImageStar reachable set for the ReLU layer. However, the number of predicate variables and the number of constraints in the obtained reachable set increase.
Lemma 8
The worst case complexity of the increment of predicate variables and constraints is and respectively, where is the number of pixels in the ImageStar input set.
Proof
In the worst case, splits occur at all pixels in the ImageStar input set. In this case, we need to introduce new predicate variables to overapproximate the exact intermediate reachable set. For each new predicate variable, we add new constraints.
One can see that determining where splits occur is crucial in the exact and overapproximate analysis of a ReLU layer. To do this, we need to know the ranges of all pixels in the ImageStar input set. However, as mentioned earlier, the computation of the exact range is expensive. To reduce the computation cost, we first use the estimated ranges of all pixels to get rid of a vast amount of nonsplitting pixels. Then we compute the exact ranges for the pixels where splits may occur to compute the exact or overapproximate reachable set of the layer.
4.7 Reachabilty algorithm and parallelization
We have presented the core ideas for reachability analysis of different types of layers in a CNN. The reachable set of a CNN is constructed layerbylayer in which the output reachable set of the previous layer is the input for the next layer. For the convolutional layer, average pooling layer and fully connected layer, we always can compute efficiently the exact reachable set of each layer. For the max pooling layer and ReLU layer, we can compute both the exact and the overapproximate reachable sets. However, the number of ImageStars in the exact reachable set may grow quickly. Therefore, in the exact analysis, a layer may receive multiple input sets which can be handled in parallel to speed up the computation time. The reachability algorithm for a CNN is summarized in Algorithm 4.1. The detail implementation of the reachability algorithm for each layer can be found in NNV [tran2019fm].
5 Evaluation
The proposed reachability algorithms are implemented in NNV [tran2019parallel, tran2019fm, tran2019emsoft], a tool for verification of deep neural networks and learningenabled autonomous CPS. NNV utilizes core functions in MatConvNet [vedaldi2015matconvnet] for the analysis of the convolutional and average pooling layers. The evaluation of our approach consists of two parts. First, we evaluate our approach in comparison with the zonotope [singh2018fast] and polytope methods [singh2019abstract]
reimplemented in NNV via robustness verification of deep neural networks. Second, we evaluate the scalability of our approach and the DeepPoly polytope method using realworld image classifiers, VGG16, and VGG19
[simonyan2014very]. The experiments are done on a computer with following configurations: Intel Core i76700 CPU @ 3.4GHz 8 Processor, 62.8 GiB Memory, Ubuntu 18.04.1 LTS OS.5.1 Robustness Verification of MNIST Classification Networks
We compare our approach with the zonotope and polytope methods in two aspects including verification time and conservativeness of the results. To do that, we train 3 CNNs a small, a medium, and a large CNN with and accuracy respectively using the MNIST data set consisting of images of handwritten digits with a resolution of pixels [lecun1998mnist]. The network architectures are given in Figure 13 in the Appendix. The networks classify images into ten classes: . The classified output is the index of the dimension that has maximum value, i.e., the argmax across the outputs. We evaluate the robustness of the network under the wellknown brightening attack used in [gehr2018ai2]. The idea of a brightening attack is that we can change the value of some pixels independently in the image to make it brighter or darker to fool the network, to misclassify the image. In this case study, we darken a pixel of an image if its value (between 0 and 255) is larger than a threshold , i.e., . Mathematically, we reduce the value of that pixel to the new value such that .
The robustness verification is done as follows. We select images that are correctly classified by the networks and perform the brightening attack on these, which are then used to evaluate the robustness of the networks. A network is robust to an input set if, for any attacked image, this is correctly classified by the network. We note that the input set contains an infinite number of images. Therefore, to prove the robustness of the network to the input set, we first compute the output set containing all possible output vectors of the network using reachability analysis. Then, we prove that in the output set, the correctly classified output always has the maximum value compared with other outputs. Note that we can neglect the softmax and classoutput layers of the networks in the analysis since we only need to know the maximum output in the output set of the last fully connected layer in the networks to prove the robustness of the network.
We are interested in the percentage of the number of input sets that a network is provably robust and the verification times of different approaches under different values of and . When is small, the number of pixels in the image that are attacked is large and vice versa. For example, the average number of pixels attacked (computed on cases) corresponding to , and are , and respectively. The value of dictates the size of the input set that can be created by a specific attack. Stated differently it dictates the range in which the value of a pixel can be changed. For example, if and , the value of an attacked pixel many range from to .
The experiments show that using the zonotope method, we cannot prove the robustness of any network. The reason is that the zonotope method obtains very conservative reachable sets. Figure 8 illustrates the ranges of the outputs computed by our ImageStar (approximate scheme), the zonotope and polytope approaches when we attack a digit image with brightening attack in which and . One can see that, using ImageStar and polytope method, we can prove that the output corresponding to the digit is the one that has a maximum value, which means that the network is robust in this case. However, the zonotope method produces very large output ranges that cannot be used to prove the robustness of the network. The figure also shows that our ImageStar method produces tighter ranges than the polytope method, which means our result is less conservative than the one obtained by the polytope method. We note that the zonotope method is very timeconsuming. It needs seconds to compute the reachable set of the network in this case, while the polytope method only needs seconds, and our approximate ImageStar method needs seconds. The main reason is that the zonotope method introduces many new variables when constructing the reachable set of the network, which results in the increase in both computation time and conservativeness.
The comparison of the polytope and our ImageStar method is given in Tables 1, 2, and 3. The tables show that in all networks, our method is less conservative than the polytope approach since the number of cases that our approach can prove the robustness of the network is larger than the one proved by the polytope method. For example, for the small network, for and , we can prove cases while the polytope method can prove cases. Importantly, the number of cases proved by DeepPoly reduces quickly when the network becomes larger. For example, for the case that and , the polytope method is able to prove the robustness of the medium network for cases while our approach can prove cases. This is because the polytope method becomes more and more conservative when the network or the input set is large. The tables show that the polytope method is faster than our ImageStar method on the small network. However, it is slower than the ImageStar method on any larger networks in all cases except for and of the medium network in which our method is slight slower than the polytope method but we can prove the robustness of a much larger number of cases ( versus cases). Notably, for the large network, the ImageStar approach is significantly faster than the polytope approach, times faster in average. The results also show that the polytope approach may run into memory problem for some large input sets.
Robustness Results (in Percent)  
Polytope  ImageStar  Polytope  ImageStar  Polytope  ImageStar  
Verification Times (in Seconds)  
Robustness Results (in Percent)  
Polytope  ImageStar  Polytope  ImageStar  Polytope  ImageStar  
Verification Times (in Seconds)  
Robustness Results (in Percent)  
Polytope  ImageStar  Polytope  ImageStar  Polytope  ImageStar  
Verification Times (in Seconds)  
5.2 Robustness Verification of VGG16 and VGG19
In this section, we evaluate the polytope and ImageStar methods on realworld CNNs, the VGG16 and VGG19 classification networks [simonyan2014very]. We use Foolbox [rauber2017foolbox] to generate the wellknown DeepFool adversarial attacks [moosavi2016deepfool] on a set of bell pepper images. From an original image , Foolbox generates an adversarial image that can fool the network. The difference between two images is defined by . We want to verify if we apply percent of the attack on the original image, whether or not the network classifies the disturbed images correctly. The set of disturbed images can be represented as an ImageStar as follows , where is the percentage of the attack at which we want to verify the robustness of the network, and is a small perturbation around , i.e., . Intuitively, describes how close we are to the attack, and the perturbation represents the size of the input set.
Table 4 shows the verification results of VGG16 and VGG19 with different levels of the DeepFool attack. The networks are robust if they classify correctly the set of disturbed images as bell peppers. To guarantee the robustness of the networks, the output corresponding to the bell pepper label (index ) needs to be the maximum output compared with others. The table shows that with a small input set, small , the polytope and ImageStar can prove the robustness of VGG16 and VGG19 with a reasonable amount of time. Notably, the verification times as well as the robustness results of the polytope and ImageStar methods are similar when they deal with small input sets except for two cases where ImageStar is faster than the polytope method. It is interesting to note that according to the verification results for the VGG and MNIST networks, deep networks seem to be more robust than shallow networks.
Robustness Results (in percentage)  
VGG16  VGG19  
Verification Times (in Seconds)  
5.3 Exact Analysis vs. Approximate Analysis
We have compared our ImageStar approximate scheme with the zonotope and polytope approximation methods. It is interesting to investigate the performance of ImageStar exact scheme in comparison with the approximate one. To illustrate the advantages and disadvantages of the exact scheme and approximate scheme, we consider the robustness verification of VGG16 and VGG19 on a single ImageStar input set created by an adversarial attack on a bell pepper image. The verification results are presented in Table 5. The table shows that for a small perturbation , the exact and overapproximate analysis can prove the robustness of the VGG16 around some specific levels of attack in approximately one minute. We can intuitively verify the robustness of the VGG networks via visualization of their output ranges. An example of the output ranges of VGG19 for the case of is depicted in Figure 9. One can see from the figure that the output of the index corresponding to the bell pepper label is always the maximum one compared with others, which proves that VGG19 is robust in this case. From the table, it is interesting that VGG19 is not robust if we apply of the attack. Notably, the exact analysis can give us correct answers with a counterexample set in this case. However, the overapproximate analysis cannot prove that VGG19 is not robust since its obtained reachable set is an overapproximation of the exact one. Therefore, it may be the case that the overapproximate reachable set violates the robustness property because of its conservativeness. A counterexample generated by the exact analysis method is depicted in Figure 10 in which the disturbed image is classified as strawberry instead of bell pepper since the strawberry output is larger than the bell pepper output in this case.
VGG16  VGG19  
Exact  Approximate  Exact  Approximate  
Robust  VT  Robust  VT  Robust  VT  Robust  VT  
50%  Yes  64.56226  Yes  60.10607  Yes  234.11977  Yes  72.08723  
Yes  63.88826  Yes  59.48936  Yes  1769.69313  Yes  196.93728  
80%  Yes  64.92889  Yes  60.31394  Yes  67.11730  Yes  63.33389  
Yes  64.20910  Yes  59.77254  Yes  174.55983  Yes  200.89500  
95%  Yes  67.64783  Yes  59.89077  Yes  73.13642  Yes  67.56389  
Yes  63.83538  Yes  59.23282  Yes  146.16172  Yes  121.91447  
97%  Yes  64.30362  Yes  59.79876  Yes  77.25398  Yes  64.43168  
Yes  64.06285  Yes  61.23296  Yes  121.70296  Yes  107.17331  
98%  Yes  64.06183  Yes  59.89959  No  67.68139  Unkown  64.47035  
Yes  64.01997  Yes  59.77469  No  205.00939  Unknown  107.42679  
98.999%  Yes  64.24773  Yes  60.22833  No  71.90568  Unknown  68.25916  
Yes  63.67108  Yes  59.69298  No  106.84492  Unknown  101.04668 
To optimize the verification time, it is important to know the times consumed by each type of layers in the reachability analysis step. Figure 11 described the total reachability times of the convolutional layers, fully connected layers, max pooling layers and ReLU layers in the VGG19 with attack and perturbation. As shown in the figure, the reachable set computation in the convolutional layers and fully connected layers can be done very quickly, which shows the advantages of the ImageStar data structure. Notably, the total reachability time is dominated by the time of computing the reachable set for max pooling layers and ReLU layers. This is because the computation in these layers concerns solving a large number of linear programing (LP) optimization problems such as finding lower bound and upper bound, and checking max point candidates. Therefore, to optimize the computation time, we need to minimize the number of LP problems in the future.
6 Discussion
When we apply our approach on realworld networks, it has been shown that the size of the input set is the most important factor that affects the performance of verification approaches. However, this important issue has not been emphasized in the existing literature. Most of the existing approaches focus on the size of the network that they can analyze. We believe that all methods (including the method we proposed in this paper) are scalable for large networks only for small input sets. When the input set is large, it causes three major problems in the analysis, which are the explosions in 1) computation time; 2) memory usage; and 3) conservativeness. In the exact analysis method, a large input set causes more splits in the maxpooling layer and the ReLU layer. A single ImageStar may split into many new ImageStars after these layers, which leads to the explosion in the number of ImageStars in the reachable set as shown in Figure 12. Therefore, it requires more memory to handling the new ImageStars and more time for the computation. One may think that the overapproximate method can overcome this challenge since it obtains only one ImageStar at each layer and the cost we need to pay is only the conservativeness of the result. The fact is, an overapproximate method usually helps reduce the computation time, as shown in the experimental results. However, it is not necessarily efficient in terms of memory consumption. The reason is, if there is a split, it introduces a new predicate variable and new generator. If the number of generators and the dimensions of the ImageStar are large, it requires a massive amount of memory to store the overapproximate reachable set. For instance, if there are splits happened in the first ReLU layer of the VGG19, the second convolutional layer will receive an ImageStar of size with generators. To store this ImageStar with double precision, we need approximately of memory. In practice, the dimensions of the ImageStars obtained in the first several convolutional layers are usually large. Therefore, if splitting happens in these layers, we may need to deal with “out of memory” problem. We see that all existing approaches such as the zonotope [singh2018fast] and polytope [singh2019abstract], all face the same challenges. Additionally, the conservativeness of an overapproximate reachable set is a crucial factor in evaluating an overapproximation approach. Therefore, the exact analysis still plays an essential role in the analysis of neural networks since it helps to evaluate the conservativeness of the overapproximation approaches.
7 Conclusion
We have proposed a new setbased method for robustness verification of deep CNNs using the concept of the ImageStar. The core of our method are the exact and overapproximate reachability algorithms for ImageStar input sets. The experiments show that our approach is less conservative than the recent zonotope [singh2018fast] and polytope [singh2019abstract] approaches. It is also faster than these approaches when dealing with deep networks. Notably, our approach can be applied to verify the robustness of realworld CNNs with small perturbed input sets. It can also compute the exact reachable set and visualize the exact output range of deep CNNs, and the analysis can sped up significantly using parallel computing. We have found and shown the size of the input set to be an important factor that affects the reachability algorithms performance. Our future work is improving the proposed method to deal with larger input sets and optimizing the memory and time complexity of our computations.
Comments
There are no comments yet.