The deployment of Artificial Neural Networks (ANNs) in safety-critical applications such as medical image processing or semi-autonomous vehicles poses a number of new assurance, verification, and certification challenges [2, 5]. For ANN-based end-to-end steering control of self-driving cars, for example, it is important to know how much noisy or even maliciously manipulated sensory input is tolerated . Here we are addressing these challenges by establishing maximum and verified bounds for the resilience of given ANNs on these kinds of input disturbances.
More precisely, we are defining and computing safe perturbation bounds for multi-class ANN classifiers. This measure compares the relative ratio-ordering of multiple, so-called softmax
output neurons for capturing scenarios where one only wants to consider inputs that classify to a certain class with high probability. The problem of finding minimal perturbation bounds is reduced to solving a correspondingmixed-integer programming (MIP). In particular, the encoding of some non-linear functions such as ReLU and max-pooling nodes require the introduction of integer variables. These integer constraints are commonly handled by off-the-shelf MIP-solvers such as CPLEX111https://www-01.ibm.com/software/commerce/optimization/cplex-optimizer/ which are based on branch-and-bound algorithms. In the MIP reduction, a number of nonlinear expressions are linearized using a variant of the well-known big-  encoding strategy. We also define a dataflow analysis  for generating relatively small big- as the basis for speeding up MIP solving. Other important heuristics in encoding the MIP problem include the usage of solving several substantially simpler MIP problems for speeding up the overall generation of satisfying instances by the solver. Lastly, branch-and-bound is run in parallel on a number of computing cores.
We demonstrate the effectiveness and scalability of our approach and encoding heuristics by computing maximum perturbation bounds for benchmark sets such as MNIST  and agent games . These cases studies include ANNs for image recognition and for high-level maneuver decisions for autonomous control of a robot. Using the heuristic encodings outlined above we experienced a speed-up of about two orders of magnitude compared with vanilla MIP encodings. Moreover, parallelization of branch-and-bound  on different computing cores can yield, up to a certain threshold, linear speed-ups using a high-performance parallelization framework.
The practical advantages of our approach for validating and qualifying ANNs for safety-relevant applications are manifold. First, perturbation bounds provide a formal interface between sensor sets and ANNs in that they provide a maximum tolerable bound on possible sensor errors. These assume-guarantee interfaces therefore form the basis for decoupling the design of sensor sets from the design of the classifier itself. Second, our method also computes minimally perturbed inputs of different classification, which might be included into ANN training sets for potentially improving classification results. Third, maximum perturbation bounds are a useful measure of the resilience of an ANN towards (adversarial) perturbation, and also for objectively comparing different ANNs. Last, large perturbation bounds are intuitively inversely related with the problem of overfitting, that is poor generalization to new inputs, which is a common issue with ANNs.
An overview of concrete problems and various approaches to the safety of machine learning is provided in. We compare our results only with work that is most closely related to ours. Techniques including the generation of test cases [17, 18, 9] or strengthening the resistance of a network with respect to adversarial perturbation  are used for validating and improving ANNs. In contrast to our work, these methods do not actually establish verified properties on the input-output behavior of ANNs. Formal methods-based approaches for verifying ANNs include abstraction-refinement based approaches , bounded model checking for neural network for control problems  and neural network verification using SMT solvers or other specialized solvers [21, 13, 11]. Instead we rely on solving MIP problems and parallelization of branch-and-bound algorithms. In contrast to previous approaches we also go beyond verification and solve optimization problems for ANNs for establishing maximum perturbation bounds. These kinds of problems might also be addressed in SMT-based approaches either by using binary search over SMT or by using SMT solvers that support optimization such as , but it is not clear how well these approaches scale to complex ANNs. Recent work also targets ReLU  or application of a single image [11, 3] (point-wise robustness or computing measures by taking samples). Our proposed resilience measure for ANNs goes beyond [13, 11, 3] in that it applies to multi-classification network using the softmax descriptor. Moreover, our proposed measure is a property of the classification network itself rather than just a property of a single image (as in ) or by only taking samples from the classifier without guarantee (as in ).
The paper is structured as follows. Section 2 reviews the foundations of feed-forward ANNs. Section 3 presents an encoding of various neurons in terms of linear constraints. Section 4 defines our measure for quantifying the resilience of an ANN, that is, its capability to tolerate random or even adversarial input perturbations. Section 5 summarizes our MIP encoding heuristics for substantially increasing the performance of the MIP-solver in establishing in minimal perturbation bounds of ANN. Finally, we present the results of some of our experiments in Section 6, and we describe possible improvements and extensions in Section 7.
We introduce some basic concepts of feed-forward artificial neural networks (ANN) . These networks consist of a sequence of layers labeled from , where is the index of the input layer, is the output layer, and all other layers are so-called hidden layers. For the purpose of this paper we assume that each input is of bounded domain. Superscripts are used to index layer -specific variables, but these superscripts may be omitted for input layers. Layers are comprised of nodes (so-called neurons), for , where is the dimension of the layer . By convention nodes of index have a constant output ; these bias nodes are commonly used for encoding activation thresholds of neurons. In a feed-forward net, nodes of layer are connected with nodes in layer by means of directed edges of weight . For the purpose of this paper we are assuming that all weights in a network have fixed values, since we do not consider re-learning. Figure 2 illustrates a small feed-forward network structure with four layers, where each layer comes with a different type of node functions, which are also main ingredients of convolutional neural networks. These node functions are specified in Figure 2. The first hidden layer of the network in Figure 2 is a fully-connected ReLU layer. Node , for example, computes the weighted linear sum of all inputs from the previous layer as , and outputs the maximum of and this weighted sum. The second hidden layer is using max-pooling for down-sampling an input representation by reducing its dimensionality; node , for example, just outputs the maximum of its inputs. Node in the output layer applies the sigmoid-shaped on the weighted linear input sum.
Given an input to the network these node functions are applied successively from layer to for computing the corresponding network output at layer . For to we use to denote the output value of node and denotes the output value for the input , sometimes abbreviated by .
For the purpose of multi-class classification, outputs in layer are often transformed into a probability distribution by means of the softmax function
In this way, the output is interpreted as the probability of the input to be in class . For the inputs , , of the nodes in Figure 3, for example, the corresponding outputs for sum up to .
3 Arithmetic Encoding of Artificial Neural Networks
In a first step, we are encoding the behavior of ANNs in terms of linear arithmetic constraints. In addition to  we are also considering , max-pooling and softmax
nodes as commonly found in many ANNs in practice. These encodings are based on the input-output behavior of every node in the network, and the main challenge is to handle the non-linearities, which are arising from non-linear activation functions (e.g., ReLU and), max-pooling and softmax nodes.
Constraints for ReLU and nodes as defined in Figure 2 are separated into, first, an equality constraint (1) for the intermediate value and, second, several linear constraints for encoding the non-linear behavior of these nodes.
We now describe the encoding of the non-linear functions ( or ).
Encoding ReLU activation function.
The non-linearity in ReLU constraints is handled using the well-known big- method , which introduces a binary integer variable together with a positive constant such that and for all possible values of and . A derivation of the following reduction is listed in the appendix.
The efficiency of a MIP-solver via big- encoding heavily depends on the size of , because MIP-solvers typically relax binary integer variables to real-valued variables, resulting in a weak LP-relaxation for large big-s. It is therefore essential to choose relatively small values for . We apply static analysis  based on interval arithmetic for propagating the bounded input values through the network, as the basis for generating “good” values for .
The output of a max-pooling node is rewritten as , where and . Encoding the function into MIP constraints is accomplished by introducing three binary integer variables to encode using the big- method.
Property-directed encoding of softmax.
The exponential function in the definition of softmax, of course, can not be encoded into a linear MIP constraint. However, using the proposition below, one confirms that if the property to be analyzed does not consider the concrete value of output values from neurons but only the ratio ordering, then (1) it suffices to omit the construction of the output layer, and (2) one may rewrite the property by replacing each by .
Given a feed-forward ANN with softmax output layer and a constant , then for all :
This equivalence is simply derived by using the definition of softmax, multiplying by the positive denominator, and by applying the logarithm and the resulting inequality. The derivation is listed in the appendix.
Encoding with error bounds.
The handling of non-linearity in is based on results in digital signal processing for piece-wise approximating with quadratic constraints and error bounds. In case the quadratic approximation methods (Eq. (7) of ) are used, and is approximated by with a maximum error smaller than . The absolute value in the formula is removed by encoding case splits between and using big- methods. Otherwise, when considering the case or , the symmetry condition of  states that (1) if then , and (2) if then . This implies that we can create a variable with a constraint that , i.e., variable is the inverse of im. By utilizing the fact that , the value of can be computed by the formula in (i).
Moreover, case splits are encoded using the big- method as outlined above. Since quadratic terms are used, our approach for handling nodes requires solving mixed integer quadratic constraint problem (MIQCP) problems.
Using these approximations for , we obtain lower and upper bounds for the value of the node variable , where the interval between lower and upper bound is determined by the approximation error of . Since the approximation error propagates through the network and using lower and upper bounds instead of an equality constraint relaxes the problem, our method computes approximations for the measure when it is used for ANNs with as activation function.
Pre-processing based on dataflow analysis.
We use interval arithmetic to obtain relatively small values for big-, in order to avoid a weak LP-relaxation of the MIP. Interval bounds for the values of are denoted by . We are assuming that all input values (at layer ) are bounded, and the output of bias nodes is restricted by the singleton (the value of the bias is given by the weight of a bias node). Interval bounds for the values of node outputs are obtained from the interval bounds of connected nodes from the previous layers by means of interval arithmetic.
The output of ReLU nodes is defined by and the ReLU function . Therefore, interval bounds for are computed by first considering the interval bounds and , which are determined by weights of the linear sum and the bounds on . The bounds and are obtained from interval arithmetic as follows:
Given and the bounds on are derived using the definition of ReLU, i.e.,
is directly encoded using linear constraints, which reduces the number of binary variables.
In the case of max-pooling nodes, the output is simply the maximum of its four inputs. Therefore, the bounds and on the output are given by the maximum of the lower and uppers bounds of the four inputs respectively. Interval bounds of the outputs for are obtained using a polynomial approximation for (see below). Finally, the output of softmax nodes is a probability in which might also be further refined using interval arithmetic. These bounds on softmax nodes, however, are not used in our encodings, because of the property-driven encoding of softmax output layers as described previously.
4 Perturbation Bounds
We define concrete measures for quantifying the resilience of multi-classification neural networks with softmax output neurons. This measure for resilience is defined over all possible inputs of the network. In particular, our developments do not depend on probability distributions of training and test data as in previous work . Maximum resilience of these ANNs is obtained by means of solving corresponding MIP problems (or MIQCPs in the case of activation functions).
We illustrate the underlying principles of maximum resilience using examples from the MNIST database for digit recognition of input images (see Fig. 6). Input images in MNIST are of dimension
and are represented as a vector. Input layers of ANN-based multi-digit classifiers for MNIST therefore consist of input neurons, and the output layer is comprised of 10 softmax neurons. Let the output at the last layer be the computed probabilities for an input image to be classified to characters ‘0’ to ‘9’.
To formally define a perturbation, we allow each input () to have a small disturbance , so the input after perturbation is . We sometimes use the concise notation of for the perturbed input. The global value of the perturbation is obtained by taking the sum of the absolute values of each disturbance , i.e., .
Definition 1 (Maximum Perturbation Bound for -th classifier)
For a given ANN with neurons in a softmax output layer and given constants and , we define the maximum perturbation bound for the -th classifier, denoted by ,222For clarity, we usually omit the dependency of from . to be the maximum value such that:For all inputs where on all other classes , we have that for all perturbations where , there exist at most classes such that .
Intuitively, the bound guarantees that for all inputs that strongly (defined by ) classify to class , if the total amount of perturbation is limited to a value strictly below , then either (1) the perturbed input can still be classified as , or (2) the probability of classifying to is among the highest probabilities. Dually, is the smallest value such that there exists an input that originally classifies to , for which the computed probability for class may not be among the highest after being perturbed with value greater than or equal to . Fig. 6 illustrates an example of an MNIST image being perturbed, where the neural network considers the perturbed image to be ‘0’ or ‘3’ with at least the probability of being a ‘5’. The “not among the highest” property is an indicator that the confidence of classifying to class has decreased under perturbation, as the perturbed input can be interpreted as at least other classes. In our experiment evaluations below we used the fixed value .
Constant may be interpreted as indicating the level of confidence of being classified to a class . When setting to , the analysis takes all inputs for which the probability of class is greater than or equal to the probabilities of the other classes. Since there might exist an image that has the same probability for all classes, setting may result in a maximum perturbation of zero. Increasing helps to avoid this effect, because it requires that at most other classes have probabilities greater than or equal to the probility of . By picking an low-confidence inputs are removed and part (II) of Definition 1 forces the perturbation to be greater than zero. E.g., assume if point B in Fig. 6 is classified to ‘5’ with probability and to ‘0’ with probability , then even by setting , point B will not be considered in the analysis. By setting to one already only considers inputs that classifies to with probability higher than 0.95.
Provided that can be computed for each class (as shown below), one defines a measure for safe perturbation by taking the minimum of all , and the measure is computed by computing each independently.
Definition 2 (Perturbation Bound for ANN)
For an ANN with layers and softmax neurons in the output layer, a given , , and the perturbation bound for the -th classifier of this ANN from Definition 1, the perturbation bound for ANN is defined as .
Based on the dual interpretation above of Definition 1 we are now ready to encode the problem of finding in terms of the following optimization problem, where and .
The first set of conjunctive constraints specifies that the input strongly classifies to (i.e., satisfies condition I in Def. 1), while the second set of disjunctive constraints specifies that by feeding the image after perturbation, the neural network outputs that at least classes in are more likely (or equally likely) than class (i.e., the second condition in Def. 1 is violated). Therefore, for input and its associated perturbation , we have that . By computing the minimum objective of satisfying the constraints we obtain .
We now address the following issues in order to transform optimization problem (4) into a MIP: (1) the objective is not linear due to the introduction of the absolute value function, (2) the non-linearity of softmax due to the function , and (3) the disjunction in the second set of constraints.
(i) Transforming objectives.
(ii) Removing softmax output layer.
(iii) Transforming disjunctive constraints.
disjunctive constraint in problem (4) guarantees at least classifications with probability equal or higher as .
We rewrite it by introducing a binary variable for each class . Then we use (1) an integer constraint to select classifications
and (2) the big- method to enforce that if classification is selected (i.e., ), the probability of classifying to is higher or equal to the probability of classifying to .
By applying the transformations (i)-(iii) to the optimization problem (4) we obtain problem (4), which is a MIP, and it follows from Proposition 3 that maximum perturbations bounds can be obtained by solving the MIP in (4).
For a given and , the optimum of the MIP in (4) equals for ANNs with ReLU nodes and softmax output layer. For ANNs using it yields an under-approximation.
5 Heuristic Problem Encodings
We list some simple but essential heuristics for efficiently solving MIP problems for the verification of ANNs. Notice that these heuristics are not restricted to computing the resilience of ANNs, and may well be applicable for other verification tasks involving ANNs.
1. Smaller big-s by looking back at multiple layers.
The dataflow analysis in Section 3 essentially views neurons at the same layer to be independent. Here we propose a more fine-grained analysis by considering a fixed number of predecessor layers at once. Finding the bound for the output of a neuron , for example, can be understood as solving a substantially smaller MIP problem by considering neurons from layer and when considering two preceding layers. These MIP problems are independent for each node in these layers and can therefore be solved in parallel. For each node, we first set the upper bound as a variable to be maximized in the objective, and trigger the MIP-solver to find such a value. Relations over integer binary variables can be derived by applying similar techniques. Notice that these MIPs only generate correct lower and upper bounds if they can be solved to optimality.
2. Branching priorities.
This encoding heuristics uses the given structure of feed-forward ANNs in that binary integer variables originating from lower layers are prioritized for branching. Intuitively, variables from the first hidden layer only depend on the input and it influences all other binary integer variables corresponding to neurons in deeper layers.
3. Constraint generation from samples and solver initialization.
For computing on complex systems via MIP, we use the following three-step process. First, find an input assignment such that the probability of classifying to is times larger, i.e., for all . Finding is equivalent to solving a substantially simpler MIP problem without introducing variables and . Second, use Eq. (4) to compute the minimum perturbation by considering the domain to be size 1, i.e., . As the domain is restricted to a single input, all variables in Eq. (4) are replaced by constants . This also yields substantially simpler MIP problems, and the computed bound is denoted by . Third, and finally, initialize the MIP-solver by using the computed values from steps 1 and 2, such that the search directly starts with a feasible solution with objective . Also, the constraint , as , can be further added to restrict the search space.
6 Implementation and Evaluation
We implemented an experimental platform in C++ for verifying and computing perturbation bounds for neural networks, which is based on IBM CPLEX Optimization Studio 12.7 (academic version) for MIP solving. We used three different benchmark sets as the basis for our evaluations: (1) MNIST333http://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html for number characterization, (2) agent games444http://cs.stanford.edu/people/karpathy/convnetjs/demo/rldemo.html, and (3) deeptraffic for simulating highway overtaking scenarios555http://selfdrivingcars.mit.edu/deeptrafficjs/. These benchmarks are denoted by , , and respectively, in the following. For each of the benchmarks we created neural networks with different numbers of hidden layers and numbers of neurons, which are shown in Tables 1 and 2. All the networks were trained using ConvNetJS .
|ID||instance & output m||# inputs; # neurons in hidden layers||status||Time(s)||Time(s) dataflow||Time(s) heuristic 1.+2.|
Agents in agent games have 9 sensors, each pointing into a different direction and returning the distances to an apple, poison or a wall, which amounts to the 27 inputs. Neural networks of various size were trained for an agent that gets rewarded for eating red things (apples) and gets negative reward when it eats green things (poison).
deeptraffic is used as a gamified simulation environment for highway traffic. The controller is trained based on a grid sensor map, and it outputs high-level driving decisions to be taken such as switch lane, accelerate or decelerate.
For MNIST digit recognition  has input nodes for the pixels of a gray-scale image, where we trained three networks with different numbers of neurons in the hidden layers.
In our experimental validation we focus on efficiency gains of our MIP encodings and parallelization for verifying neural networks, and the computation of perturbation bound by means of the optimization problem stated in Eq. (4).
Evaluation of MIP Encodings.
To understand how dataflow analysis and our heuristic encodings reduce the overall execution time, we have created synthetic benchmarks where for each example, we only ask for a given input instance (e.g., an image) that classifies to , whether the perturbation bound is below . By restricting ourselves to only verify a single input instance and by not minimizing , the problem under verification (local robustness related to an input) is substantially simpler and is similar to those stated in [13, 3]. Table 1 gives a summary over results being evaluated using Google Computing Engine (16 CPU and 60 GB RAM) by only allowing 12 threads to be used. Compared to a naïve approach that sets uniformly to a large constant, applying dataflow analysis can bring benefits for instances that take a longer time to solve. The first two heuristics we have implemented are useful for solving some very difficult problems. Admittedly, it can also result in longer solutions times for simpler instances, but as our ultimate goal is for scalability such an issue is currently minor. More difficult instances (see in Table 1) could only be solved using heuristic 1. for preprocessing.
Effects of Parallelization.
For we further measured the solution time for local robustness with for 10 test inputs using 8, 16, 24, 32 and 64 threads on machines that have at least as many CPUs as we allow CPLEX to have threads. The results are shown in Figure 7. It is clearly visible that using more threads can bring a significant speed-up till 32 cores, especially for instances that cannot be solved fast with few threads. Interestingly, one can also observe that for this particular problem (200 neurons in hidden layers), increasing the number of threads from 32 to 64 does not improve performance (many lines just flatten from 32 cores onwards). However, for some other problems (e.g., 400 neurons in hidden layers in hidden layers or computing resilience), the parallelization effect can last longer to some larger number of threads. We suspect that for problems that have reached a certain level of simplicity, adding additional parallelization may not further help.
|Net: # input; # neurons in hidden layers, output||# of parallelization||Time (s)|
Computing by solving problem (4).
Table 2 shows the result of computing precise . For simpler problems, we can observe from the first 4 rows of Table 2 that the computed increases, when the value of the parameter increases. This is a natural consequence - for inputs being classified with higher confidence, it should allow for more perturbation to bring to ambiguity. Notably, using a value of above its maximum makes the problem infeasible, because there does not exist an input for which the neural network has such high confidence. For complex problems, by setting is closer to its maximum (which can be computed by solving another substantially simpler MIP that maximizes for all inputs that classify to class ), one shrinks the complete input space to inputs with high confidence. Currently, scalability of our approach relies on sometimes setting a high value of , as can be observed in the lower part of Table 2.
7 Concluding Remarks
Our definition and computation of maximum perturbation bounds for ANNs using MIP-based optimization is novel. By developing specialized encoding heuristics and using parallelization we demonstrate the scalability and possible applicability of our verification approach for neural networks in real-world applications. Our verification techniques also allow to formally and quantitatively compare the resilience of different neural networks. Also, perturbation bounds provide a formal assume-guarantee interface for decoupling the design of sensor sets from the design of the neural network itself. In our case, the network assumes a maximum sensor input error for resilience, and the input sensor sets need to be designed to guarantee the given error bound. These kinds of contract-based interfaces may form the basis for constructing more modularized safety cases for autonomous systems.
Nevertheless, we consider the developments in this paper as only a first tiny step towards realizing the full potential of formal verification techniques for artificial neural networks and their deployment for realizing new safety-critical functionalities such as self-driving cars. For simplicity we have restricted ourselves to 1-norms for measuring perturbations but other vector norms may, of course, also be used depending on the specific needs of the application context. Also, the development of specialized MIP solving strategies for verifying ANNs, which go beyond the encoding heuristics provided in this paper, may result in considerable efficiency gains. Notice also that the offline verification approach as presented here is applied a posteriori to fixed and ”fully trained” networks, whereas real-world networks are usually trained and improved in the field and during operation. Furthermore, the exact relationship of our perturbation bounds with the common phenomena of over-fitting in a neural network classifier deserves a closer examination, since perturbation may also be viewed as generalization from samples. And, finally, investigation of further measures of the resilience of ANN is needed, as perturbation bounds do not generally cover the resilience of ANNs to input transformations such as scaling or rotation.
-  Y. S. Abu-Mostafa, M. Magdon-Ismail, and H.-T. Lin. Learning from data, volume 4. AMLBook New York, NY, USA:, 2012.
-  D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
-  O. Bastani, Y. Ioannou, L. Lampropoulos, D. Vytiniotis, A. Nori, and A. Criminisi. Measuring neural net robustness with constraints. CoRR, abs/1605.07262, 2016.
-  N. Bjørner, A.-D. Phan, and L. Fleckenstein. Z-An Optimizing SMT Solver. In TACAS, pages 194–199. Springer, 2015.
-  S. Bhattacharyya, D. Cofer, D. Musliner, J. Mueller, and E. Engstrom. Certification considerations for adaptive systems. In ICUAS, pages 270–279. IEEE, 2015.
-  M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to end learning for self-driving cars. CoRR, abs/1604.07316, 2016.
-  P. Cousot and R. Cousot. Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In POPL, pages 238–252. ACM, 1977.
-  G. Dantzig. Linear programming and extensions. Princeton university press, 2016.
-  I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
-  I. E. Grossmann. Review of nonlinear mixed-integer and disjunctive programming techniques. Optimization and engineering, 3(3):227–252, 2002.
-  X. Huang, M. Kwiatkowska, S. Wang, and M. Wu. Safety verification of deep neural networks. CoRR, abs/1610.06940, 2016.
Convnetjs: Deep learning in your browser (2014).URL http://cs.stanford.edu/people/karpathy/convnetjs, 2014.
-  G. Katz, C. W. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. CoRR, abs/1702.01135, 2017.
-  A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.
-  Y. LeCun, C. Cortes, and C. J. Burges. The mnist database of handwritten digits, 1998.
-  V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
-  A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CPVR, pages 427–436, 2015.
-  N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black-box attacks against deep learning systems using adversarial examples. arXiv preprint arXiv:1602.02697, 2016.
-  N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Oakland, pages 582–597. IEEE, 2016.
-  L. Pulina and A. Tacchella. An abstraction-refinement approach to verification of artificial neural networks. In CAV, pages 243–257. Springer, 2010.
-  L. Pulina and A. Tacchella. Challenging SMT solvers to verify neural networks. AI Communications, 25(2):117–135, 2012.
-  S. Rajan, S. Wang, R. Inkol, and A. Joyal. Efficient approximations for the arctangent function. IEEE Signal Processing Magazine, 23(3):108–111, 2006.
-  K. Scheibler, L. Winterer, R. Wimmer and B. Becker Towards Verification of Artificial Neural Networks. In MBMV, pages 30–40, 2015.
-  A. Ukil, V. H. Shah, and B. Deck. Fast computation of arctangent functions for embedded applications: A comparative analysis. In ISIE, pages 1206–1211. IEEE, 2011.
-  Y. Xu, T. K. Ralphs, L. Ladányi, and M. J. Saltzman. Computational experience with a software framework for parallel integer programming. INFORMS Journal on Computing, 21(3):383–397, 2009.
First we establish a lemma to assist the proof.