1 Introduction
We consider the problem of verifying the local robustness of piecewiselinear neural networks for a given bound. Precisely, given a point, , network, , and norm bound, , this entails determining whether Equation 1 holds.
(1) 
This problem carries practical significance, as such networks have been extensively shown to be vulnerable to adversarial examples (Szegedy et al., 2014; Papernot et al., 2016), wherein smallnorm perturbations are chosen to cause aribtrary misclassifications.
Numerous solutions have been proposed to address variants of this problem. These can be roughly categorized into three groups: learning rules that aim for robustness on known training data (Madry et al., 2018; Wong & Kolter, 2018), postprocessing methods that provide stochastic guarantees at inference time (Lecuyer et al., 2018; Cohen et al., 2019), and verification routines that attempt to answer Equation 1 (or a close analog of it) exactly (Katz et al., 2017, 2019; Jordan et al., 2019; Tjeng & Tedrake, 2017; Fischetti & Jo, 2018; Cheng et al., 2017) or via conservative overapproximation (Weng et al., 2018; Singh et al., 2019b; Balunovic et al., 2019; Ehlers, 2017; Wang et al., 2018; Dutta et al., 2018). While approximate verification methods have shown promise in scaling to larger networks, they may introduce an additional penalty to robust accuracy by flagging nonadversarial points, thus limiting their application in practice. Exact methods impose no such penalty, but as they rely on expensive constraintsolving techniques, they often do not scale to even moderatelysized networks.
In this paper, we present a verification technique for Equation 1
that neither relies on expensive constraint solving nor conservative overapproximation. We build on the observation that in Euclidean space, the angle between two random vectors of equal norm is with high probability
(Blum et al., 2020, Chapter 2). In other words, random vectors in high dimension are likely to be almost orthogonal, so the distance from a point to a hyperplane segment can often be approximated well by the distance from the point to its projection onto the corresponding hyperplane.
Our algorithm (Section 2) leverages this insight to exhaustively search the model’s decision boundaries around a point using only geometric projections. The performance benefits of this approach are substantial, especially in the case of robustness, since Euclidean projections can be efficiently computed using the dot product and accelerated on GPU hardware. Our algorithm is embarassingly parallel, and straightforward to implement with facilities for batching that are available in many popular libraries (Section 3). Additionally, we show how the algorithm can be easily modified to find certified lower bounds for , rather than verifying a given fixed value (Section 2.3).
Because our algorithm relies exclusively on local projections, it may encounter scenarios in which there is evidence to suggest nonrobust behavior, but the network’s exact boundaries cannot be conclusively determined without accounting for global constraints (Section 2, Figure 1). In such cases, the algorithm must either return , or fall back on constraint solving. However, we prove that if the algorithm terminates with a decision, then the model satisfies Equation 1, and likewise if it returns , then an adversarial example exists (Section 2.2). Note that unlike prior work on approximate verification, our approach can often separate cases from , providing a concrete adversarial example in the former. In this sense, the algorithm can be characterized as sound but incomplete.
Finally, our empirical evaluation (Section 4) demonstrates that for several models trained on MNIST and FashionMNIST, the algorithm returns only a small fraction of the time on randomlychosen test instances. Moreover, we show that our implementation outperforms existing exact techniques (Jordan et al., 2019; Katz et al., 2019) by up to two orders of magnitude (Section 4, Tables 1 & 2), while rarely being inconclusive on instances for which other techniques do not time out. Additionally, we show that while our implementation is not as fast as previous work on efficient lowerbound computation for large models (Weng et al., 2018), our certified lower bounds are consistently tighter, and in some cases minimal (Section 4, Table 3).
2 Algorithm
In this section we give a highlevel overview of our proposed algorithm (Section 2.1), and argue for its correctness (Section 2.2). We also propose a variant (Section 2.3) to compute certified lower bounds on the minimal adversarial distortion. We note that both the algorithm and the correctness argument are general and apply to arbitrary norms, and so for the remainder of this section we use the unsubscripted notation to refer to a general norm. However, in our implementation (Section 3), we focus on local robustness for the norm. We do this for the following two reasons: (1) the algorithm relies on projections that can be computed efficiently in Euclidean spaces, and (2) there are few existing tools that can efficiently guarantee local robustness with respect to , as they are usually limited to processing linear computations (see Section 5).
2.1 The Algorithm
The algorithm we propose (Algorithm 1) takes a model, , an input, , and a bound, , and either proves that Equation 1 is satisfied (i.e., local robustness), finds an adversarial input at distance less than from , or returns . Our algorithm assumes , where
is the function computed by a neural network composed of linear transformations with ReLU activations.
The algorithm relies on the analysis of all possible activation regions around . An activation region is a maximal set of inputs such that all elements have the same activation pattern.
Formally, let
denote the preactivation value of neuron
in network when evaluating . We say that neuron is activated if . An activation pattern, , is a Boolean function over neurons that characterizes whether each neuron is activated. Then the activation region, , associated with pattern is the set of inputs that realize :(2) 
Because we assume that is a piecewiselinear composition of linear transformations and ReLU activations, we can associate the activation status of any neuron with a closed halfspace of (Jordan et al., 2019). The activation constraint, , for neuron and pattern is the linear inequality , where and satisfy Equation 3.
(3) 
The intersection of these constraints yields the activation region , and the facets of correspond to the nonredundant constraints. The convexity of activation regions follows from this observation, as does the fact that the decision boundaries are also linear constraints of the form for classes and . More details on computing these constraints are given in Section 3.1.
Algorithm 1 performs a search of all activation regions that might be at distance less than from . Starting from the region associated with the activation pattern of the input, , we collect all decision boundaries and activation constraints, and enqueue those at distance less than from . Here we define the distance, , from to a constraint, , as . We then analyze each constraint in the queue until the queue is empty.
If an analyzed constraint, , is a decision boundary, we take the projection, , from onto this constraint. Intuitively, the projection, , is an input satisfying that is at minimal distance from , i.e., . If does not have the same class as according to (or lies directly on the decision boundary between classes), then is an adversarial example. If has the same class as , it means that the projection to the decision boundary is outside of the activation region we are currently analyzing; however, this is not sufficient to conclude that no point on the decision boundary is inside both the current activation region and the ball (see Figure 1). Therefore, we return .
On the other hand, if an analyzed constraint, , is an activation constraint, it means that the neighboring activation region with neuron flipped may intersect with the ball centered at . This means that, if we are currently searching in activation region, , given by activation pattern , then we must search the region, , given by activation pattern , where except when ; in other words, we search the region corresponding to the activation pattern that would be obtained by flipping the neuron corresponding to the constraint we are analyzing. As previously, we search by gathering all activation constraints and decision boundaries corresponding to , enqueuing the ones at distance less than from .
Exhausting the queue means that we did not find any adversarial examples in any activation region that might intersect with the ball centered at . We therefore conclude that is robust in this case.
Note that since there are finitely many activation regions and we only explore each region at most once, this algorithm terminates.
2.2 Correctness
In this section, we argue for the correctness of the proposed algorithm. We show that when Algorithm 1 returns , there exists an adversarial example, and when it returns , the model is locally robust at , with radius . However, the algorithm may also return , in which case we do not claim anything about the robustness of the model.
Theorem 1.
Proof.
In the first case, where Algorithm 1 returns , the proof of Theorem 1 is trivial: we exhibit a point, , such that , and for which .
The interesting case is when Algorithm 1 returns . We prove by contradiction that in this case, is in fact locally robust with radius .
Let us assume for the sake of contradiction that Algorithm 1 returns , but there exists a point, , such that and . Let and be the activation regions associated with and respectively.
We define a path of activation regions as a sequence, , of activation regions such that the underlying activation patterns and differ in exactly one neuron for all , and there exists at least one input, , that has activation pattern for all . For instance, in a network with three neurons, if , , and , and there exist inputs, , , and with activation patterns , , and , then , , is a path.
Our proof relies on three facts, that we prove hereafter:

There exists a path, , from to where each region in the path contains at least one input at distance less than from , and either or all in the path are different.

Our algorithm visits all regions in the path, .

If a visited activation region contains an adversarial input, our algorithm either detects it, returning , or returns .
(1) Existence of a Path
Consider the segment going from to in a straight line. As , all points on this segment are also at distance less than . As is a neural network with ReLU activations, is a continuous function, as are the activation functions, for each of its internal neurons, . Furthermore, input points at the boundary between two activation regions (i.e., , such that ) belong to both activation regions. Therefore, listing all activation regions on the segment between and yields a path, , from to , with each region on the path containing an input point, , on the segment, such that .
That each , in is unique follows from the convexity of activation regions: If there exist in such that there exists an in between, then cannot be convex, as there exists a line segment with its end points in and that is not entirely contained within the region . This ensures that paths are of finite length.
(2) Exploration by the Algorithm
Given the existence of the path, , from to , we now prove that Algorithm 1 would visit all activation regions in if it returns . We proceed by induction on the length of paths induced (similarly to as above) by a line segment included in the ball centered on .
In the base case, if the path is of length one, then it contains only , and the claim holds trivially since Algorithm 1 starts by visiting .
In the inductive case, let us assume that for any path with length at most induced by a segment, , beginning at with , Algorithm 1 visits all regions in the path. Now consider a path, , of length , induced by a segment, , beginning at with . Since is on the path, there exists a point, , on such that the subsegment from to induces a path of length ; thus we can apply our induction hypothesis to conclude that Algorithm 1 visits . Now, since and are neighbors in , they must share some boundary, , that is intersected by . Thus, since , ; thus when Algorithm 1 visits , it will add to the queue via . Therefore, since Algorithm 1 returns only when all regions in the queue have been visited, Algorithm 1 will visit , concluding the proof of (2).
(3) Detection of Adversarial Examples
We conclude by proving that if there exists an adversarial example in a region visited by Algorithm 1, then we either return or .
If in is an adversarial example, then . By continuity of , this means that there exists an input, on the segment from to that is exactly on a decision boundary, . As , must have been analyzed when exploring .
However, when analyzing a decision boundary, Algorithm 1 always returns either or .
Thus, the decision boundary in the activation region containing would have been analyzed by Algorithm 1; this yields a contradiction, as the algorithm must have returned or rather than . ∎
2.3 Certified Lower Bounds
We now consider the related problem of finding a lower bound on the local robustness of the network. A variant of Algorithm 1, given by Algorithm 2, can provide certified lower bounds by using a priority queue.
In this variant, constraints are enqueued with a priority corresponding to their distance from , such that the closest constraint to is always at the front of the queue. We keep track of the current certified lower bound in a variable, . At each iteration, we set to the maximum of the old and the distance from to the constraint at the front of the queue.
The algorithm terminates either when all constraints at distance less than the initially specified were handled, or when a decision boundary for which the initial algorithm returns or is found. In the first case, we return ; in the second, we return the value stored in at this iteration.
The proof of this variant is similar to the proof of Algorithm 1. It relies on the following loop invariant.
Loop Invariant 1.
(1) At each iteration, all activation regions at distance less than from were previously visited, (2) is always smaller than , and (3) there is no adversarial point at distance less than .
Proof.
Loop Invariant 1 trivially holds on entry of the loop since is initialized to 0. Proving that the invariant is maintained is more interesting. Suppose that increases to . It must then be shown that there is no unvisited activation region at distance less than . We proceed again by contradiction: assume there was such a region, , containing a point, , such that . Again, let us consider the segment from to , and the path, , it induces. Let us consider , the first region of at distance greater than that was not previously visited. If no such region exists, then is at distance less than from , and so by our induction hypothesis, it was already visited. otherwise, was visited, and the activation constraint, , between and is such that . Therefore, (which leads to ) was already added to the queue with priority less than , and by virtue of the priority queue, it was explored before the current iteration, which yields a contradiction.
If does not increase, the invariant still trivially holds. This case can happen because our computation of the distance to constraints is an underapproximation of the true distance to the feasible portion of the constraint, as is illustrated in Figure 2. ∎
3 Efficient Implementation
In this section, we describe the key insights that allow us to implement Algorithm 1 efficiently for the norm, and describe various optimizations we employ to improve its practicality. In Sections 3.1 and 3.2 we elaborate on the insights and implementation details relevant to computing constraints and approximating their distance from . Section 3.3
describes heuristics for reducing the number of
results Algorithm 1 returns, and Section 3.4 presents optimizations we implement that speed up our analysis.3.1 Computing Activation Region Boundaries
Recall our setting in which we assume our model, , is given by , where is the function computed by a neural network consisting of layers of linear transformations with ReLU activations between layers. For a particular activation pattern, , we observe that we can replace each ReLU, in layer, , with an elementwise multiplication by a vector, , where the position in corresponding to neuron, , takes value 1 if is , and 0 otherwise. Thus, when ,
which we observe is a linear function with the same slope for all in . By the same argument, is a linear function, , for all in .
Now consider the activation constraint, , which is satisfied when . Given our above observation, we see that this is a linear constraint with coefficients, , and intercepts, . Note that the computation of these weights and intercepts does not depend on a particular point in — only on the activation pattern, . Thus, we can compute the boundaries of an activation region, , knowing only the corresponding activation pattern, .
In practice, the coefficients, , correspond to the gradient of with respect to its inputs, evaluated at a point in
. However, frameworks that perform automatic differentiation typically require a concrete point for evaluation of the gradient. Thus, we compute the gradient via backpropagation with the activation vectors,
. The intercepts, , are computed via a forward computation using the activation vectors, with set to 0. These operations can easily be implemented to run efficiently on a GPU.Decision Boundaries
For a sigmoid network, there is at most a single decision boundary within any given activation region, which can be computed via gradient backpropagation as discussed above. In the case of a softmax model with multiple classes, we have an output for each class. A decision boundary for a point, , given class, , states that , for . Thus, for models with classes, the decision boundary consists of linear constraints derived from the difference of the model’s output at the selected class and the model’s output at each of the other classes.
3.2 Distances to Constraints
We observe that when working with the norm, we can efficiently compute projections using the dot product. Note that this computation can be trivially performed in parallel on many constraints at once by a GPU.
The distance from to its projection onto a hyperplane constraint, , gives a lower bound on the distance from to the part of that is feasible when the other constraints are taken into consideration. This allows us to soundly determine if a boundary constraint is further than from , implying local robustness, and is substantially faster than using constraint solving to discover the true distance. In our evaluation, we find that in high dimensions, the projected distance is an accurate enough lower bound to make Algorithm 1 practical.
3.3 Improving the Completeness of the Algorithm
The vanilla version of our algorithm presented in Section 2 stops and possibly returns at the first decision boundary encountered. We now present two heuristics to reduce the number of results returned, at the price of increased complexity. These heuristics are disabled by default in our implementation. We leave it to the user to enable them when deemed necessary.
Exploring the Full Queue
If the analysis of a decision boundary is inconclusive, Algorithm 1, as presented so far, stops and returns . We implement a variant of Algorithm 1 where we instead record the presence of an ambiguous decision boundary but continue exploring other regions in the queue. If we entirely empty the queue after unsuccessfully analyzing a decision boundary, we nevertheless return . However, oftentimes a true adversarial example is found during the continued search, allowing us to return conclusively.
The correctness of our algorithm still holds in this variant: an inconclusive analysis of a decision boundary leads to either returning , or exhibiting a true adversarial example found at a later stage.
Falling Back on LP Solving
Existing complete tools (Katz et al., 2017, 2019; Jordan et al., 2019; Tjeng & Tedrake, 2017; Fischetti & Jo, 2018; Cheng et al., 2017) often rely heavily on solvers to analyze models. When we cannot rule out adversarial examples in a specific activation region, we can query an LP solver, since both the activation constraints delimiting a region and the decision boundaries inside this region are linear constraints. This allows us to efficiently discharge many verification conditions through fast geometric projections while providing a complete, but slower, variant of our algorithm.
3.4 Speeding up the Analysis
In Section 2, Algorithm 1 is presented at a high level without regard to performance. In this section, we give an overview of the key optimizations we include in our implementation of Algorithm 1 that are independent of the highlevel behavior of the algorithm.
Analyzing Decision Boundaries First
The correctness of Algorithm 1 is independent of how we iterate over the queue of constraints to analyze. Since our algorithm terminates as soon as we analyze a decision boundary, one useful optimization is thus to ensure that such constraints are always handled first. In our implementation of Algorithm 1, we use a queue datastructure consisting of two standard FIFO queues: one for decision boundaries and one for activation constraints. When dequeuing an element, we always try to dequeue a decision boundary first. This ensures that the most relevant constraints are handled first without harming the asymptotic complexity of our algorithm.
This optimization cannot be used when searching for a certified lower bound without breaking the correctness of our algorithm. It is thus only allowed when checking local robustness.
Batching
Most deep learning software is designed to easily enable efficient batch computations on a GPU. In particular, the gradient backpropagations for computing constraints (Section
3.1) and the dot product projections for calculating distances (Section 3.2) lend themselves well to batching on the GPU. We leverage this in our implementation of Algorithm 1 by dequeuing multiple elements from the queue at once and calculating the constraints and constraint distances for the corresponding regions in parallel. We find that this optimization enables a speedup of up to 2x on large examples for a wellchosen batch size (10100).4 Evaluation
In this section, we evaluate the performance of our implementation of Algorithms 1 and 2 on several models adversarially trained (Madry et al., 2018) on MNIST and FashionMNIST. We also include a subset of MNIST (binary MNIST) in which the classification task is to distinguish between the digits “1” and “7,” as models trained on this dataset were used in the evaluation of GeoCert (Jordan et al., 2019). We selected 100 arbitrary instances from the test set of each dataset to perform our evaluation on.
Each model in our evaluation is named with the following convention: the name begins with an identifier for the dataset it was trained on, either “mnist_bin” (binary MNIST), “mnist”, or “fmnist” (FashionMNIST), followed by the sizes of the hidden layers. For example, the model, “fmnist40.40.40” was trained on FashionMNIST and has three hidden layers with 40 neurons each.
All experiments were run on a 4.2GHz Intel Core i77700K with 32 GB of RAM, and a Titan X GPU with 12 GB of RAM. Our implementation is written in Python using the Keras framework
(Chollet et al., 2015)with the TensorFlow backend
(Abadi et al., 2015).4.1 Local Robustness Certification
Algorithm 1, ,  GeoCert, ,  Marabou, ,  
Model  Time (s)  Time (s)  Time (s)  
mnist_bin10.50.10  0.020  100  0  0  0  0.675  99  0  1  0.136  100  0  0 
mnist_bin20.20  0.015  99  0  1  0  0.463  99  0  1  0.162  99  1  0 
mnist20.20.20  0.029  87  6  6  1  4.839  72  2  26  1.875  95  5  0 
mnist40.40.40  3.111  50  7  16  27  >60.0  13  12  75  4.344  96  4  0 
fmnist20.20.20  0.035  89  4  7  0  5.916  84  8  8  3.669  93  7  0 
fmnist40.40.40  1.528  65  3  9  23  >60.0  0  0  100  5.147  97  3  0 
Vanilla, ,  Full queue, ,  
Model  Time (s)  Time (s)  
mnist20.20.20  0.029  87  6  6  1  0.029  87  10  2  1 
mnist40.40.40  3.111  50  7  16  27  6.166  50  14  6  30 
fmnist20.20.20  0.035  89  4  7  0  0.036  89  9  2  0 
fmnist40.40.40  1.528  65  3  9  23  2.910  65  7  1  27 
Model  Algorithm 2, Mean Bound  FastLin, Mean Bound  Median Ratio 

fmnist  0.124  0.078  0.329 
fmnist  0.134  0.092  0.693 
fmnist  0.083  0.021  0.035 
We first compare the efficiency of our implementation to that of other tools certifying local robustness. GeoCert (Jordan et al., 2019) is the tool most comparable to ours as it is able to exactly check for robustness with an norm. We also compare our implementation to the Marabou verification tool (Katz et al., 2019), an improvement over Reluplex (Katz et al., 2017), that does not handle the norm, but instead supports the norm. While these norms are not entirely commensurable, we perform the comparison by checking the largest ball contained in the ball of size . For MNIST models, this ball has size as the input space is of dimension 784. Note that as a consequence, we expect Marabou to label a higher fraction of the points as , as the search space we give it is significantly smaller.
We evaluate each of these tools with . Specifically, we compare the number of examples on which each tool terminates, the result of the analyses that terminate, and the median run time over the analyses of each of the selected 100 instances. We set a timeout of 20 seconds for the (smaller) binary MNIST models and of 60 seconds for the (larger) MNIST and FashionMNIST models. The results for these experiments are presented in Table 1.
We observe that our implementation always outperforms GeoCert by one to two orders of magnitude, without sacrificing precision — we rarely return when GeoCert terminates. While the comparison with Marabou’s verification is not directly comparable, it is worth noting that we are 1.4x100x faster than Marabou despite the specified ball requiring analysis of a much smaller input space.
Table 2 displays the results of experiments evaluating the impact of exploring the whole queue of constraints (Section 3.3) instead of stopping at the first decision boundary as described in the vanilla version of Algorithm 1. We compare the performance of both variants on the larger networks for which the analysis of the vanilla implementation was occasionally inconclusive. Experimental results show that this heuristic decreases the number of results while only having a minor impact on the speed of the execution. Moreover, when enabling this heuristic, we reduce the number of results to the point that we are able to recover the results obtained by GeoCert in almost every instance for which GeoCert terminates, while nevertheless performing our analysis one to two orders of magnitude faster.
4.2 Certified Lower Bounds
We now evaluate the variant of our algorithm computing certified lower bounds on the minimum adversarial distortion (Algorithm 2 in Section 2.3). To this end, we compare the performance of our approach to the FastLin (Weng et al., 2018) tool, which is designed to provide quick lower bounds on the local robustness of ReLU networks.
FastLin is intended to scale to very large networks for which exact checks like GeoCert and Marabou are computationally infeasible. On these large networks (e.g., with thousands of neurons), FastLin is substantially faster than our implementation; however, the bounds it provides may be quite loose.
We compare the certified lower bound reported by our implementation of Algorithm 2 after 60 seconds of computation to the lower bound reported by FastLin on 100 instances arbitrarily picked from FashionMNIST; the results are presented in Table 3. The mean lower bound is reported for both methods, and we observe that on the models tested, Algorithm 2 is able to find a better lower bound on average, though it requires considerably more computation time.
Since the optimal bound may vary between instances, we also report the median ratio of the lower bounds obtained by the two methods on each individual instance. Here we see that FastLin may indeed be quite loose, as on a typical instance it achieves as low as 4% and only as high as 69% of the bound obtained by Algorithm 2.
Finally, we note that, when Algorithm 2 terminates by finding a decision boundary, if the projection onto that boundary is a true adversarial example, the lower bound is tight. In our experiments, there were few such examples — three on fmnist and one on fmnist — however, in these cases, the lower bound obtained by FastLin was very loose, achieving 415% of the optimal bound on fmnist, and only 0.8% of the optimal bound on fmnist. This suggests that while FastLin has been demonstrated to scale to large networks, one must be careful with its application, as there may be cases in which the bound it provides is too conservative.
5 Related Work
As mentioned, our work can be grouped with approaches for verifying neural network that aim to check local robustness exactly (Katz et al., 2019, 2017; Jordan et al., 2019); the main difference is that we aim to avoid expensive constraint solving at the price of incompleteness.
GeoCert (Jordan et al., 2019) is the closest work to ours; it aims to compute exact pointwise robustness of deep neural networks for convex norms. Unlike our approach, GeoCert computes the largest ball, centered at an input point, within which the network is robust. Our experimental comparison with GeoCert shows that our approach scales much better. This is not surprising as GeoCert relies on projections to facets (polytopes), which are solved by a quadratic program with linear constraints. In contrast, our approach uses projection to affine subspaces, which has a simpler, closedform solution.
Reluplex (Katz et al., 2017) and its successor, Marabou (Katz et al., 2019), use SMT solving techniques to check local robustness and more general safety properties that can be expressed as linear constraints. Our experiments show that our approach performs better even in a setting that is advantageous to Reluplex/Marabou.
ERAN (Singh et al., 2019a), and its predecessor, AI2 (Gehr et al., 2018), rely on abstract interpretation (Cousot & Cousot, 1977) to automatically verify local robustness of neural networks against both and geometric input perturbations. As such, they use sound, conservative overapproximations to perform their analysis, which leads to false positives, i.e. robust inputs incorrectly classified as not robust. In comparison, all of the inputs flagged as by our approach have an adversarial example at distance less than .
FastLin (Weng et al., 2018) explores the special structure of ReLU networks to efficiently compute lower bounds on adversarial minimal distortions. This was expanded to general activation functions in a tool called CROWN (Zhang et al., 2018). Although FastLin has been shown to be very scalable, our experiments indicate that the computed bounds may be quite imprecise.
Recently, another quite different approach has been proposed for robustness certification. Randomized Smoothing (Cohen et al., 2019; Lecuyer et al., 2018) is a postprocessing technique that provides a stochastic robustness guarantee at inference time. This approach differs from our approach in that it (1) modifies the predictions of the original model (increasing the complexity of making predictions), and (2) provides a probabilistic robustness guarantee that is quantified via a confidence bound. As such it provides an alternative set of costs and benefits as compared to our approach. Its complexity also differs from that of our approach, as it is dependent less on the architecture of the model, but rather on the number of samples required to perform its postprocessing of the model’s output. We find that in our experimental setup, achieving the same probabilistic guarantee as the experiments described in (Cohen et al., 2019) (requiring samples), Randomized Smoothing takes approximately 4.5 seconds per instance to make a certified prediction on each of the models tested in Section 4.1. Thus, for these models, our implementation is on average faster or comparable in performance.
6 Conclusion and Future Work
In this paper, we have presented a new approach to verify the local robustness of networks with piecewise linear activation functions. Compared to previous work, our approach does not rely on constraint solving nor overapproximations but rather on geometric projections. While most existing tools focus on linear constraints, and therefore on and norms, we provide an efficient, highly parallel implementation to certify robustness. Our implementation outperforms existing exact tools by up to two orders of magnitude on MNIST and FashionMNIST, while empirically maintaining the same precision under a time constraint.
Several possible improvements to our implementation are left as future work. The algorithm presented in Section 2 is normindependent, but its performance for the norm relies on efficiently computing projections using a GPU. Extending our approach to other norms would require similar techniques, which already exist for the norm (Condat, 2015; Duchi et al., 2008)
for instance. As our implementation runs mostly on a GPU, we should also be able to more aggressively fall back on parallel executions of LP solvers, which run on CPUs, when our analysis is inconclusive. Finally, we plan to add support for convolutional neural networks.
Acknowledgments
The work described in this paper has been supported by the Software Engineering Institute under its FFRDC Contract No. FA870215D0002 with the U.S. Department of Defense, Bosch Corporation, an NVIDIA GPU grant, NSF Award CNS1801391, a Google Faculty Fellowship, and the Alfred P. Sloan Foundation.
References
 Abadi et al. (2015) Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. TensorFlow: Largescale machine learning on heterogeneous systems. https://www.tensorflow.org/, 2015.
 Balunovic et al. (2019) Balunovic, M., Baader, M., Singh, G., Gehr, T., and Vechev, M. Certifying geometric robustness of neural networks. In Advances in Neural Information Processing Systems (NIPS). 2019.

Blum et al. (2020)
Blum, A., Hopcroft, J., and Kannan, R.
Foundations of Data Science
. Cambridge University Press, 2020.  Cheng et al. (2017) Cheng, C.H., Nührenberg, G., and Ruess, H. Maximum resilience of artificial neural networks. In Automated Technology for Verification and Analysis (ATVA), 2017.
 Chollet et al. (2015) Chollet, F. et al. Keras. https://keras.io, 2015.
 Cohen et al. (2019) Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), 2019.
 Condat (2015) Condat, L. Fast projection onto the simplex and the ball. Mathematical Programming, 158, 09 2015.
 Cousot & Cousot (1977) Cousot, P. and Cousot, R. Abstract interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Symposium on Principles of Programming Languages (POPL), 1977.
 Duchi et al. (2008) Duchi, J., ShalevShwartz, S., Singer, Y., and Chandra, T. Efficient projections onto the ball for learning in high dimensions. International Conference on Machine Learning (ICML), 2008.
 Dutta et al. (2018) Dutta, S., Jha, S., Sankaranarayanan, S., and Tiwari, A. Output range analysis for deep feedforward neural networks. In NASA Formal Methods Symposium (NFM), 2018.
 Ehlers (2017) Ehlers, R. Formal verification of piecewise linear feedforward neural networks. In Conference on Logic in Computer Science (LICS), 2017.
 Fischetti & Jo (2018) Fischetti, M. and Jo, J. Deep neural networks and mixed integer linear optimization. Constraints, 23(3):296–309, Jul 2018.
 Gehr et al. (2018) Gehr, T., Mirman, M., DrachslerCohen, D., Tsankov, P., Chaudhuri, S., and Vechev, M. AI2: Safety and robustness certification of neural networks with abstract interpretation. In Symposium on Security and Privacy (S&P), 2018.
 Jordan et al. (2019) Jordan, M., Lewis, J., and Dimakis, A. G. Provable certificates for adversarial examples: Fitting a ball in the union of polytopes. In Advances in Neural Information Processing Systems (NIPS), 2019.
 Katz et al. (2017) Katz, G., Barrett, C. W., Dill, D. L., Julian, K., and Kochenderfer, M. J. Reluplex: An efficient SMT solver for verifying deep neural networks. In Computer Aided Verification (CAV), 2017.
 Katz et al. (2019) Katz, G., Huang, D. A., Ibeling, D., Julian, K., Lazarus, C., Lim, R., Shah, P., Thakoor, S., Wu, H., Zeljić, A., Dill, D. L., Kochenderfer, M. J., and Barrett, C. The marabou framework for verification and analysis of deep neural networks. In Computer Aided Verification (CAV), 2019.
 Lecuyer et al. (2018) Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., and Jana, S. Certified robustness to adversarial examples with differential privacy. In Symposium on Security and Privacy (S&P), 2018.
 Madry et al. (2018) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
 Papernot et al. (2016) Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A. The limitations of deep learning in adversarial settings. In European Symposium on Security and Privacy (EuroS&P), 2016.
 Singh et al. (2019a) Singh, G., Ganvir, R., Püschel, M., and Vechev, M. Beyond the single neuron convex barrier for neural network certification. In Advances in Neural Information Processing Systems (NIPS). 2019a.
 Singh et al. (2019b) Singh, G., Gehr, T., Püschel, M., and Vechev, M. Robustness certification with refinement. In International Conference on Learning Representations (ICLR), 2019b.
 Szegedy et al. (2014) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
 Tjeng & Tedrake (2017) Tjeng, V. and Tedrake, R. Verifying neural networks with mixed integer programming. CoRR, abs/1711.07356, 2017.
 Wang et al. (2018) Wang, S., Pei, K., Whitehouse, J., Yang, J., and Jana, S. Formal security analysis of neural networks using symbolic intervals. In USENIX Conference on Security Symposium, 2018.
 Weng et al. (2018) Weng, L., Zhang, H., Chen, H., Song, Z., Hsieh, C.J., Daniel, L., Boning, D., and Dhillon, I. Towards fast computation of certified robustness for ReLU networks. In International Conference on Machine Learning (ICML), 2018.
 Wong & Kolter (2018) Wong, E. and Kolter, J. Z. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning (ICML), 2018.
 Zhang et al. (2018) Zhang, H., Weng, T.W., Chen, P.Y., Hsieh, C.J., and Daniel, L. Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems (NIPS). 2018.
Comments
There are no comments yet.