1 Introduction
Neural networks are now recognized as powerful function approximators with impressive performance across a wide range of applications. Current techniques, however, provide no correctness guarantees — there is currently no way to verify that a neural network provides correct outputs (within a specified tolerance) for all inputs of interest. The closest the field has come is robustness verification, which aims to verify if the network prediction is stable for all inputs in some neighborhood around a selected input point Gehr et al. (2018); Xiang et al. (2018); Weng et al. (2018); Bastani et al. (2016); Tjeng et al. (2019); Dvijotham et al. (2018); Wong and Kolter (2018); Katz et al. (2017). But robustness verification does not guarantee that the output, even if stable, is actually correct — there is no specification that defines the correct output for any input except for the manuallylabeled center point of each region.
We present the first correctness verification of neural networks — the first verification that a neural network produces a correct output within a specified tolerance for every input of interest. Neural networks are often used to predict some property of the world given an observation such as an image or audio recording. We therefore define correctness relative to a specification which identifies 1) a state space consisting of all relevant states of the world and 2) an observation process that produces neural network inputs from the states of the world. Then the inputs of interest are all inputs that can be observed from the state space via the observation process. We define the set of inputs of interest as the feasible input space. Because the quantity of interest that the network predicts is some property of the state of the world, the state defines the ground truth output (and therefore defines the correct output for each input to the neural network).
We present Tiler, the first algorithm for correctness verification of neural networks. Evaluating the correctness of the network on a single state is straightforward — use the observation process to obtain the possible inputs for that state, use the neural network to obtain the possible outputs, then compare the outputs to the ground truth from the state. To do correctness verification, we generalize this idea to work with tiled state and input spaces. We cover the state and input spaces with a finite number of tiles: each state tile comprises a set of states; each input tile is the image of the corresponding state tile under the observation process. The state tiles provide ground truth bounds for the corresponding input tiles. We use recently developed techniques from the robustness verification literature to obtain network output bounds for each input tile Xiang et al. (2018); Gehr et al. (2018); Weng et al. (2018); Bastani et al. (2016); Lomuscio and Maganti (2017); Tjeng et al. (2019). A comparison of the ground truth and output bounds delivers an error upper bound for that region of the state space. The error bounds for all the tiles jointly provide the correctness verification result.
We demonstrate how to do correctness verification using Tiler via a case study. Consider a world with a (idealized) fixed road and a camera that can vary its horizontal offset and viewing angle with respect to the centerline of the road. The state of the world is therefore characterized by the offset and the viewing angle . A neural network predicts the offset and the viewing angle of the camera, taking the image taken by the camera as input. The state space contains the ranges of and of interest. The observation process is the camera imaging process, which maps camera positions to images taken. This state space and the camera imaging process provide the specification: the feasible input space is the set of camera images that can be observed from all camera positions of interest; for each image, the camera positions of all the states that can produce this image give the possible ground truths. We tile the state space using a grid on . Each state tile gives a bound on the ground truth of and . We then apply the observation process to project each state tile into the image space. By computing a bounding box for each resulting input tile and applying techniques from robustness verification Tjeng et al. (2019), we obtain neural network output bounds for each input tile. Comparing the ground truth bounds and the network output bounds gives upper bounds on network prediction error for each tile. We are able to verify that our trained neural network provides good accuracy across the vast majority of the state space of interest. We are also able to effectively bound the maximum error the network will ever produce on any feasible input.
This paper makes the following contributions:
Specification: We show how to use state spaces and observation processes to specify the correctness of neural networks that predict properties of states given observations of the states as input.
Verification: We present the first algorithm for verifying that a neural network produces the correct output (up to a specified tolerance) for every input of interest. The algorithm can also compute tighter correctness bounds for focused regions of the state and input spaces.
Case Study: We apply this algorithm to the problem of predicting camera offsets and viewing angles to obtain the first correctness verification of a neural network. The verification proves that the network always produces correct outputs (within the specified tolerance) for all inputs of interest and characterizes how the correctness bounds vary across the state and input spaces.
2 Related Work
Motivated by the vulnerability of neural networks to adversarial attacks (Papernot et al., 2016; Szegedy et al., 2014), researchers have developed a range of techniques for verifying robustness — they aim to verify if the neural network prediction is stable in some neighborhood around a selected input point. Liu et al. (2019) provides an overview of the field. A range of approaches have been explored, including layerbylayer reachability analysis (Xiang et al., 2017, 2018) with abstract interpretation (Gehr et al., 2018) or bounding the local Lipschitz constant (Weng et al., 2018), formulating the network as constraints and solving the resulting optimization problem (Bastani et al., 2016; Lomuscio and Maganti, 2017; Cheng et al., 2017; Tjeng et al., 2019), solving the dual problem (Dvijotham et al., 2018; Wong and Kolter, 2018; Raghunathan et al., 2018), and formulating and solving using SMT/SAT solvers (Katz et al., 2017; Ehlers, 2017; Huang et al., 2017). In the context of control systems, Dutta et al. (2018) introduces an approach to verify state reachability and region stability of closedloop systems with neural network controllers. Xiang et al. (2018) verifies safety by computes the reachable set of states and checks if they overlap with the unsafe states. Unlike the research presented in this paper, none of this prior research formalizes or attempts to verify that the neural network computes correct outputs within a specified tolerance for all inputs of interest.
Prior work on neural network testing focuses on constructing better test cases to expose problematic network behaviors. Researchers have developed approaches to build test cases that improve coverage on possible states of the neural network, for example neuron coverage
(Pei et al., 2017; Tian et al., 2018) and generalizations to multigranular coverage (Ma et al., 2018) and MC/DC (Kelly J. et al., 2001) inspired coverage (Sun et al., 2018). Odena and Goodfellow (2018) presents coverageguided fuzzing methods for testing neural networks using the above coverage criteria. Tian et al. (2018) generates realistic test cases by applying natural transformations (e.g. brightness change, rotation, add rain) to seed images. Unlike this prior research, which tests the neural network on only a set of input points, the research presented in this paper verifies correctness for all inputs of interest.3 Correctness Verification of Neural Networks
Consider the general problem of taking an input observation and trying to predict some quantity of interest . It can be a regression problem (continuous ) or a classification problem (discrete ). Some neural network model is trained for this task. We denote its function by , where is the space of all possible inputs to the neural network and is the space of all possible outputs. Behind the input observation there is some state of the world . Denote as the space of all states of the world that we want to verify. For each state of the world, a set of possible inputs can be observed. We denote this observation process using a mapping , where is the set of inputs that can be observed from . Here is the power set, and is the feasible input space, the part of input space that may be observed from the state space . Concretely, .
The quantity of interest is some aspect of the state of the world. We denote the ground truth of using a function . This specifies the ground truth for each input, which we denote as a mapping . is the set of possible ground truth values of for a given :
(1) 
The feasible input space and the ground truth mapping together form a specification. In general, we cannot compute and represent and directly — indeed, the purpose of the neural network is to compute an approximation to this ground truth which is otherwise not available given only the input . and are instead determined implicitly by , , and .
The error of the neural network is then characterized by the difference between and . Concretely, the maximum possible error at a given input is:
(2) 
where is some measurement on the size of the error between two values of the quantity of interest. For regression, we consider the absolute value of the difference .^{1}^{1}1For clarity, we formulate the problem with a onedimensional quantity of interest. Extending to multidimensional output (multiple quantities of interest) is straightforward: we treat the prediction of each output dimension as using a separate neural network, all of which are the same except the final output layer. For classification, we consider a binary error measurement (indicator function), i.e. the error is 0 if the prediction is correct, 1 if the prediction is incorrect.
The goal of correctness verification is to compute upper bounds on network prediction errors with respect to the specification. We formulate the problem of correctness verification formally here:
Problem formulation of correctness verification:
Given a trained neural network and a specification determined implicitly by , , and , compute upper bounds on error for any feasible input .
4 Tiler
We next present Tiler, an algorithm for correctness verification of neural networks. We present here the algorithm for regression settings, with sufficient conditions for the resulting error bounds to be sound. The algorithm for classification settings is similar (see Appendix B).
The first step is to use the state space to tile the feasible input space:
Step 1: Divide the state space into local regions such that .
The image of each under is a tile on the input space: . The resulting tiles satisfy the following condition:
Condition 4.1. .
The second step is to compute a ground truth bound for each input tile , specifically by computing the ground truth bound for the corresponding under the ground truth function :
Step 2: For each , compute the ground truth bound as an interval , such that .
We use this ground truth bound for tile . The bounds computed this way satisfy the following condition, which (intuitively) states that the possible ground truth values for an input point must be covered jointly by the ground truth bounds of all the input tiles that contain this point:
Condition 4.2(a). For any such that and .
Previous research has produced a variety of methods that bound the neural network output over given input region. Examples include layerbylayer reachability analysis (Xiang et al., 2018; Gehr et al., 2018; Weng et al., 2018) and formulating constrained optimization problems (Bastani et al., 2016; Lomuscio and Maganti, 2017; Tjeng et al., 2019). Each method typically works for certain classes of networks (e.g. piecewise linear networks) and certain classes of input regions (e.g. polytopes). For each input tile , we therefore introduce a bounding box that 1) includes and 2) is supported by the solving method:
Step 3: Using and , compute a bounding box for each tile .
The bounding boxes ’s must satisfy the following condition:
Condition 4.3. .
We then solve the neural network output range for each bounding box:
Step 4: Given and bounding boxes , use an appropriate solver to solve for the network output ranges .
The neural network has a single output entry for each quantity of interest. Denote the value of the output entry as , . The network output bounds returned by the solver must satisfy the following condition:
Condition 4.4(a) .
With the ground truth bounds and network output bounds, we can now compute the error bounds:
Step 5: For each tile, use the ground truth bound and network output bound to compute the error bound :
(3) 
gives the upper bound on prediction error when the state of the world is in . This is because covers the ground truth values in , and covers the possible network outputs for all inputs that can be generated from . From these error bounds , we compute a global error bound:
(4) 
We can also compute a local error bound for any feasible input :
(5) 
Note that provides a tighter local error bound. But since it is generally much easier to check containment of in ’s than in ’s, we adopt the current formulation.
We establish the soundness of the error bounds obtained from Tiler below.
Theorem 1 (Local error bound for regression).
Theorem 2 (Global error bound for regression).
Algorithm 1 formally presents the Tiler algorithm (for regression). The implementations of DivideStateSpace, GetGroundTruthBound, and GetBoundingBox are problem dependent. The choice of Solver needs to be compatible with and . Conditions 4.1 to 4.4 specify the sufficient conditions for the returned results from these four methods such that the guarantees obtained are sound.
The complexity of this algorithm is determined by the number of tiles, which scales with the dimension of the state space . Because the computations for each tile are independent, our Tiler implementation executes these computations using parallelism.
5 Case Study: Position Measurement from Road Scene
We next present a case study where we use Tiler
to verify the correctness of a neural network that estimates the position of a camera with respect to a road.
5.1 Problem Setup
Consider a world containing a road with a centerline, two side lines, and a camera taking images of the road. The camera is positioned at a fixed height above the road, but can vary its horizontal offset and viewing angle with respect to the centerline of the road. Figure 0(a) presents a schematic of the scene. For clarity, we describe the scene in a Cartesian coordinate system. Treating 1 length unit as roughly 5 centimeters will give a realistic scale. The scene contains a road in the plane, extending along the axis. A schematic view of the road down along the axis is shown in Figure 1(a). The road contains a centerline and two side lines, each with width . The width of the road (per lane) is , measuring from the center of the centerline to the center of each side line. Each point in the scene is associated with an intensity value in grayscale. The intensity of the side lines is , centerline , road , and sky . The intensity adopts a ramp change at each boundary between the lines and the road. The half width of the ramp is .
The schematic of the camera is shown in Figure 1(b). The camera’s height above the road is fixed at . The focal length . The image plane is divided into pixels, with pixel side length . Table 1 summarizes all the scene parameters described above.
Parameter  Description  Value  Parameter  Description  Value 

Road width  50.0  Intensity (side line)  1.0  
Line width  4.0  Intensity (centerline)  0.7  
Ramp half width  1.0  Intensity (road)  0.3  
Camera height  20.0  Intensity (sky)  0.0  
Focal length  1.0  Pixel number  32  
Pixel side length  0.16 
The state of the world is characterized by the offset and angle of the camera position. We therefore label the states as . In this problem, we consider the camera position between the range (length unit of the scene) and . This gives the state space .
The input to the neural network is the image taken by the camera. The observation process is the camera imaging process. The camera imaging process we use can be viewed as an onestep ray tracing: the intensity value of each pixel is determined by shooting a ray from the center of that pixel through the focal point, then taking the intensity of the intersection point between the ray and the scene. In this example scene, the intersection points for the top half of the pixels are in the sky (intensity 0.0). The intersection points for the lower half of the pixels are on the plane. The position of the intersection point (in the world coordinates) can be computed using a transformation from the pixel coordinates in homogeneous coordinate systems:
(6) 
is the transformation from pixel coordinates to camera coordinates. Camera coordinates have the origin at the focal point, and axes aligned with the orientation of the camera. We define the focal coordinates to have the origin also at the focal point, but with axes aligned with the world coordinates (the coordinate system used to describe the scene). is the rotation matrix that transforms from camera coordinates to focal coordinates. represents the projection to the road plane through the focal point, in the focal coordinates. Finally, is the translation matrix that transforms from the focal coordinates to the world coordinates. The transformation matrices are given below:^{2}^{2}2Note that for this imaging process, flipping the image on the image plane to the correct orientation (originally upsidedown) is equivalent to taking the image on a virtual image plane that is in front of the focal point by the focal length, by shooting rays from the focal point through the (virtual) pixel centers. We compute the intersection points from the pixel coordinates on the virtual image plane.
The variables are defined as in Table 1, with and being the offset and angle of the camera. After the intensity values of the pixels are determined, they are scaled and quantized to the range . The resulting input images ’s are 3232 gray scale images with intensities in . Figure 0(b) presents an example image. The feasible input space is the set of all images that can be taken with and .
The quantity of interest is the camera position . The ground truth function is simply . For the neural network, we use the same ConvNet architecture as in Tjeng et al. (2019) and the small network in Wong et al. (2018). It has 2 convolutional layers (size 4
4, stride 2) with 16 and 32 filters respectively, followed by a fully connected layer with 100 units. All the activation functions are ReLUs. The output layer is a linear layer with 2 output nodes, corresponding to the predictions of
and . The network is trained on 130k images and validated on 1000 images generated from our imaging process. The camera positions of the training and validation images are sampled uniformly from the range and . The network is trained with anloss function, using
Adam (Kingma and Ba, 2014) (see Appendix C for more training details).For error analysis, we treat the predictions of and separately. The goal is to find upper bounds on the prediction errors and for any feasible input .
5.2 Tiler
Figure 0(c) presents a schematic of how we apply Tiler to this problem. Tiles are constructed by dividing on into a grid of equalsized rectangles with length and width . Each cell in the grid is then , with and . Each tile is the set of images that can be observed from . The ground truth bounds can be naturally obtained from : for , and ; for , and .
We next encapsulate each tile with an norm ball by computing, for each pixel, the range of possible values it can take within the tile. A tile in this example corresponds to images taken with camera position in a local range , . For pixels in the upper half of the image, their values will always be the intensity of the sky. For each pixel in the lower half of the image, if we trace the intersection point between the projection ray of this pixel and the road plane, it will sweep over a closed region as the camera position varies in the  cell. The range of possible values for that pixel is then determined by the range of intensities in that region. In this example, there is an efficient way of computing the range of intensities in the region of sweep. Since the intensities on the road plane only vary with , it suffices to find the span on for the region. The extrema on can only be achieved at: 1) the four corners of the  cell; 2) the points on the two edges and where makes the ray of that pixel perpendicular to the axis (if that is contained in ). Therefore, by computing the location of these critical points, we can obtain the range of . We can then obtain the range of intensities covered in the region of sweep, which will give the range of pixel values. The resulting is an norm ball in the image space covering , represented by 3232 pixelwise ranges.
To solve the range of outputs of the ConvNet for inputs in the norm ball, we adopt the approach from Tjeng et al. (2019)
. They formulate the robustness verification problem as mixed integer linear program (MILP), by expressing the network computations and input constraints as linear or integer constraints. Presolving on ReLU stability and progressive bound tightening are used to improve efficiency. We adopt the same formulation but change the MILP objectives. For each
norm ball, we solve 4 optimization problems: maximizing and minimizing the output entry for , and another two for . To speed up solving, we reuse the results from presolving across the 4 optimization problems of the same tile (they share the same input constraints and thus the same bounds on the internal variables of the network). Denote the optimized objectives we obtain as , , , .5.3 Experimental Results
We run Tiler with a cell size of 0.1 (the side length of each cell in the grid is ). The step that takes the majority of time is the optimization solver. With parallelism, the optimization step takes about 15 hours running on 40 CPUs@3.00 GHz, solving MILP problems.
Global error bound: We compute global error bounds by taking the maximum of and over all tiles. The global error bound for is 12.66, which is 15.8% of the measurement range (80 length units for ); for is (5.94% of the measurement range). We therefore successfully verify the correctness of the network with these tolerances for all feasible inputs.
Error bound landscape: We present the visualizations of the error bound landscape by plotting the error bounds of each tile as heatmaps over the space. Figures 2(a) and 2(d) present the resulting heatmaps for and , respectively. To further inspect the distribution of the error bounds, we compute the percentage of the state space (measured on the grid) that has error bounds below some threshold value. The percentage varying with threshold value can be viewed as a cumulative distribution. Figures 2(c) and 2(f) present the cumulative distributions of the error bounds. It can be seen that most of the state space can be guaranteed with much lower error bounds, with only a small percentage of the regions having larger guarantees. This is especially the case for the offset measurement: 99% of the state space is guaranteed to have error less than 2.65 (3.3% of the measurement range), while the global error bound is 12.66 (15.8%).
A key question for the error bounds is how well they reflect the actual maximum error made by the neural network. To study the tightness of the error bounds, we compute empirical estimates of the maximum errors for each , denoted as and . To do this, we sample multiple within each cell and generate input images, then take the maximum over the errors of these points as the empirical estimate of the maximum error for . The sample points are drawn on a subgrid within each cell, with sampling spacing 0.05. Notice that this estimate is a lower bound on the maximum error for . This provides a reference for evaluating the tightness of the error upper bounds we get from Tiler.
We take the maximum of ’s and ’s to get a lower bound estimate of the global maximum error. The lower bound estimate of the global maximum error for is 9.12 (11.4% of the measurement range); for is (3.4% of the measurement range). We can see that the error bounds that Tiler delivers are close to the lower bound estimates derived from the observed errors that the network exhibits for specific inputs.
Having visualized the heatmaps for the bounds and , we subtract from them the error estimates and and plot the heatmaps for the resulting gaps in Figures 2(b) and 2(e). We can see that most of the regions that have large error bounds are due to the fact that the network itself has large errors there. By computing the cumulative distributions of these gaps between bounds and estimates, we found that for angle measurement, 99% of the state space has error gap below (1.6% of measurement range); and for offset measurement, 99% of the state space has error gap below 1.41 length units (1.8%). The gaps indicate the maximum possible improvements on the error bounds.
Contributing factors to the gap: There are two main contributing factors to the gap between the bound from Tiler and the actual maximum error. First is the systematic error introduced in the way Tiler computes the error bound. Tiler takes the maximum distance between the range of possible ground truths and the range of possible network outputs as the bound. For a perfect network, this still gives a bound equal to the range of ground truth, instead of zero. The second factor is the extra space included in the box that is not on the tile . This results in a larger range on network output being used for calculating error bound, which in turn makes the error bound itself larger.
Effect of tile size: Both of the factors described above are affected by the tile size. We run Tiler with a sequence of cell sizes (0.05, 0.1, 0.2, 0.4, 0.8) for the grid. Figure 3(a) shows how the 99 percentiles of the error upper bounds and the gap between error bounds and estimates vary with cell size. As tile size gets finer, Tiler provides better error bounds, and the tightness of bounds improves.
As another perspective, consider the case where we only trust the network when the error is less than some threshold. Figure 3(b) shows how the percentage of the state space where the network is proved to be trustworthy varies with cell size, given the required accuracy being 3% of the measurement range. It can be seen that with finer tile sizes, Tiler is able to prove larger trusted region.
These results show that we can get better error bounds with finer tile sizes. But this improvement might be at the cost of time: reducing tile sizes also increases the total number of tiles and the number of optimization problems to solve. Figure 3(c) shows how the total solving time varies with cell size. For cell sizes smaller than 0.2, the trend can be explained by the above argument. For cell sizes larger than 0.2, total solving time increases with cell size instead. The reason for this is that each optimization problem becomes harder to solve as the tile becomes large. Specifically, the approach we adopt (Tjeng et al., 2019) relies on the presolving on ReLU stability to improve speed. The number of unstable ReLUs will increase drastically as the cell size becomes large, which makes the solving slower.
6 Discussion
The techniques presented in this paper work with specifications provided by the combination of a state space of the world and an observation process that converts states into neural network inputs. Results from the case study highlight how well the approach works for a state space characterized by two attributes ( and ) and a camera imaging observation process. We anticipate that the technique will also work well for other problems that have a low dimensional state space with a tractable tiling and an observation process that can be precisely modelled.
Tiler generalizes to cases where alternative methods exist for characterizing the feasible input space and the ground truth: we express the sufficient conditions (Condition 4.1 to 4.4) for soundness independent of the state space and the observation process. Tiler can therefore be applied and give sound correctness guarantees whenever there exist methods that can produce tiles, ground truth bounds, and bounding boxes that satisfy these conditions.
A final question is whether it is possible to determine whether an arbitrary input is feasible (i.e., whether ) and therefore whether the neural network is known to compute a correct output for . In general, we expect the answer to this question to depend on the precise characteristics of the specific problem, its state space, and the observation process. In particular, we are not aware of any computationally tractable method for determining if an input is feasible for the problem in our case study. However, we anticipate that it will often be straightforward to determine if an arbitrary input is contained in some bounding box . In our case study, because we use bounding boxes defined by the norm, it is straightforward to search the bounding boxes and determine if is contained in one of the bounding boxes. If so, the technique guarantees that there is an input close to (i.e., and are both within the same bounding box ) for which 1) the neural network computes an output that is correct up to the defined tolerance and 2) the output is close to the output for (i.e., and are both within the network output bound for ).
Acknowledgements
We thank Vincent Tjeng for help on the MIPVerify software package. We thank Jie Xu for helpful discussions on the graphics process. We thank Aleksander Madry and Dimitris Tsipras for useful discussions on verifying robustness of neural networks. We thank Dimitris Tsipras for helpful feedback on an earlier draft of this paper. We thank Dragos Margineantu, Charles Erignac, Tyler Staudinger, Jeffery Saunders, and Craig Bonnoit for inspirational discussions on various aspect of machine learning.
References
 (1)
 Bastani et al. (2016) Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi. 2016. Measuring Neural Net Robustness with Constraints. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS’16). Curran Associates Inc., USA, 2621–2629. http://dl.acm.org/citation.cfm?id=3157382.3157391
 Cheng et al. (2017) ChihHong Cheng, Georg Nührenberg, and Harald Ruess. 2017. Maximum Resilience of Artificial Neural Networks. In ATVA.
 Dutta et al. (2018) Souradeep Dutta, Susmit Jha, Sriram Sankaranarayanan, and Ashish Tiwari. 2018. Learning and Verification of Feedback Control Systems using Feedforward Neural Networks. IFACPapersOnLine 51, 16 (2018), 151 – 156. https://doi.org/10.1016/j.ifacol.2018.08.026 6th IFAC Conference on Analysis and Design of Hybrid Systems ADHS 2018.

Dvijotham et al. (2018)
Krishnamurthy Dvijotham,
Robert Stanforth, Sven Gowal,
Timothy Mann, and Pushmeet Kohli.
2018.
A dual approach to scalable verification of deep
networks. In
Proceedings of the ThirtyFourth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI18)
. AUAI Press, Corvallis, Oregon, 162–171.  Ehlers (2017) Rüdiger Ehlers. 2017. Formal Verification of PieceWise Linear FeedForward Neural Networks. CoRR abs/1705.01320 (2017). arXiv:1705.01320 http://arxiv.org/abs/1705.01320
 Gehr et al. (2018) T. Gehr, M. Mirman, D. DrachslerCohen, P. Tsankov, S. Chaudhuri, and M. Vechev. 2018. AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation. In 2018 IEEE Symposium on Security and Privacy (SP). 3–18. https://doi.org/10.1109/SP.2018.00058
 Huang et al. (2017) Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. 2017. Safety Verification of Deep Neural Networks. In Computer Aided Verification, Rupak Majumdar and Viktor Kunčak (Eds.). Springer International Publishing, Cham, 3–29.
 Katz et al. (2017) Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2017. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In Computer Aided Verification. Springer International Publishing, 97–117. https://doi.org/10.1007/9783319633879_5
 Kelly J. et al. (2001) Hayhurst Kelly J., Veerhusen Dan S., Chilenski John J., and Rierson Leanna K. 2001. A Practical Tutorial on Modified Condition/Decision Coverage. Technical Report.
 Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980
 Liu et al. (2019) Changliu Liu, Tomer Arnon, Christopher Lazarus, Clark Barrett, and Mykel J. Kochenderfer. 2019. Algorithms for Verifying Deep Neural Networks. CoRR abs/1903.06758 (2019). arXiv:1903.06758 http://arxiv.org/abs/1903.06758
 Lomuscio and Maganti (2017) Alessio Lomuscio and Lalit Maganti. 2017. An approach to reachability analysis for feedforward ReLU neural networks. CoRR abs/1706.07351 (2017). arXiv:1706.07351 http://arxiv.org/abs/1706.07351
 Ma et al. (2018) Lei Ma, Felix JuefeiXu, Jiyuan Sun, Chunyang Chen, Ting Su, Fuyuan Zhang, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Comprehensive and MultiGranularity Testing Criteria for Gauging the Robustness of Deep Learning Systems. CoRR abs/1803.07519 (2018). arXiv:1803.07519 http://arxiv.org/abs/1803.07519
 Odena and Goodfellow (2018) Augustus Odena and Ian J. Goodfellow. 2018. TensorFuzz: Debugging Neural Networks with CoverageGuided Fuzzing. CoRR abs/1807.10875 (2018).

Papernot et al. (2016)
Nicolas Papernot,
Patrick D. McDaniel, Somesh Jha,
Matt Fredrikson, Z. Berkay Celik, and
Ananthram Swami. 2016.
The Limitations of Deep Learning in Adversarial Settings.
2016 IEEE European Symposium on Security and Privacy (EuroS&P) (2016), 372–387.  Pei et al. (2017) Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP ’17). ACM, New York, NY, USA, 1–18. https://doi.org/10.1145/3132747.3132785
 Raghunathan et al. (2018) Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified Defenses against Adversarial Examples. In International Conference on Learning Representations. https://openreview.net/forum?id=Bys4obRb
 Sun et al. (2018) Youcheng Sun, Xiaowei Huang, and Daniel Kroening. 2018. Testing Deep Neural Networks. CoRR abs/1803.04792 (2018). arXiv:1803.04792 http://arxiv.org/abs/1803.04792
 Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. CoRR abs/1312.6199 (2014).
 Tian et al. (2018) Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deepneuralnetworkdriven Autonomous Cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18). ACM, New York, NY, USA, 303–314. https://doi.org/10.1145/3180155.3180220
 Tjeng et al. (2019) Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. 2019. Evaluating Robustness of Neural Networks with Mixed Integer Programming. In International Conference on Learning Representations. https://openreview.net/forum?id=HyGIdiRqtm
 Weng et al. (2018) TsuiWei Weng, Huan Zhang, Hongge Chen, Zhao Song, ChoJui Hsieh, Duane Boning, Inderjit S. Dhillon, and Luca Daniel. 2018. Towards Fast Computation of Certified Robustness for ReLU Networks. In International Conference on Machine Learning (ICML).
 Wong and Kolter (2018) Eric Wong and Zico Kolter. 2018. Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, Stockholmsmässan, Stockholm Sweden, 5286–5295. http://proceedings.mlr.press/v80/wong18a.html
 Wong et al. (2018) Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J. Zico Kolter. 2018. Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett (Eds.). Curran Associates, Inc., 8400–8409. http://papers.nips.cc/paper/8060scalingprovableadversarialdefenses.pdf
 Xiang et al. (2017) Weiming Xiang, HoangDung Tran, and Taylor T. Johnson. 2017. Reachable Set Computation and Safety Verification for Neural Networks with ReLU Activations. CoRR abs/1712.08163 (2017). arXiv:1712.08163 http://arxiv.org/abs/1712.08163
 Xiang et al. (2018) W. Xiang, H. Tran, J. A. Rosenfeld, and T. T. Johnson. 2018. Reachable Set Estimation and Safety Verification for Piecewise Linear Systems with Neural Network Controllers. In 2018 Annual American Control Conference (ACC). 1574–1579. https://doi.org/10.23919/ACC.2018.8431048
 Xiang et al. (2018) Weiming Xiang, HoangDung Tran, and Taylor T. Johnson. 2018. Output Reachable Set Estimation and Verification for MultiLayer Neural Networks. IEEE Transactions on Neural Networks and Learning Systems (TNNLS) (March 2018). https://doi.org/10.1109/TNNLS.2018.2808470
Appendices
Appendix A Proofs for Theorem 1 and 2
Theorem 1 (Local error bound for regression).
Proof.
For any , we have
Condition 4.1 and 4.2(a) guarantees that for any and , we can find a tile such that and . Let be a function that gives such a tile for a given and . Then
Since and (Condition 4.3), . By Condition 4.4(a),
This gives
Since for all , we have , which gives
∎
Theorem 2 (Global error bound for regression).
Proof.
Appendix B Classification
We present here the algorithm of Tiler for classification settings.
Step 1 (tiling the space) is the same as regression.
Step 2: For each , compute the ground truth bound as a set , such that .
The bounds computed this way satisfy the following condition:
Condition 4.2(b). For any , such that and .
The idea behind Condition 4.2(b) is the same as that of Condition 4.2(a), but formulated for discrete .
Step 3 (compute bounding box for each input tile) is the same as regression.
The next step is to solve the network output range. Suppose the quantity of interest has
possible classes. Then the output layer of the neural network is typically a softmax layer with
output nodes. Denote the th output score before softmax as . We use the solver to solve the range of each output score:Step 4: Given and each , use appropriate solver to solve the range of each output score for .
The bounds need to satisfy:
Condition 4.4(b) .
Step 5: For each tile, compute an error bound , using the bound on ground truth and the bound on network output :
if and only if the following conditions hold:  (7) 

only contains one element, and

for all , where is the class index of the only element in .
Otherwise, . We can then compute the global and local error bounds using Equation 4 and 5, same as in the regression case. The error bounds for classification will be binary, with 0 meaning the network is guaranteed to be correct, and 1 meaning no guarantee.
Theorem 3 (Local error bound for classification).
Proof.
We aim to prove that , if , then . First, according to the definition of , implies for all . Arbitrarily pick a , we have . According to Equation 7, only contains one element with class index , and for all . Since , Condition 4.4(b) gives:
for all . This gives . For any , we can similarly derive . Therefore, we must have for all , otherwise there will be a contradiction. In another word, we have
Now, according to Condition 4.2(b), such that and . Rewriting this condition, we get
which also means . But (Condition 4.3), which gives
Since is not empty, we have . ∎
Theorem 4 (Global error bound for classification).
Proof.
Algorithm 2 formally presents the Tiler algorithm for classification.
Appendix C Additional Training Details
This section presents the additional details of the training of the neural network in the case study. We use Adam optimizer with learning rate
. We use early stopping based on the loss on the validation set: we terminate training if the validation performance does not improve in 5 consecutive epochs. This results in the training terminated at 21 epochs. We take the model from the epoch that has the lowest loss on the validation set.
Comments
There are no comments yet.