1. Introduction
From simple logical constructs to complex deep neural network models, Artificial Intelligence (AI)agents are increasingly controlling physical/mechanical systems. Selfdriving cars, drones, and smart cities are just examples of such systems to name a few. However, regardless of the explosion in the use of AI within a multitude of cyberphysical systems (CPS) domains, the safety and reliability of these AIenabled CPS is still an understudied problem. It is then unsurprising the failure of these AIcontrolled CPS in several, safetycritical, situations leading to human fatalities
(AccidentWiki).Motivated by the urgency to study the safety, reliability, and potential problems that can rise and impact the society by the deployment of AIenabled systems in the real world, several works in the literature focused on the problem of designing deep neural networks that are robust to the socalled adversarial examples (ferdowsi2018robust; everitt2018agi; charikar2017learning; steinhardt2017certified; munoz2017towards; paudice2018label; ruan2018global)
. Unfortunately, these techniques focus mainly on the robustness of the learning algorithm with respect to data outliers without providing guarantees in terms of safety and reliability of the decisions taken by these neural networks. To circumvent this drawback, and motivated by the wealth of adversarial example generation approaches for neural networks, recent works focused on three main techniques namely (i) testing of neural networks, (ii) falsification (semiformal verification) of neural networks, and (iii) formal verification of neural networks.
Representatives of the first class, namely testing of neural networks, are the work reported in (pei2017deepxplore; tian2017deeptest; wicker2018feature; YouchengTesting2018; LeiDeepGauge2018; Wang2018Testing; LeiDeepMutation2018; srisakaokul2018multiple; MengshiDeepRoad2018; YouchengConcolic2018)
in which the neural network is treated as a white box, and test cases are generated to maximize different coverage criteria. Such coverage criteria include neuron coverage, condition/decision coverage, and multigranularity testing criteria. On the one hand, maximizing test coverage give system designers confidence that the networks are reasonably free from defect. On the other hand, testing do not formally guarantee that a neural network satisfy a formal specification.
Unfortunately, testing techniques focuses mainly on the neural network as a component without taking into consideration the effect of its decisions on the entire system behavior. This motivated researchers to focus on falsification (or semiformal verification) of autonomous systems that include machine learning components (dreossi2017compositional; tuncali2018simulation; zhang2018two). In such falsification frameworks, the objective is to generate corner test cases that will lead the whole system to violate a systemlevel specification. To that end, advanced 3D models and image environments are used to bridge the gap between the virtual world and the real world. By parametrizing the input to these 3D models (e.g., position of objects, position of light sources, intensity of light sources) and sampling the parameter space in a fashion that maximizes the falsification of the safety property, falsification frameworks can simulate several test cases until a counterexample is found (dreossi2017compositional; tuncali2018simulation; zhang2018two).
While testing and falsification frameworks are powerful tools to find corner cases in which the neural network or the neural network enabled system will fail, they lack the rigor promised by formal verification methods. Therefore, several researchers pointed to the urgent need of using formal methods to verify the behavior of neural networks and neural network enabled system (kurd2003establishing; seshia2016towards; seshia2018formal; leikeAIsafety2017; leofante2018automated; scheibler2015towards). As a result, recent works in the literature attempted the problem of applying formal verification techniques to neural network models.
Applying formal verification to neural network models comes with its unique challenges. First and foremost is the lack of widelyaccepted, precise, mathematical specifications capturing the correct behavior of a neural network. Therefore, recent works focused entirely on verifying neural networks against simple inputoutput specifications (katz2017reluplex; ehlers2017formal; bunel2018unified; ruan2018reachability; dutta2018output; pulina2010abstraction)
. Such inputoutput techniques compute a guaranteed range for the output of a deep neural network given a set of inputs represented as a convex polyhedron. To that end, several algorithms that takes advantage of the piecewise linear nature of the Rectified Linear Unit (ReLU) activation functions (one of the most famous nonlinear activation functions in deep neural networks) have been proposed. For example, by using binary variables to encode piecewise linear functions, the constraints of ReLU functions are encoded as a MixedInteger Linear Programming (MILP). Combining output specifications that are expressed in terms of Linear Programming (LP), the verification problem eventually turns to a MILP feasibility problem
(dutta2018output; tjeng2017verifying).Using offtheshelf MILP solvers does not lead to a scalable approach to handle neural networks with hundreds and thousands of neurons (ehlers2017formal). To circumvent this problem, several MILPlike solvers targeted toward the neural network verification problem are proposed. For example, the work reported in (katz2017reluplex) proposed a modified Simplex algorithm (originally used to solve linear programs) to take into account ReLU nonlinearities as well. Similarly, the work reported in (ehlers2017formal) combines a Boolean satisfiability solving along with a linear overapproximation of piecewise linear functions to verify ReLU neural networks against convex specifications. Other techniques that exploit specific geometric structures of the specifications are also proposed (gehr2018ai; xiang2017reachable). A thorough survey on different algorithms for verification of neural networks against inputoutput range specifications can be found in (xiang2017survey) and the references within.
Unfortunately, the inputoutput range properties, studied so far in the literature, are simplistic and fails to capture the safety and reliability of cyberphysical systems when controlled by a neural network. Therefore, in this paper, we focus instead on the problem of formal verification of a neural network controlled robot against systemlevel safety specifications. In particular, we consider the problem in which a robot utilizes a LiDAR scanner to sense its environment. The LiDAR image is then processed by a neural network controller to compute the control inputs. Such scenario is common in the literature of behavioral cloning and imitation control in which the neural network is trained to imitate the actions generated by experts controlling the robot (kahn2017plato). With the objective to verify the safety of these robots, we develop a framework that can take into account the robot continuous dynamics, the workspace configuration, the LiDAR imaging, and the neural network, and compute the set of initial states for the robot that is guaranteed to produce robot trajectories that are safe and collision free.
To carry out the prescribed formal verification problem, we need a mathematical model that captures the LiDAR imaging process. This is the process that generates the LiDAR images based on the robot pose relative to the workspace objects. Therefore, the first contribution of this paper is to introduce the notion of imagingadapted sets. These are workspace partitions within which the map between robot pose and LiDAR images are mathematically captured by an affine map. Given this notion, and thanks to the advances in the literature of computational graphics, we develop a polynomialtime algorithm that can partition the workspace into imagingadapted sets along with the corresponding affine maps.
Given the partitioned workspace along with a pretrained neural network and the robot dynamics, we compute a finite state abstraction of the closed loop system that enjoys a simulation relation with the original system. The main challenge in computing this finite state abstraction is to analyze the behavior of the neural network controller. Similar to previous works in the literature, we strict our focus to neural networks with Rectified Linear Unit (ReLU) nonlinearities and we develop a Satisfiability Modulo Convex (SMC) programming algorithm that uses a combination of a Boolean satisfiability solver and a convex programming solver to iteratively reason about the neural network nonlinearity along with the dynamics and the imaging constraints. At each iteration, the boolean satisfiability solver searches for a candidate assignment determining whether ReLU units are active while ignoring the neural network weights, the robot dynamics, and the LiDAR imaging. The convex programming solver is then used to check the feasibility of the proposed ReLU assignment against the neural network weights, the robot dynamics, and the LiDAR imaging. If the ReLU assignment is deemed infeasible, then the SMC solver will generate succinct explanations for their infeasibility to reduce the search space. To accelerate the process, we develop a preprocessing algorithm that can reduce the space of ReLU assignments.
Once the finite state abstraction is computed, we use standard reachability analysis techniques to compute the set of safe initial states.
To summarize, the contributions of this paper can be summarized as follows:
1 A framework for formally proving safety properties of autonomous robots controlled by neural network controllers that process LiDAR images to compute control inputs.
2 A notion of imagingadapted sets along with a polynomialtime algorithm for partitioning the workspace into such sets while computing an affine model capturing the LiDAR imaging process.
3 An SMCbased algorithm combined with an SMCbased preprocessing for computing finite abstractions of the neural network controlled autonomous robot.
2. Problem Formulation
2.1. Notation
The symbols , and denote the set of natural, real, positive real, and Boolean numbers, respectively. The symbols and
denote the logical AND, logical NOT, and logical IMPLIES operators, respectively. Given two realvalued vectors
and , we denote by the column vector . Similarly, for a vector , we denote by the th element of . For two vectors , we denote by the elementwise maximum. For a set , we denote the boundary and the interior of this set by and , respectively. Given two sets and , and denote a setvalued and ordinary map, respectively. Finally, given a vector , we denote by .2.2. Dynamics and Workspace
We consider an autonomous robot moving in a 2dimensional polytopic (compact and convex) workspace . We assume that the robot must avoid the workspace boundaries along with a set of obstacles , with which is assumed to be polytopic. We denote by the set of the obstacles and the workspace boundaries which needs to be avoided, i.e., . The dynamics of the robot is described by a discretetime linear system of the form:
(1) 
where is the state of robot at time and is the robot input. The matrices and represent the robot dynamics and have appropriate dimensions. For a robot with nonlinear dynamics that is either differentially flat or feedback linearizable, the state space model (1) corresponds to its feedback linearized dynamics. We denote by the natural projection of onto the workspace , i.e., is the position of the robot at time .
2.3. LiDAR Imaging
We consider the case when the autonomous robot uses a LiDAR scanner to sense its environment. The LiDAR scanner emits a set of lasers evenly distributed in a degree fan. We denote by the heading angle of the LiDAR at time . Similarly, we denote by , with , the angle of the th laser beam at time where and by the vector of the angles of all the laser beams. While the heading angle of the LiDAR, , changes as the robot poses changes over time, i.e.:
for some nonlinear function , in this paper we focus on the case when the heading angle of the LiDAR, , is fixed over time and we will drop the superscript from the notation. Such condition is satisfied in many realworld scenarios whenever the robot is moving while maintaining a fixed pose (e.g. a Quadrotor whose yaw angle is maintained constant).
For the th laser beam, the observation signal is the distance measured between the robot position and the nearest obstacle in the direction, i.e.:
(2) 
The final LiDAR image is generated by processing the observations as follows:
(3) 
2.4. Neural Network Controller
We consider a pretrained neural network controller that processes the LiDAR images to produce control actions with internal and fully connected layers in addition to one output layer. Each layer contains a set of neurons (where ) with Rectified Linear Unit (ReLU) activation functions. ReLU activation functions play an important role in the current advances in deep neural networks (krizhevsky2012imagenet). For such neural network architecture, the neural network controller can be written as:
(4) 
where and
are the pretrained weights and bias vectors of the neural network which are determined during the training phase.
2.5. Robot Trajectories and Safety Specifications
The trajectories of the robot whose dynamics are described by (1) when controlled by the neural network controller (2)(4) starting from the initial condition is denoted by such that . A trajectory is said to be safe whenever the robot position does not collide with any of the obstacles at all times.
Definition 2.1 (Safe Trajectory).
A robot trajectory is called safe if:
Using the previous definition, we now define the problem of verifying the systemlevel safety of the neural network controlled system as follows:
3. Framework
Before we describe the proposed framework, we need to briefly recall the following set of definitions capturing the notion of a system and relations between different systems.
Definition 3.1 ().
An autonomous system is a pair consisting of a set of states and a setvalued map representing the transition function. A system is finite if is finite. A system is deterministic if is singlevalued map and is nondeterministic if not deterministic.
Definition 3.2 ().
Consider a deterministic system and a nondeterministic . A relation is a simulation relation from to , and we write , if the following conditions are satisfied:

for every there exists with ,

for every we have that in implies the existence of in satisfying .
Using the previous two definitions, we describe our approach as follows. As pictorially shown in Figure 2, given the autonomous robot system , where , our objective is to compute a finite state abstraction (possibly nondeterministic) of such that there exists a simulation relation from to , i.e., . This finite state abstraction will be then used to check the safety specification.
The first difficulty in computing the finite state abstraction is the nonlinearity in the relation between the robot position and the LiDAR observations as captured by equation (2). However, we notice that we can partition the workspace based on the laser angles along with the vertices of the polytopic obstacles such that the map (which maps the robot position to the processed observations) is an affine map. Therefore, as summarized in Algorithm 1, the first step is to compute such partitioning of the workspace (WKSPPARTITION, line 2 in Algorithm 1). While WKSPPARTITION focuses on partitioning the workspace , one need to partition the remainder of the state space (STATESPACEPARTITION, line 5 in Algorithm 1) to compute the finite set of abstract states along with the simulation relation that maps between states in and the corresponding abstract states in , and vice versa.
Unfortunately, the number of partitions grows exponentially in the number of lasers and the number of vertices of the polytopic obstacles. To harness this exponential growth, we compute an aggregatepartitioning using only few laser angles (called primary lasers and denoted by ). The resulting aggregatepartitioning would contain a smaller number of partitions such that each partition in represents multiple partitions in . Similarly, we can compute a corresponding aggregate set of states as:
where each aggregate state is a set representing multiple states in . Whenever possible, we will carry out our analysis using the aggregatedpartitioning (and ) and use the finepartitioning only if deemed necessary. Details of the workspace partitioning and computing the corresponding affine maps representing the LiDAR imaging function are given in Section 4.
The state transition map is computed as follows. First, we assume a transition exists between any two states and in (line 6 7 in Algorithm 1). Next, we start eliminating unnecessary transitions. We observe that regions in the workspace that are adjacent or within some vicinity are more likely to force the need of transitions between their corresponding abstract states. Similarly, regions in the workspace that are far from each other are more likely to prohibit transitions between their corresponding abstract states. Therefore, in an attempt to reduce the number of computational steps in our algorithm, we check the transition feasibility between a state and an aggregate state . If our algorithm (CHECKFEASIBILITY) asserted that the neural network prohibits the robot from transitioning between the regions corresponding to and (denoted by and , respectively), then we conclude that no transition in is feasible between the abstract state and all the abstract states in (lines 1319 in Algorithm 1). This leads to a reduction in the number of state pairs that need to be checked for transition feasibility. Conversely, if our algorithm (CHECKFEASIBILITY) asserted that the neural network allows for a transition between the regions corresponding to and , then we proceed by checking the transition feasibility between the state and all the states contained in the aggregate state (lines 2125 in Algorithm 1).
Checking the transition feasibility (CHECKFEASIBILITY) between two abstract states entail reasoning about the robot dynamics, the neural network, along with the affine map representing the LiDAR imaging computed from the previous workspace partitioning. While the robot dynamics is assumed linear, the imaging function is affine; the technical difficulty lies in reasoning about the behavior of the neural network controller. Thanks to the ReLU activation functions in the neural network, we can encode the problem of checking the transition feasibility between two regions as formula , called monotone Satisfiability Modulo Convex (SMC) formula (shoukry2017smc; shoukry2018smc), over Boolean and convex constraints representing, respectively, the ReLU phases and the dynamics, the neural network weights, and the imaging constraints. In addition to using SMC solver to check the transition feasibility (CHECKFEASIBILITY) between abstract states, it will be used also to perform some preprocessing of the neural network function (lines 911 in Algorithm 1) which is going to speed up the process of checking the the transition feasibility. Details of the SMC encoding are given in Section 5.
Once the finite state abstraction and the simulation relation is computed, the next step is to partition the finite states into a set of unsafe states and a set of safe states using the following fixedpoint computation:
where the represents the abstract state corresponding to the obstacles and workspace boundaries while with represents all the states that can reach in steps where:
The remaining abstract states are then marked as the set of safe states . Finally, we can compute the set of safe states as:
These computations are summarized in lines 2736 in Algorithm 1.
4. ImagingAdapted Workspace Partitioning
We start by introducing the notation of the important geometric objects. We denote by the ray originated from a point in the direction , i.e.:
Similarly, we denote by the line segment between the points and , i.e.:
For a convex polytope , we denote by , its set of vertices and by its set of line segments representing the edges of the polyhedron.
4.1. ImagingAdapted Partitions
The basic idea behind our algorithm is to partition the workspace into a set of polytypic sets (or regions) such that for each region the LiDAR rays will intersect the same obstacle/workspace edge regardless of the robot positions . To formally characterize this property, let be the set of all points in the workspace in which an obstacle or workspace boundary exists. Consider a workspace partition and a robot position that lies inside this partition, i.e., . The intersection between the th LiDAR laser beam and is a unique point characterized as:
(5) 
By sweeping across the whole region , we can characterize the set of all possible intersection points as:
(6) 
Using the set described above, we define the notion of imagingadapted set as follows.
Definition 4.1 ().
A set is said to be an imagingadapted partition if the following property holds:
(7) 
Figure 3 shows concrete examples of imagingadapted partitions. Imagingadapted partitions enjoys the following property:
Lemma 4.2 ().
Consider an imagingadapted partition with corresponding sets . The LiDAR imaging function is an affine function of the form:
(8) 
for some constant matrices and vectors that depend on the region and the LiDAR angle .
4.2. Partitioning the Workspace
Motivated by Lemma 4.2, our objective is to design an algorithm that can partition the workspace into a set of imagingadapted partitios. As summarized in Algorithm 2, our algorithm starts by computing a set of line segments that will be used to partition the workspace (lines 15 in Algorithm 2). This set of line segments are computed as follows. First, we define the set as the one that contains all the vertices of the workspace and the obstacles, i.e., . Next, we consider rays originating from all the vertices in and pointing in the opposite direction of the angles . By intersecting these rays with the obstacles and picking the closest intersection points, we acquire the line segments that will be used to partition the workspace. In other words, is computed as:
(9) 
Thanks to the fact that the vertices are fixed, finding the intersection between and is a standard raypolytope intersection problem which can be solved efficiently (berg2008computational).
The next step is to compute the intersection points between the line segments and the edges of the obstacles . A naive approach will be to consider all combinations of line segments in and test them for intersection. Such approach is combinatorial and would lead to an execution time that is exponential in the number of laser angles and vertices of obstacles. Thanks to the advancements in the literature of computational geometry, such intersection points can be computed efficiently using the PlaneSweep algorithm (berg2008computational). The planesweep algorithm simulates the process of sweeping a line downwards over the plane. The order of the line segments from left to right as they intersect the sweep line is stored in a data structure called the sweepline status. Only segments that are adjacent in the horizontal ordering need to be tested for intersection. Though the sweeping process can be visualized as continuous, the planesweep algorithm sweeps only the values in which the endpoints of segments in , which are given beforehand, and the intersection points, which are computed on the fly. To keep track of the endpoints of segments in and the intersection points, we use a balanced binary search tree as data structure to support insertion, deletion, and searching in time, where is number of elements in the data structure.
The final step is to use the line segments and their intersection points , discovered by the planesweep algorithm, to compute the workspace partitions. To that end, consider the undirected planar graph whose vertices are the intersection points and whose edges are , denoted by . The workspace partitions are equivalent to finding subgraphs of such that each subgraph contains only one simple cycle ^{1}^{1}1A cycle in an undirected graph is called simple when no repetitions of vertices and edges is allowed within the cycle.. To find these simple cycles, we use a modified DepthFirstSearch algorithm in which it starts from a vertex in the planar graph and then traverses the graph by considering the rightmost turns along the vertices of the graph. Finally, the workspace partition is computed as the convex hull of all the vertices in the computed simple cycles. It follows directly from the fact that each region is constructed from the vertices of a simple cycle that there exists no line segment in that intersects with the interior of any region, i.e., for any workspace partition , the following holds:
(10) 
This process is summarized in lines 816 in Algorithm 2. An important property of the regions determined by Algorithm 2 is stated by the following proposition.
Proposition 4.3 ().
We conclude this section by stating our first main result, quantifying the correctness and complexity of Algorithm 2.
Theorem 4.4 ().
5. Computing the Finite State Abstraction
Once the workspace is partitioned into imagingadapted regions and the corresponding imaging function is identified, the next step is to compute the finite state transition abstraction of the closed loop system along with the simulation relation . The first step is to define the state space and its relation to . To that end, we start by computing a partitioning of the state space that respects . For the sake of simplicity, we consider that is orthotope, i.e., there exists constants such that:
Now, given a discretization parameter , we define the state space as:
(11) 
where is the number of regions in the partitioning . In other words, the parameter is used to partition the state space into hypercubes. A state represents the index of a region in followed by the indices identifying a hypercube in the remaining dimensions. Note that for the simplicity of notation, we assume that is divisible by for all . We now define the relation as:
(12) 
Finally, we define the state transition function of as follows:
(13) 
It follows from the definition of in (13) that checking the transition feasibility between two states and is equivalent to searching for a robot initial and goal states along with a LiDAR image that will force the neural network controller to generate an input that moves the robot between the two states while respecting the robots dynamics. In the reminder of this section, we focus on solving this feasibility problem.
5.1. SMC Encoding of NN
We translate the problem of checking the transition feasibility in into a feasibility problem over a monotone SMC formula (shoukry2017smc; shoukry2018smc) as follows. We introduce the Boolean indicator variables with and (recall that represents the number of layers in the neural network, while represents the number of neurons in the th layer). These Boolean variables represents the phase of each ReLU, i.e., an asserted indicates that the output of the th ReLU in the th layer is while a negated indicates that . Using these Boolean indicator variables, we encode the problem of checking the transition feasibility between two states and as:
(14)  
subject to:  
(15)  
(16)  
(17)  
(18)  
(19)  
(20)  
(21)  
(22) 
where (15)(16) encodes the state space partitions corresponding to the states and ; (17) encodes the dynamics of the robot; (18) encodes the imaging function that maps the robot position into LiDAR image; (19)(22) encodes the neural network controller that maps the LiDAR image into a control input.
Compared to MixedInteger Linear Programs (MILP), monotone SMC formulas avoid using encoding heuristics like bigM encodings which leads to numerical instabilities. SMC decision procedures follow an iterative approach combining efficient Boolean Satisfiability (SAT) solving with numerical convex programming. When applied to the encoding above, at each iteration the SAT solver generates a candidate assignment for the ReLU indicator variables
. The correctness of these assignments are then checked by solving the corresponding set of convex constraints. If the convex program turned to be infeasible, indicating a wrong choice of the ReLU indicator variables, the SMC solver will identify the set of “Irreducible Infeasible Set” (IIS) in the convex program to provide the most succinct explanation of the conflict. This IIS will be then fed back to the SAT solver to prune its search space and provide the next assignment for the ReLU indicator variables. SMC solvers was shown to better handle problems (compared with MILP solvers) for problems with relatively large number of Boolean variables (shoukry2018smc).5.2. Pruning Search Space By Preprocessing
While a neural network with ReLUs would give rise to combinations of possible assignments to the corresponding Boolean indicator variables, we observe that several of those combinations are infeasible for each workspace region. In other words, the LiDAR imaging function along with the workspace region enforces some constraints on the inputs to the neural network which in turn enforces constraints on the subsequent layers. By performing preprocessing on each of the workspace regions, we can discover those constraints and augment it to the SMC encoding (15)(22) to prune several combinations of assignments of the ReLU indicator variables.
To find such constraints, we consider an SMC problem with the fewer constraints (15), (18)(22). By iteratively solving the reduced SMC problem and recording all the IIS conflicts produced by the SMC solver, we can compute a set of counterexamples that are unique for each region. By iteratively invoking the SMC solver while adding previous counterexamples as constraints until the problem is no longer satisfiable, we compute the set which represents all the counterexamples for region . Finally, we add the following constraint:
(23) 
to the original SMC encoding (15)(22) to prune the set of possible assignments to the ReLU indicator variables. In Section 6, we show that preprocessing would result in an order of magnitude reduction in the execution time.
5.3. Correctness of Algorithm 1
We end our discussion with the following results which asserts the correctness of the whole framework described in this paper. We first start by establishing the correctness of computing the finite abstraction along with the simulation relation as follows:
Proposition 5.1 ().
Recall that Algorithm 1 applies standard reachability analysis on to compute the set of unsafe states. It follows directly from the correctness of the simulation relation established above that our algorithm computes an overapproximation of the set of unsafe states, and accordingly an underapproximation of the set of safe states. This fact is captured by the following result that summarizes the correctness of the proposed framework:
Theorem 5.2 ().
Consider the safe set computed by Algorithm 1. Then any trajectory with is a safe trajectory.
While Theorem 5.2 establishes the correctness of the proposed framework in Algorithm 1, two points needs to be investigated namely (i) complexity of Algorithm 1 and (ii) maximality of the set . Although Algorithm 2 computes the imagingadapted partitions efficiently (as shown in Theorem 4.4), analyzing a neural network with ReLU activation functions is shown to be NPhard. Exacerbating the problem, Algorithm 1 entails analyzing the neural network a number of times that is exponential in the number of partition regions. In Section 6, we experiment the efficiency of using SMC decision procedures to harness this computational complexity. As for the maximality of the computed set, we note that Algorithm 1 is not guaranteed to search for the maximal .
6. Results
We implemented the proposed verification framework as described by Algorithm 1 on top of the SMC solver named SATEX (satex). All experiments were executed on an Intel Core i7 2.5GHz processor with 16 GB of memory.
6.1. Scalability of the Workspace Partitioning Algorithm:
As the first step of our verification framework, imagingadapted workspace partitioning is tested for numerical stability with increasing number of laser angles and obstacles. Table 1 summarizes the scalability results in terms of the number of computed regions and the execution time grows as the number of LiDAR lasers and obstacle vertices increase. Thanks to adopting wellstudied computational geometry algorithms, our partitioning process takes less than minutes for the scenario where a LiDAR scanner is equipped with 298 lasers (realworld LiDAR scanners are capable of providing readings from 270 laser angles).
Number of  Number of  Number of  Time 

Vertices  Lasers  regions  [s] 
8  111  0.0152  
8  38  1851  0.3479 
118  17237  5.5300  
8  136  0.0245  
10  38  2254  0.4710 
118  20343  6.9380  
8  137  0.0275  
38  2418  0.5362  
12  120  23347  8.0836 
218  76337  37.0572  
298  142487  86.6341 
6.2. Computational Reduction Due to Preprocessing
The second step is to preprocess the neural network. In particular, we would like to answer the following question: given a partitioned workspace, how many ReLU assignments are feasible in each region, and if any, what is the execution time to find them out. Recall that a ReLU assignment is feasible if there exist a robot position and the corresponding LiDAR image that will lead to that particular ReLU assignment.
Thanks to the IIS counterexample strategy, we can find all feasible ReLU assignments in preprocessing. Our first observation is that the number of feasible assignments is indeed much smaller compared to the set of all possible assignments. As shown in Table 2, for a neural network with a total of 32 neurons, only 11 ReLU assignments are feasible (within the region under consideration). Comparing this number to possibilities of ReLU assignments, we conclude that preprocessing is very effective in reducing the search space by several orders of magnitude.
Furthermore, we conducted an experiment to study the scalability of the proposed preprocessing for an increasing number of ReLUs. To that end, we fixed one choice of workspace regions while changing the neural network architecture. The execution time, the number of generated counterexamples, along with the number of feasible ReLU assignments are given in Table 2. For the case of neural networks with one hidden layer, our implementation of the counterexample strategy is able to find feasible ReLU assignments for a couple of hundreds of neurons in less than 4 minutes. In general, the number of counterexamples, and hence feasible ReLU assignments, and execution time grows with the number of neurons. However, the number of neurons is not the only deciding factor. Our experiments show that the depth of the network plays a significant role in affecting the scalability of the proposed algorithms. For example, comparing the neural network with one hidden layer and a hundred neurons per layer versus the network with two layers and fifty neurons per layer we notice that both networks share the same number of neurons. Nevertheless, the deeper network resulted in one order of magnitude increase regarding the number of generated counterexamples and one order of magnitude increase in the corresponding execution time. Interestingly, both of the architectures share a similar number of feasible ReLU assignments. In other words, similar features of the neural network can be captured by fewer counterexamples whenever the neural network has fewer layers. This observation can be accounted for the fact that counterexamples that correspond to ReLUs in early layers are more powerful than those involves ReLUs in the later layers of the network.
In the second part of this experiment, we study the dependence of the number of feasible ReLU assignments on the choice of the workspace region. To that end, we fix the architecture of the neural network to one with hidden layers and neurons per layer. Table 3 reports the execution time, the number of counterexamples, and the number of feasible ReLU assignments across different regions of the workspace. In general, we observe that the number of feasible ReLU assignments increases with the size of the region.
Number  Total  Number of  Number of  Time 

of Hidden  Number  feasibile  Counter  [s] 
Layers  of Neurons  ReLU Assignments  Examples  
32  11  60  2.7819  
72  31  183  11.4227  
92  58  265  18.4807  
102  68  364  43.2459  
152  101  540  78.3015  
172  146  778  104.4720  
202  191  897  227.2357  
1  302  383  1761  656.3668 
402  730  2614  1276.4405  
452  816  4325  1856.0418  
502  1013  3766  2052.0574  
552  1165  4273  4567.1767  
602  1273  5742  6314.4890  
652  1402  5707  7166.3059  
702  1722  6521  8813.1829  
22  3  94  1.3180  
42  19  481  10.9823  
62  35  1692  53.2246  
82  33  2685  108.2584  
2  102  58  5629  292.7412 
122  71  9995  739.4883  
142  72  18209  2098.0220  
162  98  34431  6622.1830  
182  152  44773  12532.8552  
32  5  319  5.7227  
3  47  7  5506  148.8727 
62  45  72051  12619.5353  
4  22  9  205  10.4667 
42  5  1328  90.1148 
Region  Number of  Number of  Time 

Index  feasibile  Counter  [s] 
ReLU Assignments  Examples  
A2R3  33  2685  108.2584 
A14R1  55  4925  215.8251 
A13R3  7  1686  69.4158 
A1R1  25  2355  99.2122 
A7R1  26  3495  139.3486 
A12R2  3  1348  54.4548 
A15R3  25  3095  121.7869 
A19R1  38  4340  186.6428 
6.3. Transition Feasibility
Following our verification streamline, the next step is to compute the transition function of the finitestate abstraction , i.e., check transition feasibility between regions. Table 4 shows performance comparison between our proposed strategy that uses counterexamples obtained from preprocessing and SMC encoding without preprocessing. We observe that SMC encodings empowered by counterexamples, generated through the preprocessing phase, scales more favorably compared to the ones that do not take counterexamples into account leading to 23 orders of magnitude reduction in the execution time. Moreover, and thanks to the preprocessing counterexamples, we observe that checking transition feasibility becomes less sensitive to changes in the neural network architecture as shown in Table 4.
Number of  Total Number  Time [s]  Time [s] 

Hidden Layers  of Neurons  (Exploit Counter  (Without Counter 
Examples)  Examples)  

82  0.5056  50.1263 
102  7.1525  timeout  
1  112  12.524  timeout 
122  18.0689  timeout  
132  20.4095  timeout  
22  0.1056  15.8841  
42  4.8518  timeout  
62  3.1510  timeout  
82  2.6112  timeout  
2  102  11.0984  timeout 
122  3.8860  timeout  
142  0.7608  timeout  
162  2.7917  timeout  
182  193.6693  timeout  
32  0.3884  388.549  
3  47  0.9034  timeout 
62  59.393  timeout 
7. Conclusions
We presented a framework to verify the safety of autonomous robots equipped with LiDAR scanners and controlled by neural network controllers. Thanks to the notion of imagingadapted sets, we can partition the workspace to render the problem amenable to formal verification. Using SMCencodings, we presented a framework to compute finitestate abstraction of the system that can be used to compute an underapproximation of the set of safe robot states. We demonstrated a preprocessing technique that generates a set of counterexamples which resulted in 23 orders of magnitude execution time reduction. Future work includes investigating further strategies for efficient generation of preprocessing counterexamples along with extending the proposed technique to account to uncertainties in the LiDAR scanner.
References
Appendix A Proof of Lemma 4.2
Proof.
Consider an arbitrary LiDAR laser with an angle and arbitrary robot position . The LiDAR image can be written as:
(24) 
where is defined in (5). It follows from the fact that is an imagingadapted partition that the set is a line segment. Let be the vertices of this line segment, i.e., and recall that satisfies and hence lies on the line segment . Therefore there exists a such that:
(25) 
where
Comments
There are no comments yet.