Formal Verification of Neural Network Controlled Autonomous Systems

by   Xiaowu Sun, et al.
University of Maryland

In this paper, we consider the problem of formally verifying the safety of an autonomous robot equipped with a Neural Network (NN) controller that processes LiDAR images to produce control actions. Given a workspace that is characterized by a set of polytopic obstacles, our objective is to compute the set of safe initial conditions such that a robot trajectory starting from these initial conditions is guaranteed to avoid the obstacles. Our approach is to construct a finite state abstraction of the system and use standard reachability analysis over the finite state abstraction to compute the set of the safe initial states. The first technical problem in computing the finite state abstraction is to mathematically model the imaging function that maps the robot position to the LiDAR image. To that end, we introduce the notion of imaging-adapted sets as partitions of the workspace in which the imaging function is guaranteed to be affine. We develop a polynomial-time algorithm to partition the workspace into imaging-adapted sets along with computing the corresponding affine imaging functions. Given this workspace partitioning, a discrete-time linear dynamics of the robot, and a pre-trained NN controller with Rectified Linear Unit (ReLU) nonlinearity, the second technical challenge is to analyze the behavior of the neural network. To that end, we utilize a Satisfiability Modulo Convex (SMC) encoding to enumerate all the possible segments of different ReLUs. SMC solvers then use a Boolean satisfiability solver and a convex programming solver and decompose the problem into smaller subproblems. To accelerate this process, we develop a pre-processing algorithm that could rapidly prune the space feasible ReLU segments. Finally, we demonstrate the efficiency of the proposed algorithms using numerical simulations with increasing complexity of the neural network controller.



There are no comments yet.


page 1

page 2

page 3

page 4


Safe-by-Repair: A Convex Optimization Approach for Repairing Unsafe Two-Level Lattice Neural Network Controllers

In this paper, we consider the problem of repairing a data-trained Recti...

Formal Verification of Stochastic Systems with ReLU Neural Network Controllers

In this work, we address the problem of formal safety verification for s...

Case Study: Verifying the Safety of an Autonomous Racing Car with a Neural Network Controller

This paper describes a verification case study on an autonomous racing c...

Bounding the Complexity of Formally Verifying Neural Networks: A Geometric Approach

In this paper, we consider the computational complexity of formally veri...

Two-Level Lattice Neural Network Architectures for Control of Nonlinear Systems

In this paper, we consider the problem of automatically designing a Rect...

Assured Neural Network Architectures for Control and Identification of Nonlinear Systems

In this paper, we consider the problem of automatically designing a Rect...

Abstraction of Linear Consensus Networks with Guaranteed Systemic Performance Measures

A proper abstraction of a large-scale linear consensus network with a de...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

From simple logical constructs to complex deep neural network models, Artificial Intelligence (AI)-agents are increasingly controlling physical/mechanical systems. Self-driving cars, drones, and smart cities are just examples of such systems to name a few. However, regardless of the explosion in the use of AI within a multitude of cyber-physical systems (CPS) domains, the safety and reliability of these AI-enabled CPS is still an under-studied problem. It is then unsurprising the failure of these AI-controlled CPS in several, safety-critical, situations leading to human fatalities 


Motivated by the urgency to study the safety, reliability, and potential problems that can rise and impact the society by the deployment of AI-enabled systems in the real world, several works in the literature focused on the problem of designing deep neural networks that are robust to the so-called adversarial examples (ferdowsi2018robust; everitt2018agi; charikar2017learning; steinhardt2017certified; munoz2017towards; paudice2018label; ruan2018global)

. Unfortunately, these techniques focus mainly on the robustness of the learning algorithm with respect to data outliers without providing guarantees in terms of safety and reliability of the decisions taken by these neural networks. To circumvent this drawback, and motivated by the wealth of adversarial example generation approaches for neural networks, recent works focused on three main techniques namely (i) testing of neural networks, (ii) falsification (semi-formal verification) of neural networks, and (iii) formal verification of neural networks.

Representatives of the first class, namely testing of neural networks, are the work reported in (pei2017deepxplore; tian2017deeptest; wicker2018feature; YouchengTesting2018; LeiDeepGauge2018; Wang2018Testing; LeiDeepMutation2018; srisakaokul2018multiple; MengshiDeepRoad2018; YouchengConcolic2018)

in which the neural network is treated as a white box, and test cases are generated to maximize different coverage criteria. Such coverage criteria include neuron coverage, condition/decision coverage, and multi-granularity testing criteria. On the one hand, maximizing test coverage give system designers confidence that the networks are reasonably free from defect. On the other hand, testing do not formally guarantee that a neural network satisfy a formal specification.

Unfortunately, testing techniques focuses mainly on the neural network as a component without taking into consideration the effect of its decisions on the entire system behavior. This motivated researchers to focus on falsification (or semi-formal verification) of autonomous systems that include machine learning components (dreossi2017compositional; tuncali2018simulation; zhang2018two). In such falsification frameworks, the objective is to generate corner test cases that will lead the whole system to violate a system-level specification. To that end, advanced 3D models and image environments are used to bridge the gap between the virtual world and the real world. By parametrizing the input to these 3D models (e.g., position of objects, position of light sources, intensity of light sources) and sampling the parameter space in a fashion that maximizes the falsification of the safety property, falsification frameworks can simulate several test cases until a counterexample is found (dreossi2017compositional; tuncali2018simulation; zhang2018two).

While testing and falsification frameworks are powerful tools to find corner cases in which the neural network or the neural network enabled system will fail, they lack the rigor promised by formal verification methods. Therefore, several researchers pointed to the urgent need of using formal methods to verify the behavior of neural networks and neural network enabled system (kurd2003establishing; seshia2016towards; seshia2018formal; leikeAIsafety2017; leofante2018automated; scheibler2015towards). As a result, recent works in the literature attempted the problem of applying formal verification techniques to neural network models.

Applying formal verification to neural network models comes with its unique challenges. First and foremost is the lack of widely-accepted, precise, mathematical specifications capturing the correct behavior of a neural network. Therefore, recent works focused entirely on verifying neural networks against simple input-output specifications (katz2017reluplex; ehlers2017formal; bunel2018unified; ruan2018reachability; dutta2018output; pulina2010abstraction)

. Such input-output techniques compute a guaranteed range for the output of a deep neural network given a set of inputs represented as a convex polyhedron. To that end, several algorithms that takes advantage of the piecewise linear nature of the Rectified Linear Unit (ReLU) activation functions (one of the most famous nonlinear activation functions in deep neural networks) have been proposed. For example, by using binary variables to encode piecewise linear functions, the constraints of ReLU functions are encoded as a Mixed-Integer Linear Programming (MILP). Combining output specifications that are expressed in terms of Linear Programming (LP), the verification problem eventually turns to a MILP feasibility problem 

(dutta2018output; tjeng2017verifying).

Using off-the-shelf MILP solvers does not lead to a scalable approach to handle neural networks with hundreds and thousands of neurons (ehlers2017formal). To circumvent this problem, several MILP-like solvers targeted toward the neural network verification problem are proposed. For example, the work reported in (katz2017reluplex) proposed a modified Simplex algorithm (originally used to solve linear programs) to take into account ReLU nonlinearities as well. Similarly, the work reported in (ehlers2017formal) combines a Boolean satisfiability solving along with a linear over-approximation of piecewise linear functions to verify ReLU neural networks against convex specifications. Other techniques that exploit specific geometric structures of the specifications are also proposed (gehr2018ai; xiang2017reachable). A thorough survey on different algorithms for verification of neural networks against input-output range specifications can be found in (xiang2017survey) and the references within.

Unfortunately, the input-output range properties, studied so far in the literature, are simplistic and fails to capture the safety and reliability of cyber-physical systems when controlled by a neural network. Therefore, in this paper, we focus instead on the problem of formal verification of a neural network controlled robot against system-level safety specifications. In particular, we consider the problem in which a robot utilizes a LiDAR scanner to sense its environment. The LiDAR image is then processed by a neural network controller to compute the control inputs. Such scenario is common in the literature of behavioral cloning and imitation control in which the neural network is trained to imitate the actions generated by experts controlling the robot (kahn2017plato). With the objective to verify the safety of these robots, we develop a framework that can take into account the robot continuous dynamics, the workspace configuration, the LiDAR imaging, and the neural network, and compute the set of initial states for the robot that is guaranteed to produce robot trajectories that are safe and collision free.

To carry out the prescribed formal verification problem, we need a mathematical model that captures the LiDAR imaging process. This is the process that generates the LiDAR images based on the robot pose relative to the workspace objects. Therefore, the first contribution of this paper is to introduce the notion of imaging-adapted sets. These are workspace partitions within which the map between robot pose and LiDAR images are mathematically captured by an affine map. Given this notion, and thanks to the advances in the literature of computational graphics, we develop a polynomial-time algorithm that can partition the workspace into imaging-adapted sets along with the corresponding affine maps.

Given the partitioned workspace along with a pre-trained neural network and the robot dynamics, we compute a finite state abstraction of the closed loop system that enjoys a simulation relation with the original system. The main challenge in computing this finite state abstraction is to analyze the behavior of the neural network controller. Similar to previous works in the literature, we strict our focus to neural networks with Rectified Linear Unit (ReLU) nonlinearities and we develop a Satisfiability Modulo Convex (SMC) programming algorithm that uses a combination of a Boolean satisfiability solver and a convex programming solver to iteratively reason about the neural network nonlinearity along with the dynamics and the imaging constraints. At each iteration, the boolean satisfiability solver searches for a candidate assignment determining whether ReLU units are active while ignoring the neural network weights, the robot dynamics, and the LiDAR imaging. The convex programming solver is then used to check the feasibility of the proposed ReLU assignment against the neural network weights, the robot dynamics, and the LiDAR imaging. If the ReLU assignment is deemed infeasible, then the SMC solver will generate succinct explanations for their infeasibility to reduce the search space. To accelerate the process, we develop a pre-processing algorithm that can reduce the space of ReLU assignments.

Once the finite state abstraction is computed, we use standard reachability analysis techniques to compute the set of safe initial states. To summarize, the contributions of this paper can be summarized as follows:
1- A framework for formally proving safety properties of autonomous robots controlled by neural network controllers that process LiDAR images to compute control inputs.
2- A notion of imaging-adapted sets along with a polynomial-time algorithm for partitioning the workspace into such sets while computing an affine model capturing the LiDAR imaging process.
3- An SMC-based algorithm combined with an SMC-based pre-processing for computing finite abstractions of the neural network controlled autonomous robot.

2. Problem Formulation

Figure 1. Pictorial representation of the problem setup under consideration.

2.1. Notation

The symbols , and denote the set of natural, real, positive real, and Boolean numbers, respectively. The symbols and

denote the logical AND, logical NOT, and logical IMPLIES operators, respectively. Given two real-valued vectors

and , we denote by the column vector . Similarly, for a vector , we denote by the th element of . For two vectors , we denote by the element-wise maximum. For a set , we denote the boundary and the interior of this set by and , respectively. Given two sets and , and denote a set-valued and ordinary map, respectively. Finally, given a vector , we denote by .

2.2. Dynamics and Workspace

We consider an autonomous robot moving in a 2-dimensional polytopic (compact and convex) workspace . We assume that the robot must avoid the workspace boundaries along with a set of obstacles , with which is assumed to be polytopic. We denote by the set of the obstacles and the workspace boundaries which needs to be avoided, i.e., . The dynamics of the robot is described by a discrete-time linear system of the form:


where is the state of robot at time and is the robot input. The matrices and represent the robot dynamics and have appropriate dimensions. For a robot with nonlinear dynamics that is either differentially flat or feedback linearizable, the state space model (1) corresponds to its feedback linearized dynamics. We denote by the natural projection of onto the workspace , i.e., is the position of the robot at time .

2.3. LiDAR Imaging

We consider the case when the autonomous robot uses a LiDAR scanner to sense its environment. The LiDAR scanner emits a set of lasers evenly distributed in a degree fan. We denote by the heading angle of the LiDAR at time . Similarly, we denote by , with , the angle of the th laser beam at time where and by the vector of the angles of all the laser beams. While the heading angle of the LiDAR, , changes as the robot poses changes over time, i.e.:

for some nonlinear function , in this paper we focus on the case when the heading angle of the LiDAR, , is fixed over time and we will drop the superscript from the notation. Such condition is satisfied in many real-world scenarios whenever the robot is moving while maintaining a fixed pose (e.g. a Quadrotor whose yaw angle is maintained constant).

For the th laser beam, the observation signal is the distance measured between the robot position and the nearest obstacle in the direction, i.e.:


The final LiDAR image is generated by processing the observations as follows:


2.4. Neural Network Controller

We consider a pre-trained neural network controller that processes the LiDAR images to produce control actions with internal and fully connected layers in addition to one output layer. Each layer contains a set of neurons (where ) with Rectified Linear Unit (ReLU) activation functions. ReLU activation functions play an important role in the current advances in deep neural networks (krizhevsky2012imagenet). For such neural network architecture, the neural network controller can be written as:


where and

are the pre-trained weights and bias vectors of the neural network which are determined during the training phase.

2.5. Robot Trajectories and Safety Specifications

The trajectories of the robot whose dynamics are described by (1) when controlled by the neural network controller (2)-(4) starting from the initial condition is denoted by such that . A trajectory is said to be safe whenever the robot position does not collide with any of the obstacles at all times.

Definition 2.1 (Safe Trajectory).

A robot trajectory is called safe if:

Using the previous definition, we now define the problem of verifying the system-level safety of the neural network controlled system as follows:

Problem 2.2 ().

Consider the autonomous robot whose dynamics are governed by (1) which is controlled by the neural network controller described by (4) which processes LiDAR images described by (2)-(3). Compute the set of safe initial conditions such that any trajectory starting from is safe.

Figure 2. Pictorial representation of the proposed framework.

3. Framework

Before we describe the proposed framework, we need to briefly recall the following set of definitions capturing the notion of a system and relations between different systems.

Definition 3.1 ().

An autonomous system is a pair consisting of a set of states and a set-valued map representing the transition function. A system is finite if is finite. A system is deterministic if is single-valued map and is non-deterministic if not deterministic.

Definition 3.2 ().

Consider a deterministic system and a non-deterministic . A relation is a simulation relation from to , and we write , if the following conditions are satisfied:

  1. for every there exists with ,

  2. for every we have that in implies the existence of in satisfying .

Using the previous two definitions, we describe our approach as follows. As pictorially shown in Figure 2, given the autonomous robot system , where , our objective is to compute a finite state abstraction (possibly non-deterministic) of such that there exists a simulation relation from to , i.e., . This finite state abstraction will be then used to check the safety specification.

The first difficulty in computing the finite state abstraction is the nonlinearity in the relation between the robot position and the LiDAR observations as captured by equation (2). However, we notice that we can partition the workspace based on the laser angles along with the vertices of the polytopic obstacles such that the map (which maps the robot position to the processed observations) is an affine map. Therefore, as summarized in Algorithm 1, the first step is to compute such partitioning of the workspace (WKSP-PARTITION, line 2 in Algorithm 1). While WKSP-PARTITION focuses on partitioning the workspace , one need to partition the remainder of the state space (STATE-SPACE-PARTITION, line 5 in Algorithm 1) to compute the finite set of abstract states along with the simulation relation that maps between states in and the corresponding abstract states in , and vice versa.

Unfortunately, the number of partitions grows exponentially in the number of lasers and the number of vertices of the polytopic obstacles. To harness this exponential growth, we compute an aggregate-partitioning using only few laser angles (called primary lasers and denoted by ). The resulting aggregate-partitioning would contain a smaller number of partitions such that each partition in represents multiple partitions in . Similarly, we can compute a corresponding aggregate set of states as:

where each aggregate state is a set representing multiple states in . Whenever possible, we will carry out our analysis using the aggregated-partitioning (and ) and use the fine-partitioning only if deemed necessary. Details of the workspace partitioning and computing the corresponding affine maps representing the LiDAR imaging function are given in Section 4.

1:  Step 1: Partition the workspace
2:   = WKSP-PARTITION()   
3:  Step 2: Compute the finite state abstraction
4:  Step 2.1: Compute the states of
6:  for each and in  do
8:  Step 2.2: Pre-process the neural network
9:  for each and in  do
10:      =
11:      = PRE-PROCESS()
12:  Step 2.3: Compute the transition map
13:  for each in and in where  do
14:      =
15:      =
16:     Status = CHECK-FEASIBILITY()
17:     if Status == INFEASIBLE then
18:         for each in  do
20:     else
21:         for each in  do
22:             =
23:            Status = CHECK-FEASIBILITY()
24:            if Status == INFEASIBLE then
26:  Step 3: Compute the safe set
27:  Step 3.1: Mark the abstract states corresponding to obstacles and workspace boundary as unsafe
28:  Step 3.2: Iteratively compute the predecessors of the abstract unsafe states
30:  while Status == FIXED-POINT-NOT-REACHED do
32:     if  then
33:         Status = FIXED-POINT-REACHED
35:  Step 3.3: Compute the set of safe states
37:  Return
Algorithm 1 Verify-NN()

The state transition map is computed as follows. First, we assume a transition exists between any two states and in (line 67 in Algorithm 1). Next, we start eliminating unnecessary transitions. We observe that regions in the workspace that are adjacent or within some vicinity are more likely to force the need of transitions between their corresponding abstract states. Similarly, regions in the workspace that are far from each other are more likely to prohibit transitions between their corresponding abstract states. Therefore, in an attempt to reduce the number of computational steps in our algorithm, we check the transition feasibility between a state and an aggregate state . If our algorithm (CHECK-FEASIBILITY) asserted that the neural network prohibits the robot from transitioning between the regions corresponding to and (denoted by and , respectively), then we conclude that no transition in is feasible between the abstract state and all the abstract states in (lines 13-19 in Algorithm 1). This leads to a reduction in the number of state pairs that need to be checked for transition feasibility. Conversely, if our algorithm (CHECK-FEASIBILITY) asserted that the neural network allows for a transition between the regions corresponding to and , then we proceed by checking the transition feasibility between the state and all the states contained in the aggregate state (lines 21-25 in Algorithm 1).

Checking the transition feasibility (CHECK-FEASIBILITY) between two abstract states entail reasoning about the robot dynamics, the neural network, along with the affine map representing the LiDAR imaging computed from the previous workspace partitioning. While the robot dynamics is assumed linear, the imaging function is affine; the technical difficulty lies in reasoning about the behavior of the neural network controller. Thanks to the ReLU activation functions in the neural network, we can encode the problem of checking the transition feasibility between two regions as formula , called monotone Satisfiability Modulo Convex (SMC) formula (shoukry2017smc; shoukry2018smc), over Boolean and convex constraints representing, respectively, the ReLU phases and the dynamics, the neural network weights, and the imaging constraints. In addition to using SMC solver to check the transition feasibility (CHECK-FEASIBILITY) between abstract states, it will be used also to perform some pre-processing of the neural network function (lines 9-11 in Algorithm 1) which is going to speed up the process of checking the the transition feasibility. Details of the SMC encoding are given in Section 5.

Once the finite state abstraction and the simulation relation is computed, the next step is to partition the finite states into a set of unsafe states and a set of safe states using the following fixed-point computation:

where the represents the abstract state corresponding to the obstacles and workspace boundaries while with represents all the states that can reach in -steps where:

The remaining abstract states are then marked as the set of safe states . Finally, we can compute the set of safe states as:

These computations are summarized in lines 27-36 in Algorithm 1.

4. Imaging-Adapted Workspace Partitioning

We start by introducing the notation of the important geometric objects. We denote by the ray originated from a point in the direction , i.e.:

Similarly, we denote by the line segment between the points and , i.e.:

For a convex polytope , we denote by , its set of vertices and by its set of line segments representing the edges of the polyhedron.

4.1. Imaging-Adapted Partitions

The basic idea behind our algorithm is to partition the workspace into a set of polytypic sets (or regions) such that for each region the LiDAR rays will intersect the same obstacle/workspace edge regardless of the robot positions . To formally characterize this property, let be the set of all points in the workspace in which an obstacle or workspace boundary exists. Consider a workspace partition and a robot position that lies inside this partition, i.e., . The intersection between the th LiDAR laser beam and is a unique point characterized as:


By sweeping across the whole region , we can characterize the set of all possible intersection points as:


Using the set described above, we define the notion of imaging-adapted set as follows.

Definition 4.1 ().

A set is said to be an imaging-adapted partition if the following property holds:


Figure 3 shows concrete examples of imaging-adapted partitions. Imaging-adapted partitions enjoys the following property:

Lemma 4.2 ().

Consider an imaging-adapted partition with corresponding sets . The LiDAR imaging function is an affine function of the form:


for some constant matrices and vectors that depend on the region and the LiDAR angle .

Figure 3. (left) A partitioning of the workspace that is not imaging-adapted. Within region , the LiDAR ray (cyan arrow) intersects different obstacles edges depending on the robot position. (right) A partitioning of the workspace that is imaging-adapted. For both regions and , the LiDAR ray (cyan arrow) intersects the same obstacle edge regardless of the robot position.

4.2. Partitioning the Workspace

Motivated by Lemma 4.2, our objective is to design an algorithm that can partition the workspace into a set of imaging-adapted partitios. As summarized in Algorithm 2, our algorithm starts by computing a set of line segments that will be used to partition the workspace (lines 1-5 in Algorithm 2). This set of line segments are computed as follows. First, we define the set as the one that contains all the vertices of the workspace and the obstacles, i.e., . Next, we consider rays originating from all the vertices in and pointing in the opposite direction of the angles . By intersecting these rays with the obstacles and picking the closest intersection points, we acquire the line segments that will be used to partition the workspace. In other words, is computed as:


Thanks to the fact that the vertices are fixed, finding the intersection between and is a standard ray-polytope intersection problem which can be solved efficiently (berg2008computational).

The next step is to compute the intersection points between the line segments and the edges of the obstacles . A naive approach will be to consider all combinations of line segments in and test them for intersection. Such approach is combinatorial and would lead to an execution time that is exponential in the number of laser angles and vertices of obstacles. Thanks to the advancements in the literature of computational geometry, such intersection points can be computed efficiently using the Plane-Sweep algorithm (berg2008computational). The plane-sweep algorithm simulates the process of sweeping a line downwards over the plane. The order of the line segments from left to right as they intersect the sweep line is stored in a data structure called the sweep-line status. Only segments that are adjacent in the horizontal ordering need to be tested for intersection. Though the sweeping process can be visualized as continuous, the plane-sweep algorithm sweeps only the values in which the endpoints of segments in , which are given beforehand, and the intersection points, which are computed on the fly. To keep track of the endpoints of segments in and the intersection points, we use a balanced binary search tree as data structure to support insertion, deletion, and searching in time, where is number of elements in the data structure.

The final step is to use the line segments and their intersection points , discovered by the plane-sweep algorithm, to compute the workspace partitions. To that end, consider the undirected planar graph whose vertices are the intersection points and whose edges are , denoted by . The workspace partitions are equivalent to finding subgraphs of such that each subgraph contains only one simple cycle 111A cycle in an undirected graph is called simple when no repetitions of vertices and edges is allowed within the cycle.. To find these simple cycles, we use a modified Depth-First-Search algorithm in which it starts from a vertex in the planar graph and then traverses the graph by considering the rightmost turns along the vertices of the graph. Finally, the workspace partition is computed as the convex hull of all the vertices in the computed simple cycles. It follows directly from the fact that each region is constructed from the vertices of a simple cycle that there exists no line segment in that intersects with the interior of any region, i.e., for any workspace partition , the following holds:


This process is summarized in lines 8-16 in Algorithm 2. An important property of the regions determined by Algorithm 2 is stated by the following proposition.

Proposition 4.3 ().

Consider a partition computed by Algorithm 2 and satisfies (10). The following property holds for any LiDAR ray with angle :

where defined in (6). In other words, the LiDAR ray with angle intersects the same obstacle edge regardless of the robot position.

We conclude this section by stating our first main result, quantifying the correctness and complexity of Algorithm 2.

Theorem 4.4 ().

Given a workspace with polytopic obstacles and a set of laser angles , then Algorithm 2 computes the partitioning such that:

  1. ,

  2. ,

  3. .

Moreover, the time complexity of Algorithm 2 is , where is cardinality of , and is number of intersection points between segments in .

1:  Step 1: Generate partition segments
3:  for  do
4:     Use a ray-polygon intersection algorithm to compute:
6:  Step 2: Compute intersection points
8:  Step 3: Construct the partitions
9:  Cycles = Find-Vertices-Of-Simple-Cycle()
10:  Cycles’ = Find-Vertices-Of-Simple-Cycle ().
11:  for  do
14:  for  do
17:  Return
Algorithm 2 WKSP-PARTITION ( )

5. Computing the Finite State Abstraction

Once the workspace is partitioned into imaging-adapted regions and the corresponding imaging function is identified, the next step is to compute the finite state transition abstraction of the closed loop system along with the simulation relation . The first step is to define the state space and its relation to . To that end, we start by computing a partitioning of the state space that respects . For the sake of simplicity, we consider that is -orthotope, i.e., there exists constants such that:

Now, given a discretization parameter , we define the state space as:


where is the number of regions in the partitioning . In other words, the parameter is used to partition the state space into hyper-cubes. A state represents the index of a region in followed by the indices identifying a hypercube in the remaining dimensions. Note that for the simplicity of notation, we assume that is divisible by for all . We now define the relation as:


Finally, we define the state transition function of as follows:


It follows from the definition of in (13) that checking the transition feasibility between two states and is equivalent to searching for a robot initial and goal states along with a LiDAR image that will force the neural network controller to generate an input that moves the robot between the two states while respecting the robots dynamics. In the reminder of this section, we focus on solving this feasibility problem.

5.1. SMC Encoding of NN

We translate the problem of checking the transition feasibility in into a feasibility problem over a monotone SMC formula (shoukry2017smc; shoukry2018smc) as follows. We introduce the Boolean indicator variables with and (recall that represents the number of layers in the neural network, while represents the number of neurons in the th layer). These Boolean variables represents the phase of each ReLU, i.e., an asserted indicates that the output of the th ReLU in the th layer is while a negated indicates that . Using these Boolean indicator variables, we encode the problem of checking the transition feasibility between two states and as:

subject to:

where (15)-(16) encodes the state space partitions corresponding to the states and ; (17) encodes the dynamics of the robot; (18) encodes the imaging function that maps the robot position into LiDAR image; (19)-(22) encodes the neural network controller that maps the LiDAR image into a control input.

Compared to Mixed-Integer Linear Programs (MILP), monotone SMC formulas avoid using encoding heuristics like big-M encodings which leads to numerical instabilities. SMC decision procedures follow an iterative approach combining efficient Boolean Satisfiability (SAT) solving with numerical convex programming. When applied to the encoding above, at each iteration the SAT solver generates a candidate assignment for the ReLU indicator variables

. The correctness of these assignments are then checked by solving the corresponding set of convex constraints. If the convex program turned to be infeasible, indicating a wrong choice of the ReLU indicator variables, the SMC solver will identify the set of “Irreducible Infeasible Set” (IIS) in the convex program to provide the most succinct explanation of the conflict. This IIS will be then fed back to the SAT solver to prune its search space and provide the next assignment for the ReLU indicator variables. SMC solvers was shown to better handle problems (compared with MILP solvers) for problems with relatively large number of Boolean variables (shoukry2018smc).

5.2. Pruning Search Space By Pre-processing

While a neural network with ReLUs would give rise to combinations of possible assignments to the corresponding Boolean indicator variables, we observe that several of those combinations are infeasible for each workspace region. In other words, the LiDAR imaging function along with the workspace region enforces some constraints on the inputs to the neural network which in turn enforces constraints on the subsequent layers. By performing pre-processing on each of the workspace regions, we can discover those constraints and augment it to the SMC encoding (15)-(22) to prune several combinations of assignments of the ReLU indicator variables.

To find such constraints, we consider an SMC problem with the fewer constraints (15), (18)-(22). By iteratively solving the reduced SMC problem and recording all the IIS conflicts produced by the SMC solver, we can compute a set of counter-examples that are unique for each region. By iteratively invoking the SMC solver while adding previous counter-examples as constraints until the problem is no longer satisfiable, we compute the set which represents all the counter-examples for region . Finally, we add the following constraint:


to the original SMC encoding (15)-(22) to prune the set of possible assignments to the ReLU indicator variables. In Section 6, we show that pre-processing would result in an order of magnitude reduction in the execution time.

5.3. Correctness of Algorithm 1

We end our discussion with the following results which asserts the correctness of the whole framework described in this paper. We first start by establishing the correctness of computing the finite abstraction along with the simulation relation as follows:

Proposition 5.1 ().

Consider the finite state system where is defined by (11) and is defined by (13) and computed by means of solving the SMC formula (15)-(23). Consider also the system where . For the relation defined in (12), the following holds: .

Recall that Algorithm 1 applies standard reachability analysis on to compute the set of unsafe states. It follows directly from the correctness of the simulation relation established above that our algorithm computes an over-approximation of the set of unsafe states, and accordingly an under-approximation of the set of safe states. This fact is captured by the following result that summarizes the correctness of the proposed framework:

Theorem 5.2 ().

Consider the safe set computed by Algorithm 1. Then any trajectory with is a safe trajectory.

While Theorem 5.2 establishes the correctness of the proposed framework in Algorithm 1, two points needs to be investigated namely (i) complexity of Algorithm 1 and (ii) maximality of the set . Although Algorithm 2 computes the imaging-adapted partitions efficiently (as shown in Theorem 4.4), analyzing a neural network with ReLU activation functions is shown to be NP-hard. Exacerbating the problem, Algorithm 1 entails analyzing the neural network a number of times that is exponential in the number of partition regions. In Section 6, we experiment the efficiency of using SMC decision procedures to harness this computational complexity. As for the maximality of the computed set, we note that Algorithm 1 is not guaranteed to search for the maximal .

6. Results

We implemented the proposed verification framework as described by Algorithm 1 on top of the SMC solver named SATEX (satex). All experiments were executed on an Intel Core i7 2.5-GHz processor with 16 GB of memory.

6.1. Scalability of the Workspace Partitioning Algorithm:

As the first step of our verification framework, imaging-adapted workspace partitioning is tested for numerical stability with increasing number of laser angles and obstacles. Table 1 summarizes the scalability results in terms of the number of computed regions and the execution time grows as the number of LiDAR lasers and obstacle vertices increase. Thanks to adopting well-studied computational geometry algorithms, our partitioning process takes less than minutes for the scenario where a LiDAR scanner is equipped with 298 lasers (real-world LiDAR scanners are capable of providing readings from 270 laser angles).

Number of Number of Number of Time
Vertices Lasers regions [s]
8 111 0.0152
8 38 1851 0.3479
118 17237 5.5300
8 136 0.0245
10 38 2254 0.4710
118 20343 6.9380
8 137 0.0275
38 2418 0.5362
12 120 23347 8.0836
218 76337 37.0572
298 142487 86.6341
Table 1. Scalability results for the WKSP-PARTITION Algorithm

6.2. Computational Reduction Due to Pre-processing

The second step is to pre-process the neural network. In particular, we would like to answer the following question: given a partitioned workspace, how many ReLU assignments are feasible in each region, and if any, what is the execution time to find them out. Recall that a ReLU assignment is feasible if there exist a robot position and the corresponding LiDAR image that will lead to that particular ReLU assignment.

Thanks to the IIS counterexample strategy, we can find all feasible ReLU assignments in pre-processing. Our first observation is that the number of feasible assignments is indeed much smaller compared to the set of all possible assignments. As shown in Table 2, for a neural network with a total of 32 neurons, only 11 ReLU assignments are feasible (within the region under consideration). Comparing this number to possibilities of ReLU assignments, we conclude that pre-processing is very effective in reducing the search space by several orders of magnitude.

Furthermore, we conducted an experiment to study the scalability of the proposed pre-processing for an increasing number of ReLUs. To that end, we fixed one choice of workspace regions while changing the neural network architecture. The execution time, the number of generated counterexamples, along with the number of feasible ReLU assignments are given in Table 2. For the case of neural networks with one hidden layer, our implementation of the counterexample strategy is able to find feasible ReLU assignments for a couple of hundreds of neurons in less than 4 minutes. In general, the number of counterexamples, and hence feasible ReLU assignments, and execution time grows with the number of neurons. However, the number of neurons is not the only deciding factor. Our experiments show that the depth of the network plays a significant role in affecting the scalability of the proposed algorithms. For example, comparing the neural network with one hidden layer and a hundred neurons per layer versus the network with two layers and fifty neurons per layer we notice that both networks share the same number of neurons. Nevertheless, the deeper network resulted in one order of magnitude increase regarding the number of generated counterexamples and one order of magnitude increase in the corresponding execution time. Interestingly, both of the architectures share a similar number of feasible ReLU assignments. In other words, similar features of the neural network can be captured by fewer counterexamples whenever the neural network has fewer layers. This observation can be accounted for the fact that counterexamples that correspond to ReLUs in early layers are more powerful than those involves ReLUs in the later layers of the network.

In the second part of this experiment, we study the dependence of the number of feasible ReLU assignments on the choice of the workspace region. To that end, we fix the architecture of the neural network to one with hidden layers and neurons per layer. Table 3 reports the execution time, the number of counterexamples, and the number of feasible ReLU assignments across different regions of the workspace. In general, we observe that the number of feasible ReLU assignments increases with the size of the region.

Number Total Number of Number of Time
of Hidden Number feasibile Counter [s]
Layers of Neurons ReLU Assignments Examples
32 11 60 2.7819
72 31 183 11.4227
92 58 265 18.4807
102 68 364 43.2459
152 101 540 78.3015
172 146 778 104.4720
202 191 897 227.2357
1 302 383 1761 656.3668
402 730 2614 1276.4405
452 816 4325 1856.0418
502 1013 3766 2052.0574
552 1165 4273 4567.1767
602 1273 5742 6314.4890
652 1402 5707 7166.3059
702 1722 6521 8813.1829
22 3 94 1.3180
42 19 481 10.9823
62 35 1692 53.2246
82 33 2685 108.2584
2 102 58 5629 292.7412
122 71 9995 739.4883
142 72 18209 2098.0220
162 98 34431 6622.1830
182 152 44773 12532.8552
32 5 319 5.7227
3 47 7 5506 148.8727
62 45 72051 12619.5353
4 22 9 205 10.4667
42 5 1328 90.1148
Table 2. Execution time of the SMC-based pre-processing as a function of the neural network architecture.
Region Number of Number of Time
Index feasibile Counter [s]
ReLU Assignments Examples
A2-R3 33 2685 108.2584
A14-R1 55 4925 215.8251
A13-R3 7 1686 69.4158
A1-R1 25 2355 99.2122
A7-R1 26 3495 139.3486
A12-R2 3 1348 54.4548
A15-R3 25 3095 121.7869
A19-R1 38 4340 186.6428
Table 3. Execution time of the SMC-based pre-processing as a function of the workspace region.

6.3. Transition Feasibility

Following our verification streamline, the next step is to compute the transition function of the finite-state abstraction , i.e., check transition feasibility between regions. Table 4 shows performance comparison between our proposed strategy that uses counterexamples obtained from pre-processing and SMC encoding without preprocessing. We observe that SMC encodings empowered by counterexamples, generated through the pre-processing phase, scales more favorably compared to the ones that do not take counterexamples into account leading to 2-3 orders of magnitude reduction in the execution time. Moreover, and thanks to the pre-processing counter-examples, we observe that checking transition feasibility becomes less sensitive to changes in the neural network architecture as shown in Table 4.

Number of Total Number Time [s] Time [s]
Hidden Layers of Neurons (Exploit Counter (Without Counter
Examples) Examples)

82 0.5056 50.1263
102 7.1525 timeout
1 112 12.524 timeout
122 18.0689 timeout
132 20.4095 timeout
22 0.1056 15.8841
42 4.8518 timeout
62 3.1510 timeout
82 2.6112 timeout
2 102 11.0984 timeout
122 3.8860 timeout
142 0.7608 timeout
162 2.7917 timeout
182 193.6693 timeout
32 0.3884 388.549
3 47 0.9034 timeout
62 59.393 timeout
Table 4. Performance of the SMC-based encoding for computing as a function of the neural network (timeout = 1 hour/)

7. Conclusions

We presented a framework to verify the safety of autonomous robots equipped with LiDAR scanners and controlled by neural network controllers. Thanks to the notion of imaging-adapted sets, we can partition the workspace to render the problem amenable to formal verification. Using SMC-encodings, we presented a framework to compute finite-state abstraction of the system that can be used to compute an under-approximation of the set of safe robot states. We demonstrated a pre-processing technique that generates a set of counterexamples which resulted in 2-3 orders of magnitude execution time reduction. Future work includes investigating further strategies for efficient generation of pre-processing counterexamples along with extending the proposed technique to account to uncertainties in the LiDAR scanner.


Appendix A Proof of Lemma 4.2


Consider an arbitrary LiDAR laser with an angle and arbitrary robot position . The LiDAR image can be written as:


where is defined in (5). It follows from the fact that is an imaging-adapted partition that the set is a line segment. Let be the vertices of this line segment, i.e., and recall that satisfies and hence lies on the line segment . Therefore there exists a such that: