The 2007 DARPA Urban Challenge ushered interest in autonomous driving in urban environments [DARPA_Urban]. Participating vehicles had to pass three small-scale operational test-courses designed to evaluate the autonomous car’s ability to satisfy safety, basic and advanced navigation requirements, and basic and advanced traffic scenarios [DARPA_Urban_rules]. Exhaustive verification for such complex safety-critical systems is prohibitive, creating a need for a formal operational testing framework to certify reliability of these systems [koopman2016challenges].
Due to robustness metrics from their quantitative semantics, signal temporal logic (STL) and metric temporal logic (MTL), are natural paradigms for reasoning over trajectories of low-level continuous dynamics [fainekos2009robustness, donze2010robust]. In many instances, the term testing is used inter-changeably with falsification [sankaranarayanan2012falsification]. Falsification is the problem of finding initial conditions and input signals that lead to violation of a temporal logic formula with the goal of finding such failures quickly and for black-box models [annpureddy2011s, plaku2013falsification, donze2010breach, chou2018using]. Furthermore, the black-box approaches in the related topics of falsification of hybrid systems [annpureddy2011s], and simulation-based test generation [tuncali2018simulation, fainekos2012verification], rely on stochastic optimization algorithms to minimize the robustness of temporal logic satisfaction. Since dense-time temporal logics better encapsulate the range of system behaviors at the with continuous dynamics, these techniques are successful at falsification at the low-level. However, some of the complexity can be attributed to the coupling between continuous dynamics with high-level discrete decision-making behaviors, a hierarchical approach to test-case generation could be effective.
At higher levels of abstraction comprising of discrete states, LTL specifications are often used to capture mission objectives. Covering arrays have been used to initialize discrete parameters of the test configuration at the start of the falsification procedure in [annpureddy2011s, tuncali2018simulation, majumdar2019paracosm]. In this work, we focus on a framework for testing of high-level specifications in linear temporal logic (LTL).
Here we adopt a different notion of testing – one that is focused on observing the autonomous agent undertake a certain behavior in its mission. The DARPA Urban Challenge test courses, that mainly comprised of static obstacles and (dynamic) human-driven cars, were carefully designed to observe the agent undertaking certain behaviors [DARPA_Urban_rules]. For example, a part of the test course was designed for assessing parking behavior. The static obstacles – barriers blocking the region in front of the parking lot and other parked cars – were placed such that the agent had to repeatedly reverse/pull-in to incrementally adjust its heading angle before successfully parking in the designated spot. The clever placement of static obstacles in this scenario made it a challenging test for the agent, as opposed to an environment in which the agent pulls-in straight into the parking spot. Similarly, carefully designed scenarios with human-driven cars sought to observe other behaviors of the agent. In many, but not necessarily all, of these scenarios, the high-level behavior of the agent can be described as a sequence of waypoints. In the parking lot example, the sequence of waypoints can be characterized as a sequence of agent states, which can be characterized as a product of position and heading angle in the high-level abstraction. As a step towards automatically synthesizing these test scenarios, this paper asks the following question:
Problem (Informal): Given a valid, user-defined sequence of waypoints, a reachability objective for the mission specification, find a set of possible initial conditions for the agent (if not specified by user) and determine a set of static constraints, characterized by transitions that are blocked/restricted, such that:
the agent must visit the sequence of waypoints in order before its goal, and
the test environment is minimally constrained.
Most formal analyses of discrete state systems and temporal logic specifications are based on the model-checking paradigm of constructing a product automaton of the system abstraction and the Büchi automaton of the temporal logic formula [baier2008principles]. While this is a powerful approach that encapsulates any temporal logic formula, the Büchi automaton has potentially exponential number of states in the length of the formula. However, when restricted to a specific class of temporal logic formulas, it is possible to develop an alternative approach to analyzing the properties of a discrete-state system. In the paper, we use the concept of network flows [cormen2009introduction] and integer linear programming (ILP) to synthesize “cuts” or static constraints to characterize the test environment. In particular, the waypoints and goal state constitute the sources and sinks of these flows, and we seek to enforce static constraints that maximize the flow between consecutive waypoints such that the overall flow from the initial waypoint to the final goal is maximized.
The contributions of this paper are as follows:
Formulating the static test synthesis problem, formalizing the notions of a test specification and a test graph,
An algorithm and an ILP subroutine that iteratively synthesizes static constraints using fewer variables in each iteration,
Proof of completeness and recursive feasibility (feasibility of the ILP constraint set is invariant despite static constraints determined in prior iterations) of the aforementioned iterative algorithm,
Formalizing the notion of a minimally constrained test graph, and a proof showing that the synthesized test environment is minimally constrained.
The rest of this paper is organized as follows: In Section 2, we define the test specification and the test graph, and formally present the static test environment synthesis problem. The baseline ILP and the iterative algorithm to solve the static test synthesis problem are detailed in Section 3, and subsequently, we prove recursive feasibility, completeness, and show that the synthesized static environment is minimally constrained. In Section 4, we illustrate the algorithms on parametrized gridworlds. Finally, we present future directions in Section 5.
2 Preliminaries and Problem Statement
In this section, we first introduce the notion of discrete finite state systems. Afterwards, we formally describe the specification that such system should satisfy. Finally, we present basic definitions from flow networks, which we will leverage later on for test synthesis.
Definition 1 (Kripke Structure [baier2008principles]).
A Kripke structure is a tuple ), where is a finite set of states, is a finite set of actions, is a non-deterministic transition function, is the set of initial states, and is the set of atomic propositions, and is a labeling function. A directed graph can be induced from in which the vertices represent states and the edges represent the transitions, and the labeling function assigns propositions that are true at each vertex. For a proposition , and vertex , means that is satisfied at . A run on the graph is an infinite sequence of its nodes where represents the state at index .
Linear Temporal Logic (LTL) is a formal specification language, originally developed for software systems, that has also been used for characterizing high-level specifications of cyber-physical systems [emerson1990temporal, manna2012temporal, alur2015principles, kress2009temporal]. For the syntax and semantics of LTL, refer to Appendix or [baier2008principles]. Finally, we present a few definitions on network flows that will become relevant for the purpose of synthesizing the test environment.
Definition 2 (Flows [cormen2009introduction]).
A flow network is a directed graph with a non-negative capacity function such that each edge is assigned a non-negative capacity . The flow network also comprises of a source and the sink . A flow is a function that satisfies the capacity constraint and flow conservation properties (see [cormen2009introduction]). The value of the flow is given as follows,
See Appendix or [cormen2009introduction] for a definition of residual flow networks.
Definition 3 (Augmenting Paths [cormen2009introduction]).
An augmenting path is a simple path, that is, each edge is traversed once, from source to sink on the residual network . Maximizing the flow value leaves the residual network with no augmenting paths. We say that a set of augmenting paths represents an instance of the maximum flow from to in if the flow network is constructed by identifying the augmenting paths in . The set is said to be the set of shortest augmenting paths if the residual network is recursively constructed by identifying the smallest length augmenting path from to .
In our formulation, an edge represents a transition the system can make. For this reason, we assign unit capacities for every edge in the flow network. In unit-capacity networks, given a set of augmenting paths characterizing maximum flow, any two augmenting paths are always edge-disjoint with respect to each other.
Definition 4 (Minimum-cut edge [cormen2009introduction]).
A minimum-cut edge, given a source and sink , is an edge that belongs to some minimum cut. A minimum-cut edge can be identified by removing the edge and checking if the max-flow value decreases. For definition of minimum-cut, refer to Appendix or [cormen2009introduction].
Definition 5 (-cycle).
Given a graph and propositions characterizing the sequence specification, , let represent a simple path from to for all . The simple path from to characterized by () has an -cycle if there is an edge and for some , except for the case in which both and .
Definition 6 (Sequence flow path).
Given a graph and propositions characterizing the sequence specification, , let represent a simple path from to for all . The simple path from to characterized by () is a sequence flow path if the subgraph of induced by satisfies equation (4). The notation is used to denote the -th path in . A sequence flow path has no -cycles.
Definition 7 (Sequence flow).
A sequence flow for propositions characterizing the sequence specification is a finite set of sequence flow paths, , such that any two sequence flow paths are edge-disjoint with respect to each other, and any sequence flow path, , , , , where , have no -cycles.
2.1 Test Specification
In this paper, we consider reachability specifications as mission objectives for the agent under test. For the test itself, we wish to observe a sequence-like behavior of the agent in its attempt to satisfy its mission objectives. Formally, this test behavior can be described by the following temporal logic formula,
where are propositional formulas. This is a sequence-like formula since the agent has to eventually visit every , but it cannot visit before visiting , where , for all . The agent under test does not have access to the test specification . Since LTL formulae cannot be evaluated on finite test runs, the length of the test run depends on the time the agent takes to satisfy its mission objective. Example: Consider a gridworld in Figure 1 on which the agent can transition between states (up, down, left, right) with the mission specification of reaching some goal state (formalized as ). Of the many possible paths the agent can take to meet its objective, we’re interested in observing it navigate to the goal while restricted to a class of paths described by the test specification . How would we constrain actions of the agent in certain states, such that it navigates through the sequence of waypoints before reaching the goal? Furthermore, is it possible to synthesize these constraints such that the sequence flow value from to is maximized?
2.2 Problem Statement
Now, we formalize the test environment synthesis problem. We limit our focus to static test environments, by which we mean that the test environment does not react to the actions of the agent during the test, leaving the reactive test synthesis problem for future work.
Definition 8 (Test Graph).
Given a labeled directed graph , a mission/agent specification , a test specification (equation 1), a test graph is the directed graph obtained by removing set of edges, , from the original graph . On , a run starting from state will satisfy the specification,
Definition 9 (Minimally Constrained Test Graph).
A test graph is minimally constrained if the sequence flow value from to on are maximized.
Given an agent specification , a labeled directed graph induced by the non-deterministic transition model of the agent, a test specification , static constraints such that on the resulting graph ) is a minimally-constrained test graph.
Standard algorithms, such as Edmonds-Karp [cormen2009introduction], can find the minimum-cut of a single source-sink flow problem in a graph in time. However, here we aim to find a cut that maximizes the flow from a waypoint to its consecutive waypoint , while eliminating any flow to waypoints () for all . In other words, some flows need to be cut while other flows should be maximized. The problem of constructing a minimally constrained test graph for observing a sequence-like specification can be cast as the following optimization,
where the variables are , the set of edges to be constrained, scalar , scalars that represent the total flow from source to sink on a unit-capacity graph , and the problem data is the original graph and the nodes . Solving this optimization directly will require constructing an ILP, for which constructing the constraint set is not straightforward. Furthermore, it would require solving an ILP with number of integer variables. As a result, we propose Algorithm 1 to iteratively find the optimal .
3 Algorithm for Synthesizing Test Environment
Let be a directed graph, with unit capacity on every edge, induced by the Kripke structure of the system under test. Assuming that the test environment has complete freedom to “block” any transition in the graph , Algorithm 1 returns a set of edges, , of the graph that must be removed before the test run. Let denote the length of the shortest path from vertex to vertex on graph . First, we make following assumptions on ,
Likewise, for each , let denote the vertex s.t . Assume , for all .
Informally, assumption 1 states that every propositional formula, , has a single vertex in associated with it.
There exists a set of edges such that the modified graph obtained by removing these edges, , is such that
The above Assumption 2 is equivalent to the statement that by removing some edges (or restricting certain transitions) from the original graph , there exists some set of initial conditions for which the only path(s) to the goal is through the behavior . This assumption is imperative since there might be instances for which it is impossible to construct a test graph. For example, in the following simple labeled graph (Figure 2), it is impossible to construct a test graph for the test specification . Once the system is in state , it can directly proceed to the goal state without visiting . For instances such as this one, a reactive test environment is necessary.
3.1 Finding Combinations of Augmenting Paths
In a unit-capacity flow network with source and sink with maximum flow value , let represent the set of augmenting paths, or edge-disjoint paths, that characterize an instance of the maximum flow in the network since there can be multiple flow functions that can achieve maximum flow on the graph from source to sink . For maximum flow from source to sink on , let maximum flow value be denoted . On a minimally constrained test graph , the maximum sequence flow value will be bounded as follows,
Let denote the set of all sets of augmenting paths that characterize the maximum flow. Note that is finite since the number of edges are finite, but can be combinatorial in number of simple paths from to . Then, denotes the total number of maximum flow realizations from to . Consequently, the total number of augmenting path combinations from to will be . However, not every augmenting path combination might lead to a valid test graph since there could exist a combination of augmenting paths that violates the requirement in equation (2) by constructing an -cycle. Consider the simple example of the grid in Figure 3. The combination of sequence flows will give us , but the combination of forms an -cycle.
To avoid this issue, the algorithm searches through all combinations of sequence flows before constructing the input to the ILP (6). Since this is an expensive computation, a further assumption on the input graph and set of propositions can ease this bottleneck. This assumption is as follows,
Let , represent the set of sets of shortest augmenting paths that characterizes the flow from to on . Then, there exists a combination on which a maximum sequence flow can be characterized.
In other words, Assumption 3 allows us to reason over combinations of shortest augmenting path flows, which is combinatorial in all shortest paths, instead of combinations of all augmenting flows, which is combinatorial in all simple paths. All shortest paths a subset of all simple paths between two nodes.
3.2 Iterative Synthesis of Constraints
Now we present the ILP used to iteratively compute cuts in Algorithm 1,
where () are the optimization variables, and , , , , are problem data described in more detail in Appendix 6.3.
The subroutine Find-Cut-Paths, described in the algorithm takes as input a graph and list of propositions, , and uses max-flow algorithms to find a set of augmenting paths for every source-sink pair () on such that . All of these augmenting paths are collectively returned as the output , and the edges constituting these cuts are denoted by . Note that does not return all simple paths from to , but just a set of edge-disjoint paths. As a result, we need to iteratively restrict transitions until is empty. A summary of the Minimum-Cut-Edges and the Sequence-Flow subroutine is given in the Appendix 6.
Finally, the parameters to ILP (6) are constructed as follows:
1. Parameters: The parameters used to construct the problem data for the ILP (6) are the set of paths that need to be constrained, , the set of paths whose combination constitutes sequence flow and should not be constrained, , and the set of minimum-cut edges, , on the paths constituting . The set is a set of all augmenting path combinations. For a given combination of sets of augmenting paths, , with the cardinality of being denoted as follows, , and . Suppose a combination of augmenting paths, , represents a sequence flow, then a matrix can be constructed to represent the sequence flow . This construction is outlined in the descriptions of Constraints of the ILP. An instance of can have several sequence flows, , and correspondingly, several matrices, , all of which are collectively denoted by . The set of all such is denoted by , which has cardinality , since each corresponds to an instance of . The maximum sequence flow value is given by .
2. Variables: The variable , where
, is the Boolean vector corresponding to edgessuch that for some , if , then the corresponding edge is constrained, and means that it is left in the graph for future iterations. Given , a combination of set of augmenting paths, the variable keeps track of whether an augmenting path in some () is constrained or not. For some , if , then the corresponding augmenting path in some has minimum-cut edge(s) constrained by the ILP, and if none of the minimum-cut edges of that augmented path have been constrained. The variable, , represents the sequence flow vector for a given sequence flow, , such that is the number of edge-disjoint paths constituting the sequence flow.
3. Constraints: The constraints of ILP (6) are outlined here but futher details are given in Appendix 6.3. The first constraint of the ILP, , enforces the requirement that each path in is constrained. In the second and third constraints, and , is used to determine the variable from the variable . These two constraints ensure that for some , iff at least one minimum-cut edge on the path corresponding to the -th row of is constrained, and iff none of the minimum-cut edges on the path corresponding to the -th row of are constrained.
The fourth and fifth constraints, and , determine the flow value for a given set of sequence flow paths, . The fourth constraint ensures that if any of the constituent paths, , in the -th sequence flow path (for ), is constrained, then the flow value, . The last constraint ensures that if none of the constituent paths, , in the -th sequence flow path (for ), are constrained, then the flow value, .
4. Cost Function: The cost function computes the maximum sequence flow value.
Algorithm 1 does not proceed to the next iteration of until it finds the set of static constraints that return the maximum possible sequence flow value, . To guarantee completeness of Algorithm 1, we need to prove that the cuts synthesized in prior iterations do not preclude feasibility of further iterations with regards to assumption 2. See Appendix 6.5 for complexity of the subroutines in Algorithm 1.
In a graph , let represent a maximal set of sequence flow paths from to . Let be the set of paths that need to constrained, with the edges constituting the paths in denoted by . Then, the set of constraint edges can be found such that does not constrain any path in .
See Appendix 6.4 ∎
In the first iteration, from Assumption 2, we know there exists at least one test graph that satisfies equation (4). Assume that the -th iteration graph also satisfies Assumption 2. We will show by induction that the graph resulting from the the -th iteration, , also satisfies Assumption 2. By construction, Algorithm 1 chooses a combination of set of augmenting paths such that there exists a non-empty set of sequence flow paths such that the simple path from to characterized by does not form an -cycle for some . This implies that on the subgraph comprising of the edges in , equation (4) is satisfied.
If the maximum possible sequence flow in a minimally constrained test graph is , then we can find a combination such that for each , there exists a set , from which we can construct the set . By construction of the input variables to the ILP (6), the constraints of ILP (6) require that the sequence flow variable has atleast one element that is 1. This is possible only if there exists a set of edges that constrain such that there exists at least one sequence path that does not have any of its minimum-cut edges constrained, which is true as shown in Lemma (1). Therefore, the new graph satisfies Assumption (2). ∎∎
Consider iteration of the outer while loop in Algorithm 1, and let the graph at the -th iteration be . Denote . Let denote the maximum flow value from to on , for some such that . This implies that there is a set of edge-disjoint paths that characterize the maximum flow from to on . Let be the set of all simple paths from to that share an edge with the -th path in . Let be the set of minimum-cut edges on the paths in and let be the set of all edges on some path from to on . Clearly, .
For every , we can claim that because edges are removed to constrain in the -th iteration. Let be the number of iterations for to become disjoint. In the worst-case, edges continue to be constrained until iteration at which , at which point constraining edges to cut results in a cut separating and . Thus, has to be finite for every such .
On the test graph , any test run starting from state will satisfy the specification (2).
See Appendix 6.4 ∎
See Appendix 6.4. ∎
We illustrate the iterative synthesis of restrictions on a simple graph and a small gridworld, and then show runtimes of Algorithm 1 on random gridworld instances for both the case for which Assumption 2 is true, and the case for which Assumptions 2 and 3 are true.
Simple graph: Consider a simple non-deterministic Kripke structure representing an autonomous agent, shown in Figure 4, with propositional formulas labeled adjoining the states. The agent mission objective is to reach while being restricted to start from state . The test environment seeks to restrict transitions such that the agent is prompted to pass through waypoint in its trajectory to .
Inputs to Algorithm 1 include the labeled graph induced by the Kripke structure, the agent specification , the test specification , and the initial condition constraint .
Algorithm 1 constrains the edges in the first iteration, and the edges in the second iteration. Although in this simple example, searching the set of all augmented paths becomes searching over all paths, in larger examples discussed below, each augmented path represents a class of paths that share some edge(s) with it.
Simple Gridworld: In Figure 5, we illustrate the iterative synthesis of obstacles in a gridworld instance. Note that this configuration can be synthesized only by considering all sets of augmenting paths between () and (). Since there is no shortest augmenting path from to that does not form a cycle with some (in this example, there is only one) shortest augmenting path flow from to , it is imperative to use all sets of augmenting paths in the Sequence-Flows subroutine.
Random Gridworld Instances: For the case of setting all augmenting paths in the Sequence-Flows subroutine, we ran 50 random instances each for small gridworlds and propositions and plotted the average runtimes in Figure 5(a). The number of propositions are limited by the size of the gridworld instances, which is restricted by the combinatorial nature of finding all sets of augmenting paths, and all combinations of sets of augmenting paths.
If we choose initial gridworld instances that satisfy Assumption 3, then Algorithm 1 can synthesize static constraints for slightly larger grid sizes. The average runtimes for 50 random iterations for various grid sizes is plotted in Figure 5(b). The small increase to larger grid size is due to the Sequence-Flows subroutine reasoning over shortest augmenting paths, and not all augmenting paths.
The average runtimes increase exponentially with the size of the grid. The number of propositions, denoted by , is labeled if the test specification (2) is comprised of propositions (). In both Figures 5(a) and 5(b), the average runtime for fewer propositions is at times higher that the average runtime for more propositions. This can be attributed to the Sequence-Flows subroutine taking longer to enumerate all simple paths (or all shortest paths in case of Assumption 3) between two nodes, which could be greater in number due to fewer propositions constraining the graph.
in the legend. Error bars represent standard deviation of runtimes.
Another paradigm for the problem of synthesizing static test environments for sequence behaviors could be multi-commodity network flows, however, that setting considers multiple source-sink flows simultaneously drawing from the capacity of each edge, and here we compute separate network flows for every source-sink pair of nodes.
5 Conclusions and Future Work
In this paper, we proposed an algorithm to synthesize a static test environment to observe sequence-like behavior in a discrete-transition system. First, we formulated this test environment synthesis problem as a problem of synthesizing cuts on graphs using concepts of flow networks. Then, we proposed an algorithm which synthesized the cuts iteratively using an integer linear program. We proved that this algorithm is complete, and that the edges constrained by the ILP at each iteration maintain feasibility of the constraint in the next iteration. Finally, we conducted numerical experiments on random gridworld instances to assess the runtime of our algorithm. Simulation results preclude this algorithm from being tractable to larger examples. As future work, we will investigate heuristic methods to solve this problem, and explore static and reactive test environment synthesis for other classes of test specifications.
The authors would like to acknowledge Professor Mani Chandy, Dr. Shih-Hao Tseng, and Dr. Ugo Rosolia for insightful discussions. We acknowledge funding from AFOSR Test and Evaluation Program, grant FA9550-19-1-0302.
A Linear Temporal Logic (LTL) specification is inductively defined over a set of atomic propositions by the following syntax:
where is an atomic proposition, (“and”) and (“not”) are logical operators, and (“next”) and (“until”) are temporal operators. Other logical connectives such as (“or”) and can be defined: 1) , and 2) . Other temporal operators such as (“eventually”) and (“always”) can also be defined: 1) , and 2) . A propositional formula comprises only of logical connectives over atomic propositions. If a propositional formula, , is true at state , we denote it as .
Semantics: An LTL formula is evaluated over a run of the system. We denote holds at position of the run by . For a given set of atomic propositions and a run , the satisfaction of an LTL formula is defined inductively: 1) for an atomic proposition , if , 2) if it is not true that , 3) if both and , 4) if , and 5) if such that and , . Similarly, if such that , and if , . A run satisfies , denoted by if .
Definition 11 (Minimum cut).
Given a flow network , a source node and a sink node , a cut of is such that and , and and . Cut edges are edges such that and , and denote the sum of capacities of the cut edges as the cut capacity:
Definition 12 (Residual Network [cormen2009introduction]).
Given a flow network and a flow function and capacity , a residual network is one that has residual capacity function defined as follows,
6.2 Descriptions of Subroutines of Algorithm 1
The Min-Cut-Edges subroutine takes as input a graph , a list of propositions , and a non-empty set of augmenting paths for every source-sink pair () such that , and returns as output the set of minimum-cut edges on those augmenting paths. This information is then used in constructing the problem data for the ILP. The Sequence-Flows subroutine takes as input a graph , a list of propositions , and a parameter to indicate if Assumption 3 holds, computes the combination of all augmenting flows (or all shortest augmenting flows) that can result in a non-zero sequence flow from to . It returns as output the set of all sets of matrices that capture sequence-flow paths, , a set of , the total number of combinations, , and the maximum possible sequence flow value, , which is determined when is constructed.
6.3 Constraints of the Integer Linear Program (6)
The first constraint of the ILP, , enforces the requirement that each path in is constrained. Each row of corresponds to a path . The -th row of is constructed as follows,
In the second and third constraints, and , is used to determine the variable from the variable . Each row of corresponds to some path , and is a diagonal matrix. Suppose the -th row of corresponds to a path for , and is the set of minimum-cut edges on some path in , then the -th row is constructed as follows,
The -th diagonal entry of stores the total number of minimum-cut edges in the path corresponding to the -th row of .
These two constraints ensure that for some , iff at least one minimum-cut edge on the path corresponding to the -th row of is constrained, and iff none of the minimum-cut edges on the path corresponding to the -th row of are constrained.
The fourth and fifth constraints, and , determine the flow value for a given set of sequence flow paths, . Suppose the -th row of the matrix corresponds to some sequence flow path . Let denote the indices of the paths according to the ordering of the paths constituting all that is consistent with the construction of and . Then, the -th row of is defined as follows,
The -th diagonal entry of matrix stores the total number of ones in the -th row of , which is always .
The first constraint of the ILP, , enforces the requirement that each path in is constrained. Each row of corresponds to a path . The -th row of is constructed as follows,