Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems

1 Introduction

There is a recent hype of applying neural networks in automated driving, ranging from perception [3, 9] to the creation of driving strategies [14, 21] to even end-to-end driving setup [1]. Despite many public stories that seemly hint the technical feasibility of using neural networks, one fundamental challenge is to establish rigorous safety claims by considering all classes of relevant scenarios whose presence is subject to technical or societal constraints.

The key motivation of this work is that, apart from recent formal verification efforts [8, 7, 10, 5] where scalability and lack of specification are obvious concerns, the most plausible approach, from a certification perspective, remains to be testing. As domain experts or authorities in autonomous driving may suggest $n$ (incomplete) weighted criteria for describing the operating conditions such as weather, landscape, or partially occluding pedestrians, with these criteria one can systematically partition the domain and weight each partitioned class based on its relative importance. This step fits very well to the consideration as in automotive safety standard ISO 26262, where for deriving test cases, it is highly recommended to perform analysis of equivalence classes (Chap 6, Table 11, item 1b). Unfortunately, there is an exponential number of classes being partitioned, making the naïve coverage metric of having at least one data point in each class unfeasible. In addition, such a basic metric is qualitative in that it does not address the relative importance among different scenarios.

Towards above issues, in this paper we study the problem of quantitative $k$ -projection coverage, i.e., for arbitrary $k$ criteria being selected ( $k ≪ n$ being a small constant value), the data set, when being projected onto the $k$ -hyperplane, needs to have (in each region) data points no less than the associated weight. When $k$ is a constant, the size of required data points to achieve full quantitative $k$ -projection coverage remains polynomially bounded. Even more importantly, for the case where the composition of scenarios is constrained by rules, we present an NP algorithm to compute exact $k$ -projection coverage. This is in contrast to the case without projection, where computing exact coverage is $♯ {P}$ -hard.

Apart from calculating coverage, another crucial problem is to generate, based on the goal of increasing coverage, fewer scenarios if possible, as generating images or videos matching the scenario in autonomous driving is largely semi-automatic and requires huge human efforts. While we demonstrate that for unconstrained quantitative $1$ -projection, finding a minimum set of test scenarios to achieve full coverage remains in polynomial time, we prove that for $3$ -projection, the problem is NP-complete. To this end, we develop an efficient encoding to 0-1 integer programming which allows incrementally creating scenarios to maximally increase coverage.

To validate our approach, we have implemented a prototype to define and ensure coverage of a vision-based front-car detector. The prototype has integrated state-of-the-art traffic simulators and image synthesis frameworks [15, 25], in order to synthesize close-to-reality images specific to automatically proposed scenarios.

Figure 1: A total of

6

data points and their corresponding equivalence classes (highlighted as bounding boxes).

(Related Work)

The use of AI technologies, in particular the use of neural networks, has created fundamental challenges in safety certification. Since 2017 there has been a tremendous research advance in formally verifying properties of neural networks, with focuses on neurons using piecewise linear activation function (ReLU). For sound-and-complete approaches, Reluplex and Planet developed specialized rules for managing the 0-1 activation in the proof system

[10, 7]. Our previous work [5, 4] focused on the reduction to mixed integer liner programming (MILP) and applied techniques to compute tighter bounds such that in MILP, the relaxation bound is closer to the real bound. Exact approaches suffer from combinatorial explosion and currently the verification speed is not satisfactory. For imprecise yet sound approaches, recent work has been emphasizing linear relaxation of ReLU units by approximating them using outer convex polytopes [26, 11, 20]

, making the verification problem feasible for linear programming solvers. These approaches are even applied in the synthesis (training) process, such that one can derive provable guarantees

[11, 20]. Almost all verification work (apart from [10, 7, 4]) targets robustness properties, which is similar to adversarial testing (e.g., FGSM & iterative attacks [24], deepfool [16], Carlini-Wagner attacks [2]

) as in the machine learning community. All these approaches can be complemented with our approach by having our approach covering important scenarios, while adversarial training or formal verification measuring robustness within each scenario.

For classical structural coverage testing criteria such as MC/DC, they fail to deliver assurance promises, as satisfying full coverage either turns trivial (tanh) or intractable (ReLU). The recent work by Sun, Huang, and Kroening [22] borrows the concept of MC/DC and considers a structural coverage criterion, where one needs to find tests to ensure that for every neuron, its activation is supported by independent activation of neurons in its immediate previous layer. Such an approach can further be supported by concolic testing, as being recently demonstrated by same team [23]. Our work and theirs should be viewed as complementary, as we focus on the data space for training and testing neural networks, while they focus on the internal structure of a neural network. However, as in the original MC/DC concept, each condition in a conditional statement (apart from detecting errors in programming such as array out-of-bound which is not the core problem of neural networks) is designed to describe scenarios which should be viewed as natural consequences of input space partitioning (our work). Working on coverage criteria related to the internal structure of neural networks, provided that one cannot enforce the meaning of an individual neuron but can only empirically analyze it via reverse engineering (as in standard approaches like saliency maps [19]), is less likely provide direct benefits. Lastly, one major benefit of these structural testing approaches, based on the author claims, is to find adversarial examples via perturbation, but the benefit may be reduced due to new training methods with provable perturbation bounds [11, 20].

Lastly, our proposed metric tightly connects to the classic work of combinatorial testing and covering arrays [6, 17, 12, 13, 18]. However, as their application starts within hardware testing (i.e., each input variable being true or false), the quantitative aspects are not really needed and it does not need to consider constrained input cases, which is contrary to our practical motivation in the context of autonomous driving. For unconstrained cases, there are some results of NP-completeness in the field of combinatorial testing, which is largely based on the proof in [18]. It is not applicable to our case, as the proof is based on having freedom to define the set of groups to be listed in the projection. In fact, as listed in a survey paper [12], the authors commented that it remains open whether “the problem of generating a minimum test set for pairwise testing ( $k = 2$ ) is NP-complete” and “existing proof in [13] for the NP-completeness of pairwise testing is wrong” (due to the same reason where pairwise testing cannot have freedom to define the set of groups). Our new NP-completeness result in this paper can be viewed as a relaxed case by considering $k = 3$ with sampling being quantitative than qualitative.

2 Discrete Categorization and Coverage

Let $D S \subset R^{m}$ be the data space, $D \subset D S$ be a finite set called data set, and $d \in D S$ is called a data point. A categorization $C = ⟨ C_{1}, \dots, C_{n} ⟩$ is a list of functions that transform any data point $d$ to a discrete categorization point $C (d) = (C_{1} (d), \dots, C_{n} (d))$ , where for all $i \in {1, \dots, n}$ , $C_{i}$ has co-domain ${{0}, {1}, \dots, α}$ . Two data points $d_{1}$ and $d_{2}$ are equivalent by categorization, denoted by $d_{1} \equiv_{C} d_{2}$ , if $C (d_{1}) = C (d_{2})$ . The weight of a categorization $W = ⟨ W_{1}, \dots, W_{n} ⟩$ further assigns value $j \in {{0}, {1}, \dots, α}$ in the co-domain of $C_{i}$ with an integer value $W_{i} (j) \in {0, \dots, β}$ .

Next, we define constraints over categorization, allowing domain experts to express knowledge by specifying relations among categorizations. Importantly, for all data points in the data space, whenever they are transformed using $C$ , the transformed discrete categorization points satisfy the constraints.

Definition 1 (Categorization constraint).

A categorization constraint $C S = {C S_{1}, \dots, C S_{p}}$ is a set of constraints with each being a CNF formula having literals of the form $Ci(d)\emphopαi$ , where $\emphop∈{=,≠}$ and $αi∈{\emph0,…,α}$ .

Let $⊙_{i \in {1, \dots, n}} W_{i} (c_{i})$ abbreviate $W_{1} (c_{1}) ⊙ \dots ⊙ W_{n} (c_{n})$ , where $⊙ \in {+, \times, {max}}$ can be either scalar addition, multiplication, or max operators. In this paper, unless specially mentioned we always treat $⊙$ as scalar multiplication. Let $C (D)$ be the multi-set ${C (d) | d \in D}$ , and $\leq_{⊙}^{W}$ be set removal operation on $C (D)$ such that every categorization point $(c_{1}, \dots, c_{n}) \in C (D)$ has at most cardinality equal to $⊙_{i \in {1, \dots, n}} W_{i} (c_{i})$ . We define categorization coverage by requiring that for each discrete categorization point $(c_{1}, \dots, c_{n})$ , in order to achieve full coverage, have at least $⊙_{i \in {1, \dots, n}} W_{i} (c_{i})$ data points.

Definition 2 (Categorization coverage).

Given a data set $D$ , a categorization $C$ and its associated weights $W$ , define the categorization coverage $\emphcovC(D)$ for data set $D$ over $C$ and $W$ to be $|≤W⊙(C(D))|∑(c1,…,cn)∈\emphsat(CS)⊙i∈{1,…,n}Wi(ci)$ , where $\emphsat(CS)$ is the set of discrete categorization points satisfying constraints $C S$ .

(Example 1) In Fig. 1, let the data space $D S$ be $[0, 3) \times [0, 3) \times [0, 3)$ and the data set be $D = {d_{1}, \dots, d_{6}}$ . By setting $C = ⟨ C_{1}, C_{2}, C_{3} ⟩$ where $C_{i} = ⌊ x_{i} ⌋$ for $i \in {1, 2, 3}$ , then for data points $d_{2}$ , $d_{5}$ and $d_{6}$ , applying $C_{1}, C_{2}$ and $C_{3}$ creates $C (d_{2}) = C (d_{5}) = C (d_{6}) = ({2}, % 0, {2})$ , i.e., $d_{2} \equiv_{C} d_{5} \equiv_{C} d_{6}$ . Similarly, $d_{1} \equiv_{C} d_{4}$ .

If $C S$ is an empty set and $\forall i \in {1, 2, 3}, j \in {{0}, {1}, {2}} : W_{i} (j) = 1$ , then $| {sat} (C S) | = 3^{3} = 27$ , $\leq_{⊙}^{W} (C (D))$ removes $C (d_{2}), C (d_{4}), C (d_{5})$ by keeping one element in each equivalence class, and ${cov}_{C} (D)$ equals $\frac{3}{27} = \frac{1}{9}$ .
If $C S = {(C_{1} (d) \neq {0} \lor C_{2} (d) = {2})}$ and $\forall i \in {1, 2, 3}, j \in {{0}, {1}, {2}} : W_{i} (j) = 1$ , then $| {sat} (C S) | = 21$ rather than $27$ in the unconstrained case, and ${cov}_{C} (D)$ equals $\frac{3}{21} = \frac{1}{7}$ . Notice that all data points, once when being transformed into discrete categorization points, satisfy the categorization constraint.
Assume that $C S$ is an empty set, and $\forall i \in {1, 2, 3}, j \in {{0}, {1}, {2}}$ , $W_{i} (j)$ always returns $1$ apart from $W_{1} ({2})$ and $W_{3} ({2})$ returning $3$ . Lastly, let $⊙$ be scalar multiplication. Then for discrete categorization points having the form of $({2}, {-}, {2})$ , a total of $W_{1} ({2}) \times W_{3} ({2}) = 9$ data points are needed. One follows the definition and computes ${cov}_{C} (D)$ to be $\frac{3 + 1 + 1}{9 \times 3 + 3 \times 12 + 1 \times 12} = \frac{1}{13}$ .

Achieving $100 %$ categorization coverage is essentially hard, due to the need of exponentially many data points.

Proposition 1.

Provided that $C S = \emptyset$ and $\forall i \in {1, \dots, n}, j \in {{0}, \dots, α} : W_{i} (j) = 1$ , to achieve full coverage where ${cov}_{C} (D) = 1$ , $| D |$ is exponential to the number of categorizations.

Proof.

Based on the given condition, $| {sat} (C S) | = (α + 1)^{n}$ , and for each $(c_{i}, \dots, c_{n}) \in {sat} (C S)$ , $⊙_{i \in {1, \dots, n}} W_{i} (c_{i}) = 1$ . Therefore, ${cov}_{C} (D) = \frac{| \leq_{⊙}^{W} (C (D)) |}{(α + 1)^{n}}$ . As
$| \leq_{⊙}^{W} (C (D)) | \leq | C (D) |$ , to achieve full coverage $| C (D) |$ (and correspondingly $| D |$ ) needs to be exponential to the number of categorizations. ∎

Proposition 2.

Computing exact ${cov}_{C} (D)$ is $♯ {P}$ -hard.

Proof.

Computing the exact number of the denominator in Definition 2, under the condition of $α = {1}$ , equals to the problem of model counting for a SAT formula, which is known to be $♯ {P}$ -complete. ∎

3 Quantitative Projection Coverage

The intuition behind quantitative projection-based coverage is that, although it is unfeasible to cover all discrete categorization points, one may degrade the confidence by asking if the data set has covered every pair or triple of possible categorization with sufficient amount of data.

Definition 3 ( $k$ -projection).

Let set $Δ = {Δ_{1}, \dots, Δ_{k}} \subseteq {1, \dots, n}$ where elements in $Δ$ do not overlap. Given $d \in D$ , define the projection of a discrete categorization point $C (d)$ over $Δ$ to be ${Proj}_{Δ} (C (d)) = (C_{Δ_{1}} (d), C_{Δ_{2}} (d), \dots, C_{Δ_{k}} (d))$ .

Given a multi-set $S$ of discrete categorization points, we use ${Proj}_{Δ} (S)$ to denote the resulting multi-set by applying the projection function on each element in $S$ , and analogously define $\leq_{⊙_{Δ}}^{W} ({Proj}_{Δ} (S))$ to be a function which removes elements in ${Proj}_{Δ} (S)$ such that every element $(c_{Δ_{1}}, \dots, c_{Δ_{k}})$ has cardinality at most $W_{Δ_{1}} (c_{Δ_{1}}) ⊙ \dots ⊙ W_{Δ_{k}} (c_{Δ_{k}})$ . Finally, we define $k$ -projection coverage based on applying projection operation on the data set $D$ , for all possible subsets of $C$ of size $k$ .

Definition 4 ( $k$ -projection coverage).

Given a data set $D$ and categorization $C$ , define the $k$ -projection categorization coverage $\emphcovkC(D)$ for data set $D$ over $C$ and $W$ to be

∑{Δ:|Δ|=k}|≤W⊙Δ({Proj}Δ(C(D))|∑{Δ:|Δ|=k}∑(cΔ1,…,cΔk)∈\emphto−set({Proj}Δ(\emphsat(CS)))WΔ1(cΔ1)⊙…⊙WΔk(cΔk)

where function $\emphto−set()$ translates a multi-set to a set without element repetition.

(Example) Consider again Fig. 1 with $⊙$ being scalar multiplication, $C S = \emptyset$ and $\forall i \in {1, 2, 3}, j \in {{0}, {1}, {2}} : W_{i} (j) = 2$ .

For $k = 1$ , one computes ${cov}_{C}^{1} (D) = \frac{5 + 5 + 5}{(\frac{3}{1}) 3^{1} 2^{1}} = \frac{15}{18}$ . In the denominator, $Δ$ has $(\frac{3}{1})$ choices, namely $Δ = {1}$ , $Δ = {2}$ , or $Δ = {3}$ . Here we do detailed analysis over $Δ = {1}$ , i.e., we consider the projection to $C_{1}$ .
- Since $C S = \emptyset$ , ${sat} (C S)$ allows all possible $3^{3}$ assignments.
- ${Proj}_{Δ} ({sat} (C S))$ creates a set with elements 0, 1, 2 with each being repeated $9$ times, and ${to-set} ({Proj}_{Δ} ({sat} (C S)))$ removes multiplicity and creates ${{0}, {1}, {2}}$ . The sum equals $W_{1} ({0}) + W_{1} ({1}) + W_{1} ({2}) = 6$ .
The “ $5$ ” in the numerator comes from the contribution of $({2,0,2})$ with $2$ (albeit it has $3$ data points), $({1,1,1})$ with $2$ , and $({0,2,0})$ with $1$ .
For $k = 2$ , one computes ${cov}_{C}^{2} (D) = \frac{6 + 6 + 6}{(\frac{3}{2}) 3^{2} 2^{2}} = \frac{1}{6}$ . The denominator captures three hyper planes ( $x_{1} x_{2}$ , $x_{1} x_{3}$ , $x_{2} x_{3}$ ) with each having $3^{2}$ grids and with each grid allowing $2^{2}$ data points.

Notice that Definition 2 and 4 are the same when one takes $k$ with value $n$ .

Proposition 3.

${cov}_{C}^{n} (D) = {cov}_{C} (D)$ .

Proof.

When $k = n$ , the projection operator does not change ${sat} (C S)$ . Subsequently, to-set operator is not effective as is already a set, not a multi-set. Finally, we also have $W_{Δ_{1}} (c_{Δ_{1}}) ⊙ \dots ⊙ W_{Δ_{k}} (c_{Δ_{k}}) = ⊙_{i \in {1, \dots, n}} W_{i} (c_{i})$ . Thus the denominator part of Definition 2 and 4 are computing the same value. The argument also holds for the numerator part. Thus the definition of ${cov}_{C}^{n} (D)$ can be rewritten as ${cov}_{C} (D)$ . ∎

The important difference between categorization coverage and $k$ -projection coverage (where $k$ is a constant) includes the number of data points needed to achieve full coverage (exponential vs. polynomial), as well as the required time to compute exact coverage ( $♯ {P}$ vs. NP).

Proposition 4.

If $k$ is a constant, then to satisfy full $k$ -projection coverage, one can find a data set $D$ whose size is bounded by a number which is polynomial to $n$ , $α$ and $β$ .

Proof.

In Definition 4, the denominator is bounded by $(\frac{n}{k}) (α + 1)^{k} β^{k}$ .

The total number of possible $Δ$ with size $k$ equals $(\frac{n}{k})$ , which is a polynomial of $n$ with highest degree being $k$ .
For each $Δ$ , $(c_{Δ_{1}}, \dots, c_{Δ_{k}}) \in {to-set} ({Proj}_{Δ} ({sat} (C S)))$ has at most $(α + 1)^{k}$ possible assignments - this happens when $C S = \emptyset$ .
For each assignment of $(c_{Δ_{1}}, \dots, c_{Δ_{k}})$ , $W_{Δ_{1}} (c_{Δ_{1}}) ⊙ \dots ⊙ W_{Δ_{k}} (c_{Δ_{k}})$ can at most has largest value $β^{k}$ .

As one can use one data point for each element in the denominator, $D$ which achieves full coverage is polynomially bounded. ∎

(Example 2) Consider a setup of defining traffic scenarios where one has $α = 3$ and $n = 20$ . When $C S = \emptyset$ and $\forall i \in {1, \dots, n}, j \in {{0}, \dots, α} : W_{i} (j) = 1$ , the denominator of categorization coverage as defined in Definition 2 equals $3486784401$ , while the denominator of $2$ -projection coverage equals $1710$ and the denominator of $3$ -projection coverage equals $10260$ .

Proposition 5.

If $k$ is a constant, then computing $k$ -projection coverage can be done in NP. If $C S = \emptyset$ , then computing $k$ -projection coverage can be done in P.

Proof.

For the general case where $C S \neq \emptyset$ , to compute $k$ -projection coverage, the crucial problem is to know the precise value of the denominator. In the denominator, the part “ $(c_{Δ_{1}}, \dots, c_{Δ_{k}}) \in {to-set} ({Proj}_{Δ} ({sat} (C S)))$ ” is actually only checking if for grid $(c_{Δ_{1}}, \dots, c_{Δ_{k}})$ in the projected $k$ -hyperplane, whether it is possible to be occupied due to the constraint of $C S$ . If one knows that it can be occupied, simply add to the denominator by $W_{Δ_{1}} (c_{Δ_{1}}) ⊙ \dots ⊙ W_{Δ_{k}} (c_{Δ_{k}})$ . This “occupation checking” step can be achieved by examining the satisfiability of $C S$ with $C_{Δ_{i}}$ being replaced by the concrete assignment $(c_{Δ_{1}}, \dots, c_{Δ_{k}})$ of the grid. As there are polynomially many grids (there are $(\frac{n}{k})$ hyperplanes, with each having at most $(α + 1)^{k}$ grids), and for each grid, checking is done in NP (due to SAT problem being NP), the overall process is in NP.
For the special case where $C S = \emptyset$ , the “occupation checking” step mentioned previously is not required. As there are polynomially many grids (there are $(\frac{n}{k})$ hyperplanes, with each having at most $(α + 1)^{k}$ grids), the overall process is in P. ∎

4 Fulfilling $k$ -projection Coverage

As a given data set may not fulfill full $k$ -projection coverage, one needs to generate additional data points to increase coverage. By assuming that there exists a data generator function $G$ which can, from any discrete categorization point $c \in {{0}, \dots, α}^{n}$ , creates a new data point $G (c)$ in $D S$ such that $C (G (c)) = c$ and $G (c) \notin D$ (e.g., for image generation, $G$ can be realized using techniques such as conditional-GAN [15] to synthesize an image following the specified criterion, or using manually synthesized videos), generating data points to increase coverage amounts to the problem of finding additional discrete categorization points.

Definition 5 (Efficiently increasing $k$ -projection coverage).

Given a data set $D$ , categorization $C$ and generator $G$ , the problem of increasing $k$ -projection coverage refers to the problem of finding a minimum sized set $Θ \subseteq {{0}, \dots, α}^{n}$ , such that $\emphcovkC(D∪{G(c):c∈Θ})=1$ .

(Book-keeping $k$ -projection for a given data set)

For $Δ = {Δ_{1}, \dots, Δ_{k}}$ , we use $C_{Δ_{1}} \dots C_{Δ_{k}}$ to represent the data structure for book-keeping the covered items, and use subscript ” $_{{γ}}$ ” to indicate that certain categorization has been covered $γ$ times by the existing data set.

(Example 3) Consider the following three discrete categorization points ${({0}, {0}, {1}, {1}), ({1}, {0}, {0}, {0}), ({1}, {0% }, {0}, {1})}$ under $α = {1}$ . Results of applying $1$ -projection and $2$ -projection are book-kept in Equation 1 and 2 respectively.

C_{1} = {{0}_{{1}}, {1}_{{2}}}, C_{2} = {{0}_{{3}}, {1}_{{0}}}, C_{3} = {{0}_{{2}}, {1}_{{1}}}, C_{4} = {{0}_{{1}}, {1}_{{2}}}

(1)

C_{1} C_{2} = {{00}_{{1}}, {01}_{{0}}, {10}_{{2}}, {11}_{{0}}} C_{1} C_{3} = {{00}_{{0}}, {01}_{{1}}, {10}_{{2}}, {{11}%}_{{0}}} C_{1} C_{4} = {{00}_{{0}}, {01}_{{1}}, {10}_{{1}}, {11}_{{1}}} C_{2} C_{3} = {{0% 0}_{{2}}, {01}_{{1}}, {10}_{{0}}, {11}_{{0}}} C_{2} C_{4} = {{00}_{{1}}, {01}_{{2}}, {10}_{{0}}, {11}_{{0}}} C_{3} C_{4} = {{0% 0}_{{1}}, {01}_{{1}}, {10}_{{0}}, {11}_{{1}}}

(2)

(Full $k$ -projection coverage under $C S = \emptyset$ )

To achieve $k$ -projection coverage under $C S = \emptyset$ , in the worst case, one can always generate $(\frac{n}{k}) (α + 1)^{k} β^{k}$ discrete categorization points for $| Θ |$ in polynomial time. Precisely, to complete coverage on a particular projection $Δ$ , simply enumerate all possible assignments (a total of $(α + 1)^{k}$ assignments, as $k$ is a constant, the process is done in polynomial time) for all $(C_{Δ_{1}}, \dots, C_{Δ_{k}})$ , and extend them by associating $C_{i}$ , where $i \in {1, \dots, n} ∖ Δ$ , with arbitrary value within ${{0}, \dots, α}$ , and do it for $β^{k}$ times. For example, to increase $2$ -projection coverage in Equation 2, provided that $W_{i} (j) = 1$ , one first completes $C_{1} C_{2}$ by adding ${{01-\;-},{11-\;-}}$ where “-” can be either 0 or 1. One further improves $C_{1} C_{3}$ using ${{0-0-}, {1-1-}}$ , and subsequently all others.

As using $| Θ |$ to be $(\frac{n}{k}) (α + 1)^{k} β^{k}$ can still create problems when data points are manually generated from discrete categorization points, in the following, we demonstrate important sub-cases with substantially improved bounds over $| Θ |$ .

Proposition 6 ( $1$ -projection coverage).

Finding an additional set of discrete categorization points $Θ$ to achieve $1$ -projection coverage, with minimum size and under the condition of $C S = \emptyset$ , can be solved in time $O (α^{2} β n^{2})$ , with $| Θ |$ being bounded by $(α + 1) β$ .

Proof.

We present an algorithm (Algo. 1) that allows generating minimum discrete categorization points for full $1$ -projection coverage. Recall for $1$ -projection, our starting point is $C_{Δ_{1}}, \dots, C_{Δ_{n}}$ with each $C_{Δ_{i}}$ recording the number of appearances for element $j \in {{0}, \dots, α}$ . We use ${C_{Δ_{i}}}_{[j]}$ to denote the number of appearances for element $j$ in $C_{Δ_{i}}$ .

Data:

C_{Δ_{1}}, \dots, C_{Δ_{n}}

of a given data set, and weight function

W

Result: The minimum set

Θ

of additional discrete categorization points to guarantee full

1

-projection

1 while true do

2 let

c := (*, \dots, *)

;

3 for $i = 1, \dots, n$ do

4 for $j = {0}, \dots, α$ do

5 if ${C_{Δ_{i}}}_{[j]} < W_{i} (j)$ then

6 replace the

i

-th element of

c

by value

j

;

{C_{Δ_{i}}}_{[j]} := {C_{Δ_{i}}}_{[j]} + 1

;

8 break /* inner-loop */;

10 end if

12 end for

14 end for

15 if $c == (*, \dots, *)$ then return

Θ

;

16 else replace every

*

c

by value 0,

Θ := Θ \cup {c}

;

18 end while

Algorithm 1 Algorithm for achieving

1

-projection.

In Algo. 1, for every projection $i$ , the inner loop picks a value $j$ whose appearance in $C_{Δ_{i}}$ is lower than $W_{i} (j)$ (line 5-9). If no value is picked for some projection $i$ , then the algorithm just replaces $*$ by 0, before adding it to the set $Θ$ used to increase coverage (line 13). If after the iteration, $c$ remains to be $(*, \dots, *)$ , then we have achieved full $1$ -projection coverage and the program exits (line 12). The algorithm guarantees to return a set fulling full $1$ -projection with minimum size, due to the observation that each categorization is independent, so the algorithm stops so long as the categorization which misses most elements is completed. In the worst case, if projection $i$ started without any data, after $(α + 1) β$ iterations, it should have reached a state where it no longer requires additional discrete characterization points. Thus, $| Θ |$ is guaranteed to be bounded by $(α + 1) β$ .

Consider the example in Eq. 1. When $W_{i} (j) = 1$ , the above algorithm reports that only one additional discrete categorization point $({0}, {1}, {0}, {0})$ is needed to satisfy full $1$ -projection.

∎

On the other hand, efficiently increasing $3$ -projection coverage, even under the condition of $C S = \emptyset$ , is hard.

Proposition 7 (Hardness of maximally increasing $3$ -projection coverage, when $C S = \emptyset$ ).

Checking whether there exists one discrete categorization point to increase $3$ -projection coverage from existing value $χ$ to value $χ^{'}$ , under the condition where $⊙$ is scalar multiplication, is NP-hard.

Proof.

(Sketch) The hardness result is via a reduction from 3-SAT satisfiability, where we assume that each clause has exactly three variables. This problem is known to be NP-complete. We consider the case where $α = {2}$ and $β = 1$ , i.e., each categorization function creates values in ${{0}, {1}, {2}}$ . Given a 3-SAT formula $ϕ_{3 S A T}$ with $δ$ clauses, with each literal within the set of variables being ${C_{1}, C_{2}, \dots, C_{n}}$ , we perform the following construction.

Set the weight of categorization such that $W_{i} ({0}) = W_{i} ({1}) = 1$ and $W_{i} ({2}) = 0$ .
For each clause such as $(C_{x} \lor \neg C_{y} \lor C_{z})$ , we create a discrete categorization point by setting $C_{x} = {0}$ , $C_{y} = {1}$ , $C_{z} = {0}$ (i.e., the corresponding assignment makes the clause false) and by setting remaining $C_{i}$ to be 2. Therefore, the process creates a total of $δ$ discrete categorization points and can be done in polynomial time.
Subsequently, prepare the data structure and record the result of $3$ -projection for the above created discrete categorization points. As there are at most $(\frac{n}{3})$ boxes of form $C_{x} C_{y} C_{z}$ , with each box having $| {{0}, {1}, {2}} |^{3} = 27$ items, the construction can be conducted in polynomial time.
One can subsequently compute the $3$ -projection coverage. Notice that due to the construction of $W_{i} ({2}) = 0$ , all projected elements that contain value 2 should not be counted. The computed denominator should be $(\frac{n}{3}) (2)^{3}$ rather than $(\frac{n}{3}) (3)^{3}$ also due to $W_{i} ({2}) = 0$ .

Then the $ϕ_{3 S A T}$ problem has a satisfying instance iff there exists a discrete categorization point which increases the $3$ -projection coverage from $\frac{a}{(\frac{n}{3}) (2)^{3}}$ to value $\frac{a + (\frac{n}{3})}{(\frac{n}{3}) (2)^{3}}$ .

( $\Rightarrow$ ) If $ϕ_{3 S A T}$ has a satisfying instance, create a discrete categorization point where $C_{i} = {0}$ ( $C_{i} = {1}$ ) if the satisfying assignment of $ϕ_{3 S A T}$ , $C_{i}$ equals false (true). The created discrete categorization point, when being projected, will
- not occupy the already occupied space (recall that overlapping with existing items in each box implies that the corresponding clause can not be satisfied), and
- not occupy a grid having $C_{i} = {2}$ (as the assignment only makes $C_{i}$ to be 0 or 1), making the point being added truly help in increasing the numerator of the computed coverage.
Overall, each projection will increase value by $1$ , and therefore, the $3$ -projection coverage increases from $\frac{a}{(\frac{n}{3}) (2)^{3}}$ to value $\frac{a + (\frac{n}{3})}{(\frac{n}{3}) (2)^{3}}$ .
( $\Leftarrow$ ) Conversely, if there exists one discrete categorization point to increase coverage by $(\frac{n}{3})$ , due to the fact that we only have one point and there are $(\frac{n}{3})$ projections, it needs to increase in each box representing $3$ -projection, without being overlapped with existing items in that box and without having value 2 being used. One can subsequently use the value of the discrete categorization point to create a satisfying assignment. ∎

In the following, we present an algorithm which encodes the problem of finding a discrete categorization point with maximum coverage increase to a 0-1 integer programming problem. Stated in Algo. 2, line 1 prepares variables and constraints to be placed in the 0-1 programming framework. For each categorization $C_{i}$ , for each possible value we create an 0-1 variable ${var}_{[C_{i} = j]}$ (line 3-5), such that ${var}_{[C_{i} = j]} = 1$ iff the newly introduced discrete categorization point has $C_{i}$ using value $j$ . As the algorithm proceeds by only generating one discrete categorization point, only one of them can be true, which is reflected in the constraint $\sum_{j = 0}^{α} {var}_{[C_{i} = j]} = 1$ in line 6.

Then starting from line 8, the algorithm checks if a particular projected value still allows improvement ${C_{Δ_{1}} \dots C_{Δ_{k}}}_{[v_{Δ_{1}} \dots v_{Δ_{k}}]} < W_{Δ_{1}} (v_{Δ_{1}}) ⊙ \dots ⊙ W_{Δ_{k}} (v_{Δ_{k}})$ . If so, then create a variable ${occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]}$ (line 10) such that it is set to $1$ iff the newly introduced discrete categorization point will occupy this grid when being projected. As our goal is to maximally increase $k$ -projection coverage, ${occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]}$ is introduced in the objective function (line 11 and 16) where the sum of all variables is the objective to be maximized. Note that ${occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]}$ is set to $1$ iff the newly introduced discrete categorization point guarantees that $C_{Δ_{1}} = v_{Δ_{1}} \land \dots \land C_{Δ_{k}} = v_{Δ_{k}}$ . For this purpose, line 12 applies a standard encoding tactic in 0-1 integer programming to encode such a condition - If ${var}_{[C_{Δ_{1}} = v_{Δ_{1}}]} = \dots = {var}_{[C_{Δ_{k}} = v_{Δ_{k}}]} = 1$ , then ${var}_{[C_{Δ_{1}} = v_{Δ_{1}}]} + \dots + {var}_{[C_{Δ_{k}} = v_{Δ_{k}}]} = k$ . Thus ${occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]}$ will be set to $1$ to enforce satisfaction of the right-hand inequality of the constraint. Contrarily, if any of ${var}_{[C_{Δ_{j}} = v_{Δ_{j}}]}$ , where $j \in {1, \dots, k}$ has value $0$ , then ${occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]}$ needs to set to $0$ , in order to enforce the left-hand inequality of the constraint. Consider the example in Eq. 2, where one has $C_{1} C_{2} = {{00}_{{1}}, {01}_{{0}}, {10}_{{2}}, {11}_{{0}}}$ . For improving ${01}_{{0}}$ , line 12 generates the following constraint $0 \leq {var}_{[C_{1} = {0}]} + {var}_{[C_{2} = {1}]} - 2 {occ}_{[C_{1} = {0}, C_{2} = {1}]} \leq 1$ .

Line 14 will be triggered when no improvement can be made by every check of line 9, meaning that the system has already achieved full $k$ -projection coverage. Lastly, apply 0-1 integer programming where one translates variable ${var}_{[C_{i} = v_{i}]}$ having value $1$ by assigning $C_{i}$ to $v_{i}$ in the newly generated discrete categorization point (line 17, 18).

Here we omit technical details, but Algo. 2 can easily be extended to constrained cases by adding $C S$ to the list of constraints.

Data: The set

{C_{Δ_{1}} \dots C_{Δ_{k}}}

of the current

k

-projection records, and weight function

W

Result: One discrete categorization point

(c_{1}, \dots, c_{n})

which maximally increase coverage, or null if current records have achieved full coverage.

1 let

{var}_{0 / 1} := \emptyset

{constraints}_{0 / 1} := \emptyset

{objvar}_{0 / 1} := \emptyset

2 forall $C_{i}$ , $i \in {1, \dots, n}$ do

3 forall $j \in {0, \dots, α}$ do

{var}_{0 / 1} := {var}_{0 / 1} \cup {{var}_{[C_{i} = j]}}

;

6 end forall

{constraints}_{0 / 1} := {constraints}_{0 / 1} \cup {\sum_{j = 0}^{α} {var}_{[C_{i} = j]} = 1}

;

9 end forall

10forall $C_{Δ_{1}} \dots C_{Δ_{k}}$ do

11 if ${C_{Δ_{1}} \dots C_{Δ_{k}}}_{[v_{Δ_{1}} \dots v_{Δ_{k}}]} < W_{Δ_{1}} (v_{Δ_{1}}) ⊙ \dots ⊙ W_{Δ_{k}} (v_{Δ_{k}})$ then

{var}_{0 / 1} := {var}_{0 / 1} \cup {{occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]}}

;

{objvar}_{0 / 1} := {objvar}_{0 / 1} \cup {{occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]}}

;

{constraints}_{0 / 1} := {constraints}_{0 / 1} \cup {0 \leq {% var}_{[C_{Δ_{1}} = v_{Δ_{1}}]} + \dots + {var}_{[C_{Δ_{k}} = v_{Δ_{k}}]} - k {occ}_{[C_{Δ_{1}} = v_{Δ_{1}}, \dots, C_{Δ_{k}} = v_{Δ_{k}}]} \leq k - 1}

;

16 end if

17 if $\emphobjvar=∅$ then return null;

18 else

19 let

{obj} := \sum_{{var} \in {objvar}} {var}

;

20 let

{assignment} := {0/1-programming}_{{{var}_{0 / 1}}} (maximize {obj} subject-to {% constraint}_{0 / 1})

;

21 return

(v_{1}, \dots, v_{n})

where in assignment

{var}_{[C_{i} = v_{i}]}

is assigned to

1

;

23 end if

25 end forall

Algorithm 2 Finding a discrete categorization point which maximally increases

k

-projection coverage, via an encoding to 0-1 integer programming.

5 Implementation and Evaluation

Figure 2: Workflow in the developed prototype for quantitative projection coverage and generation of new synthetic data.

Figure 3: Existing data points (E $_{1}$ to E $_{5}$ ), and the automatically generated data points (G $_{1}$ to G $_{6}$ ) to achieve full coverage.

We have implemented above mentioned techniques as a workbench to support training vision-based perception units for autonomous driving. The internal workflow of our developed tool is summarized in Fig. 2. It takes existing labelled/categorized data set and the user-specified $k$ value as input, computes $k$ -projection coverage, and finds a new discrete categorization point which can increase the coverage most significantly. For the underlying 0-1 programming solving, we use IBM CPLEX Optimization Studio¹¹1IBM CPLEX Optimization Studio: https://www.ibm.com/analytics/data-science/prescriptive-analytics/cplex-optimizer.

To convert the generated discrete categorization points to real images, we have further implemented a C++ plugin over the Carla simulator²²2Carla Simulator: http://carla.org/, an open-source simulator for autonomous driving based on Unreal Engine 4³³3Unreal Engine 4: https://www.unrealengine.com/. The plugin reads the scenario from the discrete categorization point and configures the ground truth, the weather, and the camera angle accordingly. Then the plugin starts the simulation and takes a snapshot using the camera mounted on the simulated vehicle. The camera can either return synthetic images (e.g., images in Fig. 3

) or images with segmentation information, where for the latter one, we further generate close-to-real image via applying conditional GAN framework Pix2Pix from NVIDIA

⁴⁴4https://github.com/NVIDIA/pix2pixHD. Due to space limits, here we detail a small example by choosing the following operating conditions of autonomous vehicles as our categories.

Weather = ${{Sunny}, {Cloudy}, {Rainy}}$
Lane orientation = ${{Straight}, {Curvy}}$
Total number of lanes (one side) = ${1, 2}$
Current driving lane = ${1, 2}$
Forward vehicle existing = ${{true}, {false}}$
Oncoming vehicle existing = ${{true}, {false}}$

We used our test case generator to generate new data points to achieve full $2$ -projection coverage (with $W_{i} (j) = 1$ ) starting with a small set of randomly captured data points (Fig. 3, images E $_{1}$ to E $_{5}$ ). Images G $_{1}$ to G $_{6}$ are synthesized in sequence until full $2$ -projection coverage is achieved. The coverage condition of each 2-projection plane is shown in Table 1. Note that there exists one entry in the sub-table (f) which is not coverable (labelled as “X”), as there is a constraint stating that if there exists only $1$ lane, it is impossible for the vehicle to drive on the $2^{n d}$ lane. Fig. 4 demonstrates the growth of $2$ -projection coverage when gradually introducing images G $_{1}$ to G $_{6}$ .

	1 Lane	2 Lanes
Sunny	G $_{2}$	G $_{1}$
Cloudy	G $_{4}$	G $_{3}$
Rainy	E $_{3}$	E $_{4}$

(a) Weather &

#

Lanes

	$1^{s t}$ Lane	$2^{n d}$ Lane
Sunny	G $_{2}$	G $_{1}$
Cloudy	G $_{4}$	G $_{3}$
Rainy	E $_{5}$	G $_{5}$

(b) Weather & Current Lane

	Straight	Curvy
Sunny	G $_{1}$	G $_{6}$
Cloudy	G $_{4}$	G $_{3}$
Rainy	G $_{5}$	E $_{5}$

	No FC	FC
Sunny	G $_{1}$	G $_{2}$
Cloudy	G $_{4}$	G $_{3}$
Rainy	G $_{5}$	E $_{4}$

(d) Weather & Forward Car

	No OC	OC
Sunny	G $_{1}$	G $_{2}$
Cloudy	G $_{4}$	G $_{3}$
Rainy	G $_{5}$	E $_{4}$

(e) Weather & Oncoming Car

	$1^{s t}$ Lane	$2^{n d}$ Lane
1 Lane	E $_{3}$	X
2 Lanes	E $_{2}$	G $_{1}$

(f)

#

Lanes & Current Lane

	Straight	Curvy
1 Lane	G $_{2}$	E $_{3}$
2 Lanes	E $_{1}$	E $_{2}$

(g)

#

Lanes & Lane Curve

	No FC	FC
1 Lane	E $_{3}$	G $_{2}$
2 Lanes	G $_{1}$	G $_{3}$

(h)

#

Lanes & Forward Car

	No OC	OC
1 Lane	G $_{4}$	E $_{3}$
2 Lanes	E $_{1}$	E $_{4}$

(i)

#

Lanes & Oncoming Car

	Straight	Curvy
$1^{s t}$ Lane	E $_{1}$	E $_{2}$
$2^{n d}$ Lane	G $_{1}$	G $_{3}$

(j) Current Lane & Lane Curve

	No FC	FC
$1^{s t}$ Lane	E $_{3}$	E $_{1}$
$2^{n d}$ Lane	G $_{1}$	G $_{3}$

(k) Current Lane & Forward Car

	No OC	OC
$1^{s t}$ Lane	E $_{1}$	E $_{3}$
$2^{n d}$ Lane	G $_{1}$	G $_{3}$

(l) Current Lane & Oncoming Car

	No FC	FC
Straight	G $_{1}$	E $_{1}$
Curvy	E $_{3}$	E $_{2}$

(m) Lane Curve & Forward Car

	No OC	OC
Straight	E $_{1}$	G $_{2}$
Curvy	E $_{4}$	E $_{2}$

(n) Lane Curve & Oncoming Car

	No OC	OC
No FC	G $_{1}$	E $_{3}$
FC	E $_{2}$	E $_{4}$

(o) Forward Car & Oncoming Car

Table 1:

2

-projection coverage tables of the final data set

Figure 4: Change of $2$ -projection coverage due to newly generated data.

6 Concluding Remarks

In this paper, we presented quantitative $k$ -projection coverage as a method to systematically evaluate the quality of data for systems engineered using machine learning approaches. Our prototype implementation is used to compute coverage and synthesize additional images for engineering a vision-based perception unit for automated driving. The proposed metric can further be served as basis to refine other classical metrics such as MTBF or availability which is based on statistical measurement.

Currently, our metric is to take more data points for important (higher weight) scenarios. For larger $k$ values, achieving full projection coverage may not be feasible, so one extension is to adapt the objective function of Algo. 2 such that the generation process favors discrete categorization points with higher weights when being projected. Another direction is to improve the encoding of Algo. 2 such that the algorithm can return multiple discrete categorization points instead of one. Yet another direction is to further associate temporal behaviors to categorization and the associated categorization constraints, when the data space represents a sequence of images.

References

[1] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to end learning for self-driving cars. CoRR, abs/1604.07316, 2016.
[2] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pages 39–57. IEEE, 2017.
[3] C. Chen, A. Seff, A. Kornhauser, and J. Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In
Proceedings of the IEEE International Conference on Computer Vision
, pages 2722–2730, 2015.
[4] C.-H. Cheng, F. Diehl, G. Hinz, Y. Hamza, G. Nührenberg, M. Rickert, H. Ruess, and M. Truong-Le. Neural networks for safety-critical applicati

Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems

Authors

Testing Autonomous Systems with Believed Equivalence Refinement

DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks

Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components

ComOpT: Combination and Optimization for Testing Autonomous Driving Systems

Optimizing the Efficiency of Accelerated Reliability Testing for the Internet Router Motherboard

1 Introduction

2 Discrete Categorization and Coverage

Definition 1 (Categorization constraint).

Definition 2 (Categorization coverage).

Proposition 1.

Proof.

Proposition 2.

Proof.

3 Quantitative Projection Coverage

Definition 3 ( $k$ -projection).

Definition 4 ( $k$ -projection coverage).

Proposition 3.

Proof.

Proposition 4.

Proof.

Proposition 5.

Proof.

4 Fulfilling $k$ -projection Coverage

Definition 5 (Efficiently increasing $k$ -projection coverage).

(Book-keeping $k$ -projection for a given data set)

(Full $k$ -projection coverage under $C S = \emptyset$ )

Proposition 6 ( $1$ -projection coverage).

Proof.

Proposition 7 (Hardness of maximally increasing $3$ -projection coverage, when $C S = \emptyset$ ).

Proof.

5 Implementation and Evaluation

6 Concluding Remarks

References

Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems

Authors

Related Research

Testing Autonomous Systems with Believed Equivalence Refinement

DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks

Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components

ComOpT: Combination and Optimization for Testing Autonomous Driving Systems

Optimizing the Efficiency of Accelerated Reliability Testing for the Internet Router Motherboard

This week in AI

1 Introduction

2 Discrete Categorization and Coverage

Definition 1 (Categorization constraint).

Definition 2 (Categorization coverage).

Proposition 1.

Proof.

Proposition 2.

Proof.

3 Quantitative Projection Coverage

Definition 3 (k-projection).

Definition 4 (k-projection coverage).

Proposition 3.

Proof.

Proposition 4.

Proof.

Proposition 5.

Proof.

4 Fulfilling k-projection Coverage

Definition 5 (Efficiently increasing k-projection coverage).

(Book-keeping k-projection for a given data set)

(Full k-projection coverage under CS=∅)

Proposition 6 (1-projection coverage).

Proof.

Proposition 7 (Hardness of maximally increasing 3-projection coverage, when CS=∅).

Proof.

5 Implementation and Evaluation

6 Concluding Remarks

References

Definition 3 ( $k$ -projection).

Definition 4 ( $k$ -projection coverage).

4 Fulfilling $k$ -projection Coverage

Definition 5 (Efficiently increasing $k$ -projection coverage).

(Book-keeping $k$ -projection for a given data set)

(Full $k$ -projection coverage under $C S = \emptyset$ )

Proposition 6 ( $1$ -projection coverage).

Proposition 7 (Hardness of maximally increasing $3$ -projection coverage, when $C S = \emptyset$ ).