 # Complete Test Sets And Their Approximations

We use testing to check if a combinational circuit N always evaluates to 0 (written as N ≡ 0). We call a set of tests proving N ≡ 0 a complete test set (CTS). The conventional point of view is that to prove N ≡ 0 one has to generate a trivial CTS. It consists of all 2^|X| input assignments where X is the set of input variables of N. We use the notion of a Stable Set of Assignments (SSA) to show that one can build a non-trivial CTS consisting of less than 2^|X| tests. Given an unsatisfiable CNF formula H(W), an SSA of H is a set of assignments to W that proves unsatisfiability of H. A trivial SSA is the set of all 2^|W| assignments to W. Importantly, real-life formulas can have non-trivial SSAs that are much smaller than 2^|W|. In general, construction of even non-trivial CTSs is inefficient. We describe a much more efficient approach where tests are extracted from an SSA built for a `projection' of N on a subset of variables of N. These tests can be viewed as an approximation of a CTS for N. We give experimental results and describe potential applications of this approach.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## I Introduction

Testing is an important part of verification flows. For that reason, any progress in understanding testing and improving its quality is of great importance. In this paper, we consider the following problem. Given a single-output combinational circuit , find a set of input assignments (tests) proving that evaluates to 0 for every test (written as ) or find a counterexample. We will call a set of input assignments proving a complete test set (CTS)111Term CTS is sometimes used to say that a test set invokes every event specified by a coverage metric. Our application of this term is quite different. . We will call the set of all possible tests a trivial CTS. Typically, one assumes that proving involves derivation of the trivial CTS, which is infeasible in practice. Thus, testing is used only for finding an input assignment refuting . We present an approach for building a non-trivial CTS consisting only of a subset of all possible tests. In general, finding even a non-trivial CTS for a large circuit is impractical. We describe a much more efficient approach where an approximation of a CTS is generated.

The circuit above usually describes a property of a multi-output combinational circuit , the latter being the real object of testing. For instance, may state that never produces some output assignments. To differentiate CTSs and their approximations from conventional test sets verifying “as a whole”, we will refer to the former as property-checking test sets. Let be the set of properties of formulated by a designer. Assume that every property of holds and is a test set generated to check property . There are at least two reasons why applying to makes sense. First, if is incomplete222That is can be incorrect even if all properties of hold., a test of can expose a bug, if any, breaking a property of that is not in . Second, if property is defined incorrectly, a test of may expose a bug breaking the correct version of . On the other hand, if produces proper output assignments for all tests of , one gets extra guarantee that is correct. In Section VI, we list some other applications of property-checking test sets such as verification of design changes, hitting corner cases and testing sequential circuits.

Let be a single-output combinational circuit where and specify the sets of input and internal variables of respectively and specifies the output variable of . Let be a formula defining the functionality of (see Section III). We will denote the set of variables of circuit (respectively formula ) as (respectively ). Every assignment333By an assignment to a set of variables , we mean a full assignment where every variable of is assigned a value. to satisfying corresponds to a consistent assignment444An assignment to a gate of is called consistent if the value assigned to the output variable of is implied by values assigned to its input variables. An assignment to variables of is called consistent if it is consistent for every gate of . to and vice versa. Then the problem of proving reduces to showing that formula is unsatisfiable. From now on, we assume that all formulas mentioned in this paper are propositional. Besides, we will assume that every formula is represented in CNF i.e. as a conjunction of disjunctions of literals.

Our approach is based on the notion of a Stable Set of Assignments (SSA) introduced in . Given formula , an SSA of is a set of assignments to variables of that have two properties. First, every assignment of falsifies . Second, is a transitive closure of some neighborhood relation between assignments (see Section II). The fact that has an SSA means that the former is unsatisfiable. Otherwise, an assignment satisfying is generated when building its SSA. If is unsatisfiable, the set of all assignments is always an SSA of . We will refer to it as trivial. Importantly, a real-life formula can have a lot of SSAs whose size is much less than . We will refer to them as non-trivial. As we show in Section II, the fact that is an SSA of is a structural property of the latter. That is this property cannot be expressed in terms of the truth table of (as opposed to a semantic property of ). For that reason, if is an SSA for , it may not be an SSA for some other formula that is logically equivalent to . In other words, a structural property is formula-specific.

We show that a CTS for can be easily extracted from an SSA of formula . This makes a non-trivial CTS a structural property of circuit that cannot be expressed in terms of its truth table. Building an SSA for a large formula is inefficient. So, we present a procedure constructing a simpler formula implied by where and building an SSA of . The existence of such an SSA means that (and hence ) is unsatisfiable. So, holds. A test set extracted from an SSA of can be viewed as a way to verify a “projection” of on variables of . On the other hand, one can consider this set as an approximation of a CTS for . We will refer to the procedure above as (“Semantics and Structure”). combines semantic and structural derivations, hence the name. The semantic part of is555Implication is a semantic property of . To verify this property it suffices to know the truth table of . to derive . Its structural part consists of constructing an SSA of thus proving that is unsatisfiable.

The contribution of this paper is fourfold. First, we introduce the notion of non-trivial CTSs (Section III). Second, we present a method for efficient construction of property-checking tests that are approximations of CTSs (Sections IV and V). Third, we describe applications of such tests (Section VI). Fourth, we give experimental results showing the effectiveness of property-checking tests (Section VII).

## Ii Stable Set Of Assignments

### Ii-a Definitions

We will refer to a disjunction of literals as a clause. Let   be an assignment to a set of variables . Let   falsify a clause . Denote by the set of assignments to satisfying that are at Hamming distance 1 from . (Here Nbhd stands for “Neighborhood”). Thus, the number of assignments in is equal to that of literals in . Let   be another assignment to (that may be equal to ). Denote by the subset of consisting only of assignments that are farther away from than (in terms of the Hamming distance).

###### Example 1

Let and =0110. We assume that the values are listed in in the order the corresponding variables are numbered i.e. , . Let . (Note that falsifies .) Then = where = 1110 and =0100. Let = 0000. Note that is actually closer to than . So =.

###### Definition 1

Let be a formula666We use the set of clauses as an alternative representation of a CNF formula . specified by a set of clauses . Let = be a set of assignments to such that every falsifies . Let denote a mapping where is a clause of falsified by . We will call an AC-mapping where “AC” stands for “Assignment-to-Clause”. We will denote the range of as . (So, a clause of is in iff there is an assignment such that .)

###### Definition 2

Let be a formula specified by a set of clauses . Let = be a set of assignments to . is called a Stable Set of Assignments777In , the notion of “uncentered” SSAs was introduced. The definition of an uncentered SSA is similar to Definition 2. The only difference is that one requires that for every , holds instead of . (SSA) of with center if there is an AC-mapping such that for every , holds where .

###### Example 2

Let consist of four clauses: , , , . Let where , , , . Let be an AC-mapping specified as . Since falsifies , ,   is a correct AC-mapping. is an SSA of with respect to and center =. Indeed, = where and = , where , . Thus, , .

### Ii-B SSAs and satisfiability of a formula

###### Proposition 1

Formula is unsatisfiable iff it has an SSA.

The proof888The proof of Proposition 1 presented in report  is inacurate. is given Appendix A. A similar proposition is proved in  for “uncentered” SSAs (see Footnote 7).

The set of all assignments to forms the trivial uncentered SSA of . Example 2 shows a non-trivial SSA. The fact that formula has a non-trivial SSA is its structural property. That is one cannot check whether is an SSA of if only the truth table of is known. In particular, may not be an SSA of a formula logically equivalent to .

The relation between SSAs and satisfiability can be explained as follows. Suppose that formula is satisfiable. Let be an arbitrary assignment to and be a satisfying assignment that is the closest to in terms of the Hamming distance. Let be the set of all assignments to that falsify and be an AC-mapping from to . Then can be reached from by procedure BuildPath shown in Figure 1. It generates a sequence of assignments where = and =. First, BuildPath checks if current assignment equals . If so, then has been reached. Otherwise, BuildPath uses clause to generate next assignment. Since satisfies , there is a variable that is assigned differently in and . BuildPath generates a new assignment obtained from by flipping the value of .

BuildPath reaches in steps where is the Hamming distance between and . Importantly, BuildPath reaches for any AC-mapping. Let be an SSA of with respect to center and AC-mapping . Then if BuildPath starts with and uses as an AC-mapping, it can reach only assignments of . Since every assignment of falsifies , no satisfying assignment can be reached.

A procedure for generation of SSAs called BuildSSA is shown in Figure 2. It accepts formula and outputs either a satisfying assignment or an SSA of , center and AC-mapping . BuildSSA maintains two sets of assignments denoted as and . Set contains the examined assignments i.e. those whose neighborhood is already explored. Set specifies assignments that are queued to be examined. is initialized with an assignment and is originally empty. BuildSSA updates and in a while loop. First, BuildSSA picks an assignment of and checks if it satisfies . If so, is returned as a satisfying assignment. Otherwise, BuildSSA removes   from and picks a clause of falsified by . The assignments of that are not in are added to . After that, is added to as an examined assignment, pair is added to and a new iteration begins. If is empty, is an SSA with center and AC-mapping .

## Iii Complete Test Sets

Let be a single-output combinational circuit where and specify the input and internal variables of respectively and specifies the output variable of . Let consist of gates . Then can be represented as where is a CNF formula specifying the consistent assignments of gate . Proving reduces to showing that formula is unsatisfiable.

###### Example 3

Circuit shown in Figure 3 represents equivalence checking of expressions and specified by gates and respectively. Formula is equal to where, for instance, , , , . Every satisfying assignment to corresponds to a consistent assignment to gate and vice versa. For instance, satisfies and is a consistent assignment to since the latter is an OR gate. Formula is unsatisfiable due to functional equivalence of expressions and . Thus, .

Let be a test i.e. an assignment to . The set of assignments to sharing the same assignment to forms a cube of assignments. Recall that Denote this set as . Only one assignment of specifies the correct execution trace produced by under . All other assignments can be viewed as “erroneous” traces under test .

###### Definition 3

Let be a set of tests where . We will say that is a Complete Test Set (CTS) for if contains an SSA for formula .

If satisfies Definition 3, set “contains” a proof that and so can be viewed as complete. If , is the trivial CTS. In this case, contains the trivial SSA consisting of all assignments to . Given an SSA of , one can easily generate a CTS by extracting all different assignments to that are present in the assignments of .

###### Example 4

Formula of Example 3 has an SSA of 21 assignments to . They have only 5 different assignments to  . The set of  those assignments is a CTS for .

Definition 3 is meant for circuits that are not “too redundant”. Highly-redundant circuits are discussed in report  and Appendix B.

## Iv SemStr Procedure

### Iv-a Motivation

Building an SSA for a large formula is inefficient. So, constructing a CTS of from an SSA of is impractical. To address this problem, we introduce a procedure called (a short for “Semantics and Structure”). Given formula and a set of variables , generates a simpler formula implied by at the same time trying to build an SSA for . If succeeds in constructing such an SSA, formula is unsatisfiable and so is . Then a set of tests is extracted from this SSA. As we show in Subsection V-A, one can view as an approximation of a CTS for (if ) or an “approximation of approximation” of a CTS (if ).

###### Example 5

Consider the circuit of Figure 3 where . Assume that . Application of to produces . also generates an SSA of of four assignments to : with center =000. (We omit the AC-mapping here.) These assignments form an approximation of CTS for .

### Iv-B Description of SemStr

The pseudocode of is shown in Figure 4. accepts formula (in our case, ) and a set of variables . outputs an assignment satisfying or formula implied by and an SSA of . Originally, the set of clauses is empty. is computed in a while loop. First, tries to build an SSA for the current formula by calling BuildSSA (line 3). If is unsatisfiable, BuildSSA computes an SSA returned by (line 5). Otherwise, BuildSSA returns an assignment satisfying . In this case, calls procedure GenCls to build a clause falsified by . Clause is obtained by resolving clauses of on variables of . (Hence is implied by .) If can be extended to an assignment satisfying , terminates (lines 7-8). Otherwise, is added to and a new iteration begins.

Procedure GenCls is shown in Figure 5. First, GenCls generates formula obtained from by discarding clauses satisfied by and removing literals falsified by . Then GenCls checks if there is an assignment satisfying . If so, is returned as an assignment satisfying . Otherwise, a proof of unsatisfiability of is produced. Then GenCls forms a set . A variable is in iff a clause of is used in proof and its parent clause from has a literal of falsified by . Finally, clause is generated as a disjunction of literals of falsified by . By construction, clause is implied by and falsified by .

## V Building Approximations Of CTS

### V-a Two kinds of approximations of CTSs

As before, let denote a formula implied by that is generated by and denote an SSA for . Projections of can be of two kinds depending on whether holds. Let hold and be the test set extracted from as described in Section III. That is consists of all different assignments to present in the assignments of . On one hand, using the reasoning of Section III one can show that is a CTS for projection of on . On the other hand, since is essentially an abstraction of , set is an approximation of a CTS for . For that reason, we will refer to as a CTSa of where superscript “a” stands for “approximation”.

Now assume that holds. Generation of a test set from for this case is described in the next section. The set can be viewed as an approximation of a set built for projection of on set . Since is a CTSa for , we will refer to as CTSaa where “aa” stands for “approximation of approximation”.

### V-B Construction of CTSaa

Consider extraction of a test set from SSA of formula when . Since , in general, contains internal variables999If the special case holds, every assignment of can be easily turned into a test by assigning values to variables of (e.g. randomly). of , translation of to a test set needs a special procedure GenTests shown in Figure 6. For every assignment of , GenTests checks if formula is satisfiable under assignment (i.e. if there exists a test under which assigns to ). If so, an assignment to is extracted from the satisfying assignment and added to as a test. Otherwise, GenTests runs a for loop (lines 8-13) of iterations. In every iteration, GenTests relaxes by removing the clauses specifying a small subset of gates picked randomly. If the relaxed version of is satisfiable, a test is extracted from the satisfying assignment and added to .

### V-C Finding a set of variables to project on

Intuitively, a good choice of the set to project on is a (small) coherent subset of variables of reflecting its structure and/or semantics. One obvious choice of is the set of input variables of . In this section, we describe generation of a set whose variables form an internal cut of denoted as Cut. Procedure GenCut for generation of set Cut consisting of Size gates is shown in Figure 7. Set is formed from output variables of the cut gates.

The current cut is specified by . Set Gts is initialized with the output gate of circuit and Inps is originally empty. GenCut computes the depth of every gate of Gts. The depth of is set to 0. Set Gts is processed in a while loop (lines 5-15). In every iteration, a gate of the smallest depth is picked from Gts. Then GenCut removes gate from Gts and examines the fan-in gates of (lines 9-15). Let be a fan-in gate of that has not been seen yet and is not a primary input of . Then the depth of is set to that of plus 1 and is added to Gts. If is a primary input of it is added to Inps.

## Vi Applications Of Property-Checking Tests

Given a multi-output circuit , traditional testing is used to verify “as a whole”. In this paper, we describe generation of a test set meant for checking a particular property of specified by a single-output circuit . In this section, we present some applications of property-checking test sets.

### Vi-a Testing properties specified by similar circuits

Let be a single-output circuit and be a test set generated when proving . Let be a circuit that is similar to . (For instance, can specify a property of a circuit whereas specifies the same property after a modification of .) Then one can use to verify if . Since is generated for a similar circuit , there is a good chance that it contains a counterexample to , if any. (Of course, the fact that evaluates to 0 for all tests of does not mean that even if is a CTS for ). In Subsection VII-B, we give experimental evidence supporting the observation above.

Assuming that was proved formally, checking if holds can be verified formally too. So applying tests of to can be viewed as a “light” verification procedure for exposing bugs. On the other hand, one can re-use test in situations where the necessity to apply a formal tool is overlooked or formal methods are not powerful enough. Let specify a property of a component of a design . Suppose that this component is modified under assumption that preserving is not necessary any more. By applying to one can invoke behaviors that break and expose a bug in , if any, caused by ignoring . If is a large design, finding such a bug by formal verification may not be possible.

### Vi-B Verification of corner cases

Let be a single-output subcircuit of circuit as shown in Figure 8. For the sake of simplicity we consider here the case where the set of input variables of is a subset of the set of input variables of . (The technique below can also be applied when input variables of are internal variables of .) Suppose evaluates, say, to value 0 much more frequently then to 1. Then one can view an input assignment of for which evaluates to 1 as specifying a “corner case” i.e. a rare event. Hitting such a corner case by a random test can be very hard. This issue can be addressed by using a coverage metric that requires setting the value of to both 0 and 1. (The task of finding a test for which evaluates to 1 can be solved, for instance, by a SAT-solver.) The problem however is that hitting a corner case only once may be insufficient.

One can increase the frequency of hitting the corner case above as follows. Let be a miter of circuits and (see Figure 9) i.e. a circuit that evaluates to 1 iff and are functionally inequivalent. Let and be two copies of circuit . So holds. Let test set be extracted from an SSA built for a projection of on a set . Set can be viewed as a result of “squeezing” the truth table of . Since this truth table is dominated by input assignments for which evaluates to 0, this part of the truth table is reduced the most. So, one can expect that the ratio of tests of for which evaluates to 1 is higher than in the truth table of . In Subsection VII-C, we substantiate this intuition experimentally. One can easily extend an assignment of to an assignment to e.g. by randomly assigning values to the variables of .

### Vi-C Dealing with incomplete specifications

One can use property-checking tests to mitigate the problem of incomplete specifications. By running tests generated for an incomplete set of properties of , one can expose bugs overlooked due to missing some properties. An important special case of this problem is as follows. Let be a property of that holds. Assume that the correctness of requires proving a slightly different property that is not true. By running a test set built for property , one may expose a bug overlooked in formal verification due to proving instead of . In Subsection VII-D, we illustrate the idea above experimentally.

### Vi-D Testing sequential circuits

There are a few ways to apply property-checking tests meant for combinational circuits to verification of sequential circuits. Here is one of them based on bounded model checking . Let be a sequential circuit and be a property of . Let be a circuit such that holds iff is true for time frames. Circuit is obtained by unrolling times and adding logic specifying property . Set consists of the subset specifying the state variables of in the first time frame and subset specifying the combinational input variables of in time frames.

Having constructed , one can build CTSs, CTSas and CTSaas for testing property of . The only difference here from the problem we have considered so far is as follows. Circuit starts in a state satisfying some formula that specifies the initial states. So, one needs to check if holds only for the assignments to satisfying . A test here is an assignment where is an initial state and , is an assignment to the combinational input variables of -th time frame. Given a test, one can easily compute the corresponding sequence of states of . In Subsection VII-D, we give an example of building an CTSaa for a sequential circuit.

## Vii Experiments

In this section, we describe experiments with property-checking tests (PCT) generated by procedure GenPCT shown in Figure 10. GenPCT accepts a single-output circuit and outputs a set of tests . (For the sake of simplicity, we assume here that holds.) GenPCT starts with generating formula and a set of variables . Then it calls (see Fig. 4) to compute an SSA of formula describing a projection of circuit on . If does not depend on a variable , all assignments of have the same value of . Procedure Diversify randomizes the value of in the assignments of . Finally, BldTests uses to extract a test set for circuit . If holds (where is the set of input variables of ), BldTests outputs all the different assignments to present in assignments of . Otherwise, BldTests calls procedure GenTests (see Fig. 6).

If , then is itself and GenPCT produces a CTS of . Otherwise, according to definitions of Subsection V-A, GenPCT generates a CTSa (if ) or CTSaa (if ).

In the following subsections, we describe results of four experiments. In the first three experiments we used circuits specifying next state functions of latches of HWMCC-10 benchmarks. (The motivation was to use realistic circuits.) In our implementation of , as a SAT-solver, we used Minisat 2.0 [6, 17]. We also employed Minisat to run simulation. To compute the output value of under test , we added unit clauses specifying to formula and checked its satisfiability.

### Vii-a Comparing CTSs, CTSas and CTSaas

The objective of the first experiment was to give examples of circuits with non-trivial CTSs and compare the efficiency of computing CTSs, CTSas and CTSaas. In this experiment, was a miter specifying equivalence checking of circuits and (see Figure 9). was obtained from by optimizing the latter with ABC .

The results of the first experiment are shown in Table I. The first two columns specify an HWMCC-10 benchmark and its latch whose next state function was used as . The next two columns give the number of input variables and that of gates in the miter . The following pair of columns describe computing a CTS for . The first column of this pair gives the size of the SSA found by GenPCT in thousands. The number of tests in the set extracted from is shown in the parentheses in thousands. The second column of this pair gives the run time of GenPCT in seconds.

The last four columns of Table I describe results of computing test sets for a projection of on a set of variables . The first column of this group shows if CTSa or CTSaa was computed whereas the next column gives the size of . The third column of this group provides the size of SSA and the test set extracted from (in parentheses). Both sizes are given in thousands. The last column shows the run time of GenPCT. For the first five examples, we used a projection of on , thus constructing a CTSa of . For the last four examples we computed a projection of on an internal cut (see Subsection V-C) thus generating a CTSaa of . GenPCT was called with parameter set to 5 (see Fig. 6 and 10).

For the first three examples, GenPCT managed to build non-trivial CTSs that are smaller than . For instance, the trivial CTS for example bob3 consists of =16,384 tests, whereas GenPCT found a CTS of 2,004 tests. (So, to prove and equivalent it suffices to run 2,004 out of 16,384 tests.) For the other examples, GenPCT failed to build a non-trivial CTS due to exceeding the memory limit (1.5 Gbytes). On the other hand, GenPCT built a CTSa or CTSaa for all nine examples of Table I. Note, however, that CTSas give only a moderate improvement over CTSs. For the last four examples GenPCT failed to compute an CTSa of due to memory overflow whereas it had no problem computing an CTSaa of . So CTSaas can be computed efficiently even for large circuits. Further, we show that CTSaas are also very effective.

### Vii-B Re-using property-checking tests to detect bugs

In the second experiment, we employed the idea of re-using property-checking tests (see Subsection VI-A) to verify relaxed equivalence . Let circuit be obtained from circuit by applying a set of changes . Regular equivalence of and means that these circuits produce the same output assignment for the same input assignment. Relaxed equivalence requires only that the difference between output assignments of and is in a specified range. So regular equivalence implies relaxed one and hence the latter is a weaker property than the former. Intuitively, this makes relaxed equivalence harder for testing (because the space of buggy behaviors is smaller).

In this experiment, we compared two-output circuits and . Namely we checked property that holds where and specify the outputs of and respectively. Property states that the Hamming distance between the output assignments produced by and for the same input assignment is less or equal to 1. Circuit was extracted from the transition relation of an HWMCC-10 benchmark. Circuit was obtained from by making changes that broke property .

Let denote a circuit specifying property . In the experiment, we tested using three approaches. The first approach was to apply tests generated to detect stuck-at faults (SAF in . We used SAF tests as an example of a test set driven by a coverage metric101010Note that SAF tests are stronger than tests satisfying traditional coverage metrics e.g. those used in software. In addition to exciting an event in , a SAF test must propagate the effect of this event to a primary output of . The second approach was random testing of . In the third approach we did the following. First, we built a CTSaa for circuit specifying property for the case where and hence was identical to . (Obviously, holds.) Then we re-used this CTSaa to verify circuit for the case .