On Verifying Designs With Incomplete Specification

04/20/2020 βˆ™ by Eugene Goldberg, et al. βˆ™ 0 βˆ™

Incompleteness of a specification 𝑆𝑝𝑒𝑐 creates two problems. First, an implementation πΌπ‘šπ‘π‘™ of 𝑆𝑝𝑒𝑐 may have some π‘’π‘›π‘€π‘Žπ‘›π‘‘π‘’π‘‘ properties that 𝑆𝑝𝑒𝑐 does not forbid. Second, πΌπ‘šπ‘π‘™ may break some π‘‘π‘’π‘ π‘–π‘Ÿπ‘’π‘‘ properties that are not in 𝑆𝑝𝑒𝑐. In either case, 𝑆𝑝𝑒𝑐 fails to expose bugs of πΌπ‘šπ‘π‘™. In an earlier paper, we addressed the first problem above by a technique called Partial Quantifier Elimination (PQE). In contrast to complete QE, in PQE, one takes out of the scope of quantifiers only a small piece of the formula. We used PQE to generate properties of πΌπ‘šπ‘π‘™ i.e. those π‘π‘œπ‘›π‘ π‘–π‘ π‘‘π‘’π‘›π‘‘ with πΌπ‘šπ‘π‘™. Generation of an unwanted property means that πΌπ‘šπ‘π‘™ is buggy. In this paper, we address the second problem above by using PQE to generate false properties i.e those that are π‘–π‘›π‘π‘œπ‘›π‘ π‘–π‘ π‘‘π‘’π‘›π‘‘ with πΌπ‘šπ‘π‘™. Such properties are meant to imitate the missing properties of 𝑆𝑝𝑒𝑐 that are not satisfied by πΌπ‘šπ‘π‘™ (if any). A false property is generated by modifying a piece of a quantified formula describing 'the truth table' of πΌπ‘šπ‘π‘™ and taking this piece out of the scope of quantifiers. By modifying different pieces of this formula one can generate a "structurally complete" set of false properties. By generating tests detecting false properties of πΌπ‘šπ‘π‘™ one produces a high quality test set. We apply our approach to verification of combinational and sequential circuits.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

One of the drawbacks of formal verification is that the set of properties describing a design usually does not specify the latter completely. Incompleteness of a specification Spec creates two problems. First, an implementation Impl of Spec may have some unwanted properties that Spec does not forbid. Second, Impl may break some desired properties that are not in Spec. In either case, Spec fails to expose bugs of Impl. In testing, the incompleteness of verification is addressed by using a set of tests that is complete structurally rather than functionally. Structural completeness is achieved by probing every piece of the design under test.

InΒ [5], we used the idea of structural completeness to attack the first problem above. The idea was to use a technique called partial quantifier elimination (PQE) to generation properties consistent with Impl. In contrast to complete quantifier elimination, PQE takes out of the scope of quantifiers only a small piece of formula. A property of Impl is produced by applying PQE to formula defining β€œthe truth table” of Impl. Here describes the functionality of Impl and are sets of external and internal variables of Impl respectively. If is not implied by Spec, then the latter is incomplete. If describes an unwanted property of Impl, the latter is buggy. Otherwise, a new property is added to Spec to make it imply . By taking different pieces of out of the scope of quantifiers, one can build a specification that is structurally complete.

In this paper, we continue this line of research by addressing the second problem above. Namely, we use PQE to generate false properties i.e. those inconsistent with Impl. They are meant to imitate the missing properties of Spec not satisfied by Impl (if any). Tests breaking a false property may expose a bug that was not discovered due to Spec’s lacking a property not satisfied by Impl. A false property is generated in two steps. First, a new formula is obtained from by a slight modification. Then the modified part is taken out of the scope quantifiers from . If and are not logically equivalent, this produces a property that is implied by but not by . By modifying different parts of one can generate a β€œstructurally complete” set of false properties. By generating tests breaking these properties one can build a high quality test.

Our contribution is as follows. First, we show that one can use PQE to generate false properties of Impl for combinational circuits. Second, we describe an algorithm that, given a combinational circuit, forms a structurally complete set of false properties of this circuit. Third, we extend our approach to sequential circuits. Fourth, to show the high quality of tests generated off false properties we relate the former to tests detecting stuck-at faults.

The main body111Some additional information is given in the appendix. of this paper is structured as follows. Basic definitions are given in SectionΒ II. In SectionΒ III, we describe generation of false properties for combinational circuits. A procedure for building a structurally-complete set of false properties is given in SectionΒ IV. SectionΒ V extends our approach to sequential circuits by showing how one can generate false safety properties. We relate our approach to fault/mutation detection in SectionΒ VI. Finally, we make some conclusions in SectionΒ VII.

Ii Basic Definitions

In this paper, we consider only propositional formulas. We assume that every formula is in conjunctive normal form (CNF). A clause is a disjunction of literals (where a literal of a Boolean variable is either itself or its negation ). So a CNF formula is a conjunction of clauses: . We also consider as the set of clauses .

Definition 1

Let be a set of variables. An assignment to is a mapping .

Definition 2

Let be a formula. denotes the set of variables of .

Definition 3

Let be a formula where are sets of variables. The Quantifier Elimination (QE) problem specified by is to find formula such that .

Definition 4

Let , be formulas where are sets of variables. The Partial QE (PQE) problem of taking out of the scope of quantifiers in is to find formula such that . Formula is called a solution to PQE.

Remark 1

Note that if is a solution to the PQE problem above and a clause is implied by alone, then is a solution too. So if all clauses of are implied by , then an empty set of clauses is a solution too (in this case, ).

Let be a combinational circuit where are sets of input, internal and output variables respectively. Let consist of gates . A formula specifying the functionality of can be built as where is a formula specifying gate . Formula is constructed as a conjunction of clauses falsified by the incorrect combinations of values assigned to . Then every assignment satisfying corresponds to a consistent assignment of values to and vice versa.

Example 1

Let be a 2-input AND gate specified by . Then formula is constructed as where , , . Here, the clause , for instance, is falsified by assignment inconsistent with the truth table of .

Iii Generation Of False Properties

In this section, we describe generation of false properties of a combination circuit by PQE. SubsectionΒ III-A explains our motivation for building false properties. SubsectionΒ III-B presents construction of false properties by PQE. In SubsectionΒ III-C, we describe generation of tests breaking these properties and argue that these tests are of very high quality.

Iii-a Motivation for building false properties

Let = be a specification of a combinational circuit. Here , are properties222The fact that is a property means that an implementation satisfying cannot output for an input if . of this circuit and and are the sets of input and output variables respectively. Let be an implementation of the specification where is the set of internal variables. Let be a formula describing (see SectionΒ II). Assume that satisfies all properties of i.e. , .

The specification is complete if it fully defines the input/output behavior of i.e. if . Suppose that is incomplete. As we mentioned in the introduction, this may lead to overlooking buggy input/output behaviors of . In this paper, we address this problem by generating false properties of . The latter are meant to imitate properties absent from that does not satisfy. The idea here is that tests breaking these false properties may expose the buggy behaviors mentioned above.

Iii-B Building false properties by PQE

Let be a formula obtained from by replacing a set of clauses with those of . Let . (So, .) Then the formula equals . Let and denote the β€œtruth tables” of and respectively. That is and . An informal requirement to is that it is unlikely to be implied by . (Otherwise, the technique we describe below cannot produce a false property.) One more requirement333If this requirement does not hold, may imply a non-empty set of clauses . Since , the formula is a trivial false property that just excludes some input assignments to . (So any input falsifying is a counterexample.) In reality, the requirement in question can be ignored if one also ignores the spurious false properties . to is that for every assignment there exists such that . (This is trivially true for because the latter is derived from specifying a circuit.)

Let be a solution to the PQE problem of taking out of the scope of quantifiers in . That is . The proposition below shows that is a false property of iff the truth tables and are incompatible i.e. . (If then can be viewed just as a relaxation of ).

Proposition 1

iff .

Proof: The β€œif” part. Let . Let (,) be an assignment to such that . (That is and .) Let =(,,) be the assignment describing the execution trace in under the input . Then satisfies and hence . Assume the contrary, i.e. . Then . Since satisfies , then == = =1 under assignment (,). So we have a contradiction.

The β€œonly if” part. Let . Let =(,,) be an assignment to that satisfies and falsifies (i.e. breaks ). Since , then . Since , then = = = = 0 under assignment (,). So, .

Iii-C Test generation

Let be a property of obtained as described in the previous subsection. We are interested in tests that break . A single test of that kind can be extracted from an assignment =(,,) that satisfies and falsifies thus breaking . Such an assignment can be found by running a SAT-solver on where is a clause of . The part of is a required test.

Intuitively, tests breaking should be of high quality. The reason is that they are supposed to break a property that is β€œalmost true”. Indeed, the proof of PropositionΒ 1 shows that an assignment (,) breaking exposes the difference in the input/output behavior specified by and . The latter is not a trivial task assuming that and are almost identical. To substantiate our intuition, in the appendix, we show that stuck-at fault tests (that are tests of a very high quality) are a special case of tests breaking false properties.

Iv A Complete Set Of False Properties

In the previous section, we introduced false properties as a means to deal with incompleteness of specification. In this section, we describe a procedure called that constructs a set of false properties that is structurally complete. Here we borrow an idea exploited in testing: if functional completeness is infeasible, run tests probing every design piece to reach structural completeness. Similarly, structural completeness of a set of false properties is achieved by generating properties relating to different parts of the design.

Iv-a Input/output parameters of the procedure

In this section, we continue the notation of the previous section. The pseudocode of is shown in FigureΒ 1. It accepts five parameters: ,,. The parameter is the set of properties of specification = that were too hard to prove/disprove. The parameter is an informal specification that is assumed to be complete444One can view as a replacement for the truth table. The role of such a replacement can be played, for instance, by the designer.. The parameter denotes a combinational circuit implementing the specification . The parameter is a formula describing the functionality of . Finally, the parameter is the set of internal variables of .

{
1
2
3
4 while {
5
6
7
8
9
10
11 if () continue
12
13 if
14 return()
15 if
16 return()
17 }
18 return(,,)}
Fig. 1: The procedure

has three output parameters: . The parameter denotes a test exposing a bug of (if any). The parameter consists of tests generated by that has identified any bug. (These tests may still be of value e.g. for regression testing). The parameter denotes the set of false properties generated by .

Iv-B The while loop of the procedure

The main body of consists of a while loop (lines 4-17). This loop is controlled by the set Gates consisting of gates of . Originally, Gates is set to the set of all gates of (line 3). starts an iteration of the loop by extracting a gate of Gates (lines 5-6). Then computes formula that replaces the clauses of (describing the gate ) in formula (line 7). After that, calls a PQE solver to find a formula such that where . The formula above represents a property of that is supposed to be false555Note that any subset of clauses of is a property as well. So, to decrease the complexity of PQE-solving, one can stop it when a threshold number of clauses is generated. Moreover, from the viewpoint of test generation, one can stop PQE as soon as a clause not implied by is generated..

checks if is indeed a false property by running a SAT-solver that looks for an assignment breaking (line 10). If this SAT-solver fails to find such an assignment, is a true property. In this case, starts a new iteration (line 11). Otherwise, the SAT-solver returns a test Tst and is added to as a new false property. Then checks if Tst breaks an unproved property of . If it does, then terminates (lines 13-14). After that, checks if Tst violates the informal specification . If so, then terminates (line 15-16). Finally, adds Tst to and starts a new iteration.

V Extension To Sequential Circuits

In this section, we extend our approach to sequential circuits. SubsectionΒ V-A provides some relevant definitions. In SubsectionΒ V-B, we give a high-level view of building a structurally complete set of false properties for a sequential circuit (in terms of safety properties). Finally, SubsectionΒ V-C describes generation of false safety properties.

V-a Some relevant definitions

Let be a sequential circuit. Here denote input and internal combinational variables respectively and denote the present and next state variables respectively. Let be a formula describing the circuit . ( is built for in the same manner as for a combinational circuit , see SectionΒ II.) Let be a formula specifying the initial states of . Let denote i.e. the transition relation of .

A state is an assignment to . Any formula is called a safety property for . A state is called a -state if . A state is called reachable in transitions (or in -th time frame) if there is a sequence of states ,…, such that is an -state, for and =.

We will denote the reachability diameter of with initial states as . That is if , every state of is reachable from -states in at most transitions. We will denote as a formula specifying the set of states of reachable from -states in transitions. We will denote as a formula specifying all states of reachable from -states. A property holds for with initial states , if no -state is reachable from an -state. Otherwise, there is a sequence of states called a counterexample that reaches a -state.

V-B High-level view

In this paper, we consider a specification of the sequential circuit above in terms of safety properties. So, when we say a specification property of we mean a safety property. Let denote where , is the formula in -th time frame i.e. expressed in terms of sets of variables . Formula can be computed by QE on formula . Here and . If , then is also specifying all states of reachable from -states.

Let be a set of properties forming a specification of a sequential circuit with initial states defined by . Let a sequential circuit be an implementation of the specification . So every property , holds for and i.e. , . Verifying the completeness of reduces to checking if . Assume that proving this implication is hard or it does not hold. If is incomplete, may be buggy for two reasons mentioned in the introduction. In particular, may falsify a property absent from . (This means that there is a reachable state of that satisfies all properties of and is supposed to be unreachable.)

One can deal with the problem above like it was done in the case of combinational circuits. Namely, one can use PQE to build false properties that are supposed to imitate properties that the specification has missed. By building counterexamples one generates β€œinteresting” reachable states. If one of these reachable states is supposed to be unreachable, has a bug.

V-C Generation of false properties

False properties of a sequential circuit can be generated as follows. Recall that formula specifies (see the previous subsection). Here and . Let be a (small) subset of clauses of . Let be a set of clauses meant to replace . We assume that satisfies two requirements similar to those mentioned in SubsectionΒ III-B. (In particular, we impose an informal requirement that is unlikely to be implied by .)

Let be a solution to the problem of taking out of the scope of quantifiers in where . That is . By definition, is a property (as a predicate depending only on variables of ). To show that is a false property one needs to find an assignment breaking . If such an assignment exists, there is a counterexample proving that a -state is reachable in transitions. In this case, is indeed a false property.

Using the idea above and a procedure similar to that of SectionΒ IV, one can build a structurally complete set of false safety properties.

Vi Our Approach And Fault/Mutation Detection

Generation of tests breaking false properties is similar to fault/mutation detection. In manufacturing testing, one generates tests detecting faults of a predefined setΒ [2, 7]. Often, these faults (e.g. stuck-at faults) do not simulate real defects but rather model logical errors. In software verification, one of old techniques gaining its popularity is mutation testingΒ [3, 1]. The idea here is to introduce code mutations (e.g. simulating common programmer mistakes) to check the quality of an existing test suite or to generate new tests.

Our approach has three potential advantages. First, PQE solving introduces a new way to generate tests detecting faults/mutations. (In the appendix, we give an example of using PQE for stuck-at fault testing.) The appeal of PQE here is that it can take into account subtle structural properties like unobservability. So PQE-solvers can potentially have better scalability than tools based purely on identifying logical inconsistencies (also known as conflicts).

The second advantage of our approach is that it transforms a fault/mutation into a property i.e. into a semantic notion. This has numerous benefits. One of them is that a false property specifies a large number of tests (rather than a single test). Suppose, for instance, that we need to find a single test detecting faults and of a circuit. An obvious problem here is that a test detecting one fault may not detect the other. In our approach, and are cast as false properties and . To break or one needs to come up with a test satisfying or . To break both properties one just needs to find a test satisfying (if any).

Third, our approach can be applied to abstract formulas that may not even describe circuits or programs. So, in a sense, the machinery of false properties can be viewed as a generalization of fault/mutation detection.

Vii Conclusions

Having an incomplete specification may lead to a buggy implementation. One of the problems here is that this implementation may not satisfy a property omitted in the specification. We address this problem by generating false properties i.e. those that are not consistent with the implementation. The idea here is that a test breaking a false property may also expose a bug in the implementation. False properties are generated by a technique called partial quantifier elimination (PQE).

Our three conclusions are as follows. First, the machinery of false properties can be applied to verification of combinational and sequential circuits. The efficiency of this machinery depends on that of PQE solving. So developing powerful PQE algorithms is of great importance. Second, the machinery of false properties can be viewed as a generalization of fault/mutation detection. On one hand, this implies that tests breaking false properties are of high quality. On the other hand, this means that the machinery of false properties can be applied to abstract formulas. Third, by generating properties whose falsehood is caused by different parts of the design, one generates a β€œstructurally complete” set of false properties. Using tests that break all properties of this set can significantly increase the quality of testing.

References

  • [1] P. Ammann and J.Offutt (2008) Introduction to software testing. Cambridge University Press, USA. Cited by: Β§VI.
  • [2] M. Breuer and A. Friedman (1994) Digital systems testing and testable design. John Wiley & Sons. Cited by: Β§VI.
  • [3] A. Budd (1980) Mutation analysis of program test data. Ph.D. Thesis, Yale University, USA. Cited by: Β§VI.
  • [4] E. Goldberg and P. Manolios (2013) Quantifier elimination via clause redundancy. In FMCAD-13, pp.Β 85–92. Cited by: Β§-D.
  • [5] E. Goldberg (2020) Generation of a complete set of properties. Technical report Technical Report arXiv:2004.05853 [cs.LO]. Cited by: Β§I.
  • [6] E. Goldberg (2020) Partial quantifier elimination by certificate clauses. Technical report Technical Report arXiv:2003.09667 [cs.LO]. Cited by: Β§-D.
  • [7] T. Juntilla and I.NiemelΓ€ (2009) Non-Clausal SAT and ATPG. In Handbook of Satisfiability, Vol. 185, pp.Β 655–694. Cited by: Β§VI.
  • [8] T. Larrabee (1992) Test pattern generation using boolean satisfiability. IEEE Transactions on Computer-Aided Design 11, pp.Β 4–15. Cited by: Β§-D.
  • [9] J. Marques-Silva and K. Sakallah (1996) GRASP – a new search algorithm for satisfiability. In ICCAD-96, pp.Β 220–227. Cited by: Β§-D.
  • [10] M. Moskewicz, C. Madigan, Y. Zhao, L. Zhang, and S. Malik (2001) Chaff: engineering an efficient sat solver. In DAC-01, New York, NY, USA, pp.Β 530–535. External Links: ISBN 1-58113-297-2 Cited by: Β§-D.
  • [11] J. Roth (1966) Diagnosis of automata failures: a calculus and a method. IBM Journal of Research and Development 10 (4), pp.Β 278–291. Cited by: Β§-D.