Efficient Verification of Multi-Property Designs (The Benefit of Wrong Assumptions) (Extended Version)

11/15/2017 ∙ by Eugene Goldberg, et al. ∙ 0

We consider the problem of efficiently checking a set of safety properties P1,....,Pk of one design. We introduce a new approach called JA-verification where JA stands for "Just-Assume" (as opposed to "assume-guarantee"). In this approach, when proving property Pi, one assumes that every property Pj for j!=i holds. The process of proving properties either results in showing that P1,....,Pk hold without any assumptions or finding a "debugging set" of properties. The latter identifies a subset of failed properties that cause failure of other properties. The design behaviors that cause the properties in the debugging set to fail must be fixed first. Importantly, in our approach, there is no need to prove the assumptions used. We describe the theory behind our approach and report experimental results that demonstrate substantial gains in performance, especially in the cases where a small debugging set exists.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The advent of powerful model checkers based on Sat [1, 2, 3, 4] has created a new wave of research in property checking. This research has been mostly focused on algorithms that verify a single property for a given design. However, in practice, engineers write many properties for one design (sometimes hundreds and even thousands). This demands efficient and scalable techniques for automatic verification of multiple properties for one design.

More specifically, the problem we address is as follows. We are given a transition relation and a set of initial states, which specify the design. In addition, we are given a set of safety properties that are expected to hold. (In Section 5, we consider the case where some properties are expected to fail.) We want to check if every property holds. If not, we want to have an efficient way to identify failed properties that point to wrong design behaviors. (Thus, identification of all failed properties is not mandatory.) One way to solve this problem is to check whether the property holds. We will call the aggregate property. If  holds, then all properties are proven. Otherwise, the generated counterexample (Cex for short) identifies a subset of the failed properties, but no information is gained about the remaining properties. The latter can be verified by removing the failed properties from the set, and re-iterating the procedure with a new aggregate property.

In this paper, we study an alternative approach where the properties are verified separately. We will refer to this approach as separate verification as opposed to joint verification of a set of properties. We will highlight three main reasons for our interest in separate verification. First, we want to study multi-property verification in the context of an Ic3-like model checker [3]. Such a model checker will benefit from separate verification by generating proofs that take into account the specifics of each property. Besides, a property is a weaker version of the aggregate property . Thus, proving should be easier than . Second, each property is an over-approximation of the same set of reachable states. Therefore, inductive invariants of already proven properties can be re-used. The third reason is as follows. If all are true, there is a common proof of this fact, namely a proof that holds. However, if some properties fail, there may not be one universal Cex that explains all failures if the latter are property specific. Separate verification is more relevant in such a context.

In this paper, we introduce a version of separate verification called Ja-verification. Here, Ja stands for “Just-Assume”, as opposed to “assume-guarantee”. In Ja-verification, one proves property , assuming that every other property , for , holds, regardless whether it is true. We will call such a proof local as opposed to a global proof that is true where no assumptions are made. Ja-verification results in constructing the set of properties that failed locally (if any). We show that if the aggregate property fails, there is at least one property that fails both globally and locally. Thus, if all properties hold locally it also means they hold globally.

If holds locally, it either holds globally as well or every Cex that breaks fails some other property before this Cex fails . That is, a Cex for contains a shorter Cex for . This suggests that if holds locally, its failure (if any) is most likely caused by failures of other properties. For this reason, we call the set of properties that fail locally a debugging set. This debugging set of properties points to design behaviors that need to be fixed in the first place. The approach guarantees that the failure of a property from the debugging set is not preceded by a failure of any other property. Ja-verification constructs a debugging set as follows: when proving it assumes that all , for , hold even when some of them fail. Thus, even the wrong assumption that all , hold proves to be useful. For that reason, we use the term “Just Assume” to name our approach.

To improve the efficiency of Ja-verification, we exploit the fact that Ic3 proves a property by strengthening it to make this property inductive. Specifically, we show that the strengthening clauses generated by Ic3 can be re-used when making any other property inductive if the same transition relation and initial states are used. Thus, in Ja-verification, clauses generated to make inductive are re-used when proving , .

Our contribution is threefold. First, we describe a new method of multi-property verification called Ja-verification (see Section 4). It is based on the machinery of local proofs (Sections 2 and 3) and re-use of strengthening clauses (Section 6). Second, we show that Ja-verification generates Cexs only for a special subset of failed properties called a debugging set. This is very important since computing a Cex can be quite expensive (e.g., if a counter is involved). Third, we provide implementation details (Section 7) and experimental results showing the viability of Ja-verification (Sections 9, 10,11). In particular, in Section 11, we give evidence that Ja-verification facilitates parallel computing.

2 Local And Global Proofs

Separate verification is based on the assumption that, in general, proving is easier than because is a weaker property. In this section, we discuss how one can make a proof of a property simpler if this proof is only needed in the context of proving a stronger property.

2-a Definitions

We will denote the predicates of the transition relation and the initial states as and respectively. Here and are sets of present and next state variables respectively. An assignment to variables is called a state. We will refer to a state satisfying a predicate as a -state. A property is just a predicate . We will say that a -state (respectively -state) is a good (respectively bad) state. A property is called inductive with respect to if holds where a primed predicate symbol means that the predicate in question depends on next state variables .

We will call a sequence of states (,…,) a trace if is true for . We will call the trace above initialized if is an -state. Given a property where , a Cex is an initialized trace (,…,) where , are -states and is a -state. We will refer to a state transition system with initial states and transition relation as an -system. Given an -system, checking a property is to find a Cex for or to show that none exists.

2-B Hardness of proving strong and weak properties

Let and be two properties where is weaker than , i.e., . On the one hand, verification of should be easier because one needs to prove unreachability of a smaller set of bad states. On the other hand, can be inductive even if is not. In fact, the essence of Ic3 is to turn a non-inductive property into an inductive one by adding strengthening clauses. This makes the modified property easier to prove despite the fact that it is stronger from a logical point of view.

The reason for the paradox above is as follows. The set of traces one needs to consider to prove is not a subset of those one considers when proving . To prove , one needs to show that there is no initialized trace of -states leading to a -state. Thus, one does not consider traces where two -states occur. Proving is reduced to showing that there is no initialized trace of -states leading to a -state. Since , a -state is a -state as well. On the other hand, a -state can also be a -state. Thus, in contrast to the case when we prove , to prove one has to consider traces that may include two or more different -states.

We will refer to a regular proof of (where one shows that no initialized trace of -states leads to a -state) as a global one. In the next subsection, we discuss reducing the complexity of proving using the machinery of local proofs.

2-C Local proofs

The intuition behind local proofs is as follows. Suppose that one needs to prove a property as a step in proving a stronger property . Then it is reasonable to ignore traces that do not make sense from the viewpoint of proving . Proving locally in the context of , or just locally for short, is to show that there does not exist an initialized trace of -states (rather than -states) leading to a -state.

The importance of local proofs is twofold. First, to prove locally, one needs to consider only a subset of the traces to prove (because the set of -states is a subset of that of -states). Thus, in terms of the set of traces to consider, a weaker property becomes also “easier”. Second, as we show in Section 4, to prove the aggregate property , it suffices to prove all properties locally with respect to .

It is convenient to formulate the notion of a local proof in terms of a modified transition relation . We will call this modification the projection of onto property and denote it as . It is defined as follows.

  • , if is a -state.

  • if is a -state and .

  • if is a -state and .

Informally, is obtained from by excluding any transitions from a -state other than a transition to itself. Hence, a trace in -system cannot have two different -states. Thus, a local proof of with respect to property , as we introduced above, is just a regular proof with respect to . (In turn, proving globally is done with respect to .)

Proposition 1

Let be inductive with respect to transition relation . Then any property weaker than (i.e., ) is inductive with respect to .

Proof: Assume the contrary. That is is not inductive with respect to and hence does not hold. Then there is a transition (, ) such that

  • and

  • and (and hence )

Since , then as well. Since and , from definition of it follows that and . So (, ) is a transition from a -state to a -state allowed by . Then is not inductive with respect to . We have a contradiction.     QED

Proposition 1 states that in terms of proofs by induction, proving locally with respect to a stronger property is at most as hard as proving itself.

3 Local Proofs And Debugging

In this section, we explain how the machinery of local proofs can be used to address the following problem. Given a property  that failed, find a weaker property that is false as well and can be viewed as an explanation for failure of . The subtlety here is that not every failed property where can be viewed as a reason for why fails. We will refer to this problem as the debugging problem.

To address the debugging problem one first needs to clarify the relation between local and global proofs.

Proposition 2

Let .

  1. If property holds with respect to transition relation (i.e., globally), it also holds with respect to (i.e., locally).

The opposite is not true.

  1. If holds with respect to , it either holds with respect to or it fails with respect to and every Cex contains at least two -states and where .

Proof: A) Assume that does not hold with respect to . Then there is an initialized trace of -states leading to a -state. Since inherits all transitions of from -states, this trace is valid with respect to . Since , every -state is also a -state. So there is an initialized trace of -states leading to a -state that is valid with respect to . Hence does not hold with respect to and we have a contradiction.

B) Assume the contrary, i.e. holds with respect to and there is a Cex with respect to containing only one -state. (So neither holds globally nor every Cex contains at least two -states). The Cex above is also a Cex with respect to and hence fails locally. So we have a contradiction.     QED

Informally, Proposition 2 means that proving locally is “as good as” proving globally modulo Cexs that do not make sense from the viewpoint of proving . These Cexs have at least two -states.

One can use Proposition 2 for solving the debugging problem as follows. Suppose that does not hold locally. This means that there is a Cex of -states leading to a -state. Since , a -state is a -state as well. So this Cex is also a regular Cex for . In other words, the fact that fails locally means that can be viewed as a reason for failure of .

Suppose that holds locally. Assume that fails globally and (,…,) is a Cex where is a -state. From Proposition 2, it follows that this Cex has at least two -states. One of these states is (because ). Another -state is one of -states , . This means that failure of is not a reason for failure of . Indeed, in every Cex for , property fails before does.

Summarizing, if property fails (respectively holds) locally with respect to , failure of is a reason (respectively cannot be a reason) for failure of .

4 Ja-verification

In this section, we present a version of separate verification called “Just-Assume” or Ja-verification. As before, denotes the aggregate property and denotes the projection of onto (see Subsection 2-C). Since every property is a weaker version of , one can use the results of Sections 2 and 3 based on the machinery of local proofs.

We now provide a justification of proving weaker properties locally in the context of multi-property verification. By using the transition relation to prove , one essentially assumes that every property , holds. While this may not be the case, nevertheless it works for two reasons. The first reason is that if the aggregate property fails, there is a time frame where (and hence some property ) fails for the first time. Let this be time frame number . For every time frame number where , the assumption that every property , holds is true. Thus, if fails, there is at least one property (in our case ) that fails even with respect to .

Here is the second reason why assuming , works. To get some debugging information when proving property , one is interested in traces where fails before any other property does. By assuming , is true, one drops the traces where fails after some , has failed.

The propositions below formalize the relation between local proofs and multi-property verification.

Proposition 3

Property holds with respect to iff every , holds with respect to .

Proof: If part. Let , hold with respect to . Assume the contrary i.e. does not hold. Then there is a Cex (,…,) where are -states and is a -state. This means that falsifies some property . Since a -state is also a -state, the Cex above is an initialized trace of -states leading to a state. Hence does not hold and we have contradiction.

Only if part. Let hold with respect to . Assume the contrary i.e. a property does not hold. Then there is a Cex (,…,) where , are -states and is a -state. Since holds, is a -state as well. As far as states , are concerned one can have the following two situations.

  • All these states are -states as well. Then the Cex above breaks and we have a contradiction.

  • At least one state , is a -state. In this case, trace (,…,) is a Cex breaking and we have a contradiction.   QED

Proposition 4

Property holds with respect to iff holds with respect to .

Proof: Any trace of -states leading to a -state is valid both with respect to and . So if fails with respect to it also does with respect to and vice versa.     QED

Proposition 5

Property holds with respect to iff every , holds with respect to .

Proof: From Proposition 4 it follows that holds with respect to iff it holds with respect to . After replacing with in Proposition 3, one concludes that holds with respect to iff every , holds with respect to .     QED

We will refer to the subset of that consists of properties that fail with respect to , i.e., locally as a debugging set. The following proposition justifies this name.

Proposition 6

Let the aggregate property fail. Let denote the debugging set of properties. Then the failure of properties of is the reason for the failure of in the following sense. For each Cex (,…,) for property , the state that falsifies also falsifies at least one property .

Proof: Assume the contrary, i.e., there is a Cex for such that falsify only properties of that are not in . Let be a property falsified by . Since this Cex consists of -states leading to a -state, then fails locally and so it is in . Thus, we have a contradiction.     QED

basicstyle = , moredelim=**[is][]  ,

Example 1

The Verilog code below gives an example of an 8-bit counter. This counter increments its value every time the enable signal is true. Once the counter reaches the value of rval it resets its value to 0. We also want to reset the counter when signal req is true (regardless of the current value of the counter). However, the code contains a buggy line (marked in blue), which prohibits a reset only unless req is true.

module counter(enable,clk,req);
   parameter  rval = 1 << 7;
   input      enable, clk, req;
   reg [7:0]  val;
   wire       reset;
   initial val = 0;
   ~assign reset = ((val == rval) && req); ~
   always @(posedge clk) begin
      if (enable) begin
         if (reset) val = 0;
         else val = val+1;
      end
   end
   P0: assert property (req == 1);
   P1: assert property (val <= rval);
endmodule

Let us consider verification of properties and specified by the last two lines of the module counter. Property fails globally in every time frame because req is an input variable taking values 0 and 1. Property fails globally due to the bug above. Note however, that only property fails locally. Indeed, fails even under assumption . However, becomes true if one assumes . The latter means that and so the counter always resets on reaching value rval. So the debugging set consists only of . The fact that holds locally means that either is true globally or that any Cex failing first fails . The latter implies that the failure of is caused by incorrect handling of variable req.

#bits solving globally solving
Abc (bmc) Abc (pdr) locally
#time frames time #time frames time
8 128 0.3 s 10 0.1 s 0.01 s
12 2,048 723 s 51 1.7 s 0.02 s
14 118 9.9 s 0.02 s
16 269 113 s 0.02 s
18 315 1,278 s 0.02 s
20 0.02 s

TABLE I: Example with a counter. Time limit is 1 hour

Note that proving false globally is hard for a large counter because a Cex consists of all states of the counter from 0 to rval. On the contrary, proving true under assumption is trivial because is inductive under this assumption. In Table I, we compare proving properties and above globally and locally. The first column gives the size of the counter. The next four columns give the results of solving and globally by Abc, a mature tool developed at UC Berkeley [5]. The first pair of columns gives the results of Bounded Model Checking [1] (the largest number of used time frames and run time). The next pair of columns provides results of Pdr (i.e., Ic3). Finally, we give the results of solving and locally by our tool (see Section 7).

The results show that bounded model checking soon becomes impractical, as the number of time frames increases exponentially. Abc’s Pdr solves more cases, but to generate a Cex, it has to consider a quickly increasing number of time frames as well. For Ja-verification, the size of the counter has no influence on the run time. While the counter is a purely synthetic example, in practice, one often has to find so-called deep counterexamples. A system with complex inner state might require a long sequence of steps to reach a buggy state.

5 Handling Properties Expected To Fail

When proving properties , in Ja-verification, as introduced in Section 4, one excludes the traces where a property , fails before does. This is based on the assumption that the properties that are the first to fail indicate design behaviors that need to be fixed first. However, this assumption is unreasonable when a property that fails before is Expected To Fail (Etf). For instance, to ensure that a state is reachable, one may formulate an Etf property where is a -state. In this case, excluding the traces where fails before is a mistake.

One can easily extend Ja-verification to handle Etf properties as follows. Suppose that our objective is to prove every property that is Expected To Hold (Eth). In addition, for every Etf property we want to find a Cex that does not break any Eth property. Then, to solve , locally one assumes that every Eth , is true. Thus, we exclude the traces where Eth properties fail before , even if the latter is an Etf property.

6 Ic3 And Clause Re-using

So far, we discussed the machinery of local proofs without specifying the algorithm used to prove a property. In this section, we describe an optimization technique applicable if property checking is performed by Ic3 [3]. The essence of this technique is to re-use strengthening clauses generated by Ic3 for property to strengthen another property , . Before describing clause re-using, we give a high-level view of Ic3.

6-a Brief description of Ic3

Let be a property of an -system where . If holds, there always exists a predicate such that is inductive with respect to . Then . (Recall that a primed predicate symbol means that the predicate in question depends on next state variables .) The fact that is inductive implies that is an over-approximation of the set of states reachable in the -system in question. Therefore, for every state reachable in -system, .

Let denote i.e property strengthened by . In Ic3, formulas are represented in conjunctive normal form and the predicate is constructed as a set of clauses (disjunctions of literals). Let denote the set of states reachable from -states in at most transitions. To construct the formula , Ic3 builds a sequence of formulas where and specifies an over-approximation of . That is, if a state is in , then satisfies , i.e., . A formula , is initialized with . Then is strengthened by adding so called inductive clauses. The objective of this strengthening is to exclude the -states from which a bad state is reachable in one transition. The exclusion of an -state may require excluding -states, (by adding inductive clauses to ) from which a bad state can be reached in transitions. If becomes inductive for some value of , property holds. Clause is called inductive relative to if and hold. In this case, every satisfies and so is still an over-approximation of .

6-B Re-using strengthening clauses

The idea of re-using strengthening clauses is based on the following observation. Suppose that is a set of clauses which makes inductive in the -system. This means that is an over-approximation of the set of all states reachable in the -system. Hence a state satisfies , for any value . Suppose one needs to prove some other property . Then, when constructing a formula over-approximating , one can initialize this formula with , rather than with .

6-C State lifting in Ic3

In this subsection, we briefly describe an important technique of Ic3 called state lifting [4, 6]. This description is used in Section 7 when explaining how local proofs are implemented. Let be a state from which a bad state is reachable. Ic3 tries to exclude by generating an inductive clause falsified by . If such a clause cannot be built immediately, Ic3 tries to exclude every state of the previous time frame from which there is a transition to state . The number of those states can be very large. So, Ic3 tries to “lift” to a cube that not only contains the state itself, but many other states that are one transition away from state . Subsequently, Ic3 tries to exclude all states of in one shot. Informally, the larger , the greater the performance boost by lifting.

7 Implementation

We use a version of Ic3 developed in our research group. We will refer to it as Ic3-db where “db” stands for Diffblue. Ic3-db uses the front-end of ebmc [7]. We will refer to our implementation of Ja-verification based on Ic3-db as Ja-ver. The latter is a Perl script that calls Ic3-db in a loop for proving individual properties.

7-a Using properties as constraints

Let be the set of properties to be proved. Proving locally means showing that there is no initialized trace of -states that leads to a -state, where is the aggregate property . To guarantee that all present states satisfy , Ic3-db adds constraints to the transition relation that force , to be equal to 1. Adding constraints to affects the lifting procedure of Ic3 (see Subsection 6-C). The reason is that one needs to guarantee that all states of the cube obtained by lifting a state satisfy the constraints in question. In our case, all states of must be -states. Unfortunately, this can drastically decrease the size of and therefore reduce the effectiveness of lifting. For that reason, Ic3-db has an option to make the lifting procedure ignore the constraints forcing to be equal 1.

The relaxation of lifting above can lead to appearance of “spurious” Cexs that contain transitions from -states to other states. (This can occur only if does not hold). If a spurious Cex is generated when proving property , then Ic3-db is invoked again. This time the lifting procedure is forced to respect the constraints specified by .

7-B Implementation of clause re-using

The correctness of re-using strengthening clauses is discussed in Subsection 6-B. Assume that properties are processed in the order they are numbered. Let denote the strengthening of property , i.e., is inductive with respect to . Ja-ver maintains an external file clauseDB that collects strengthening clauses. Therefore, after making inductive, the clauses of are written to clauseDB.

When Ic3-db is invoked to prove , the clauses of are extracted from clauseDB. When proving , the formula over-approximating is initialized with (rather than with ). Let denote the strengthening clauses added to to make the latter inductive with respect to . These clauses are written to clauseDB (that already contains ). In general, when Ic3-db is invoked to prove , all clauses are extracted from clauseDB to be used in the proof.

8 Comparing local and global proofs

In Sections 910 and 11, we experimentally compare methods based on local and global proofs. In this section, we summarize the information we provided earlier to help better understand these experimental results. We will refer to methods proving properties locally (respectively globally) as a local (respectively global) approach.

Let denote the set of properties to verify. There are two cases where local and global approaches provide the same information. Assume one managed to prove every property of locally. In this case, every property of holds globally as well. Thus global and local approaches provide the same information here. Now assume that one proved false locally (and hence globally). Assume also that in a global approach one proved false and in the generated Cex property is the first to fail. Then, is “inadvertently” proved false not only globally but locally too. In this case, both local and global approaches provide the same information for .

Now consider the cases where local and global approaches provide different information for a failed property. Assume that one proved false globally and the generated Cex falsifies at least one property , before . Consider the following two cases. The first case is that one proves false locally. Then the local approach provides more information than its global counterpart because in addition to proving false is shows that is in the debugging set. The second case is that one proves true locally. In this situation, local and global approaches do not have a clear winner. On the one hand, the global approach gives more information by finding a Cex for . On the other hand, the local approach provides more information by showing that every Cex falsifying (if any) first falsifies some other property of . Note that from the viewpoint of debugging, the information provided by the local approach in the second case is more useful since is shown not to be in the debugging set.

Finally, lets us consider the following situation. Let be the subset of properties proved true locally and . A property is in if it is proved false locally or it is too hard to solve or it has not been tried yet. Since , the fact that holds locally does not mean that it holds globally as well. Thus, proving true globally provides more information than in the local approach. However, from the debugging point of view, proving true locally is almost as good as globally. This proof means that cannot fail “on its own”. So one first needs to focus on properties of , in particular, to fix the design behaviors causing property failures.

9 Ja-verification Versus Joint Verification

In this section, we experimentally compare Ja-verification and joint verification to show the viability of our approach to multi-property verification. We use benchmarks from the multi-property track of the Hwmcc-12 and 13 competitions. As we mentioned in Section 7, Ja-verification is implemented as a Perl script Ja-ver that calls Ic3-db to process individual properties sequentially. In this paper, we do not exploit the possibility to improve Ja-verification by processing properties in a particular order.111A rule of thumb here is to verify easier properties first to accumulate strengthening clauses and use them later for harder properties. Properties are verified in the order they are given in the design description. Joint verification is also implemented as a Perl script called Jnt-ver, where Ic3-db is called to verify the aggregate property . If  fails, the individual properties refuted by the generated Cex are reported false. Jnt-ver forms a new aggregate property by conjoining the properties that are unsolved yet and calls Ic3-db again. This continues until every property is solved.

Ideally, we would like to use designs with as many properties as possible. However, for a design with a very large number of properties, Ja-verification usually outperforms joint verification (see Subsection 9-A). This problem can be addressed by partitioning into smaller clusters of properties [8], which is beyond the scope of this paper. To make joint verification more competitive, in Subsections 9-B and 9-C, we picked sixteen designs (eight designs per subsection) that have less than a thousand properties. For these designs, we cross-checked the results of Ic3-db in joint verification with those reported by the latest version of Abc [5].222Joint verification is the natural mode of operation for Abc. However, in contrast to Jnt-ver, Abc does not re-start when a property is proved false and goes on with solving the remaining properties.

The time limit for joint verification was set to 10 hours. The time limit used by Ic3-db in Ja-verification to prove one property is indicated in the tables of results. If a property of a benchmark was not solved by Ic3-db, the time limit was added to the total time of solving this benchmark. Unfortunately, the Hwmcc competitions do not identify properties of multi-property benchmarks that are expected to fail, if any (see Section 5). So, in the experiments we just assumed that every property was expected to hold.

9-a A few designs with a large number of properties

In this subsection, we compare Ja-verification with joint verification on a few benchmarks that have a very large number of properties. The point we are making here is that for such benchmarks, Ja-verification is typically more robust than joint verification. One of the reasons for that is as follows. When one conjoins a set of very different properties joint verification may fail to prove the aggregate property even if all properties are simple individually. For instance, it may be the case that each property of this set depends on a small subset of state variables and hence can be easily proved separately. However, if different properties depend on different subsets of variables, the aggregate property depends on a large subset of state variables and becomes very hard to prove. Another reason for the poor performance of joint verification is that the presence of a few too-hard-to-solve properties can blow up the complexity of the aggregate property .

name #all #props Joint verification    Ja-verification
props tried #un- time time #un- time
solved limit solved
6s400 13,784 100 100 10 h 0.3 h 3 3,167 s
6s355 13,356 100 100 10 h 0.3 h 2 2,175 s
100 0 2,817 s 0.3 h 0 1,974 s
6s289 10,789 200 0 1,095 s 0.3 h 2 1.1 h
500 100 10 h 0.3 h 3 2.1 h
100 0 919 s 0.3 h 0 1,126 s
6s403 2,382 200 0 828 s 0.3 h 1 2,329 s
500 0 717 s 0.3 h 1 3,265 s


TABLE II: A few designs with a large number of properties

In this experiment, we used Ja-verification and joint verification to verify the first properties of a benchmark. (Joint verification was performed by Ic3-db). The results of the experiment are given in Table II. The value of is given in the third column. The first and second columns of this table provide the name of a benchmark and the total number of properties. For benchmarks 6s400 and 6s355, Ja-verification clearly outperformed joint verification. For benchmark 6s289 both Ja-verification and joint verification performed well for and . However, for Ja-verification outperformed joint verification. Benchmark 6s403 was the only one out of four where joint verification outperformed Ja-verification.

name #latch #pro Joint verification Ja-verification by Ic3-db
pert. Abc Ic3-db with clause re-use
#false #false time #false total
(#true) time (#true) time limit (#true) time
6s104 84,925 124 1 (0) 10 h 1 (0) mem 0.3 h 1 (123) 2.5 h
6s260 2,179 35 1 (0) 10 h 1 (0) 10 h 0.5 h 1 (34) 1,686 s
6s258 1,790 80 25 (0) 10 h 30 (0) 10 h 0.3 h 1 (72) 2.4 h
6s175 7,415 3 2 (0) 10 h 2 (0) 10 h 0.3 h 2 (1) 554 s
6s207 3,012 33 6 (0) 10 h 10 (0) 10 h 0.3 h 2 (31) 22 s
6s254 762 14 13 (1) 25 s 13 (1) 225 s 0.3 h 1 (13) 2 s
6s335 1,658 61 26 (35) 2 h 26 (35) 260 s 0.3 h 20 (41) 56 s
6s380 5,606 897 399 (0) 10 h 395 (0) 10 h 0.3 h 3 (894) 550 s
TABLE III: Designs with failed properties. Many properties of 6s258, 6s207, 6s254, 6s335, and 6s380 are false globally in joint verification but true locally in Ja-verification. ‘mem’ means running out of memory

9-B Designs with failing properties

In this section, we describe an experiment where we verified designs with failing properties. Our objective was to show that solving properties locally can be much more efficient than globally. The results are given in Table III. The first column provides the name of the benchmark. The second and third columns give the number of latches and properties, respectively. The next two pairs of columns provide the results of joint verification performed by Abc and Ic3-db. The first column of the pair gives the number of false and true properties that Abc or Ic3-db managed to solve within the time limit. The second column of the pair reports the amount of time taken by Abc or Ic3-db. The last three columns report data about Ja-verification: the time limit per property, the number of false and true properties solved within the time limit, and the total time taken by Ic3-db. In all tables of experimental sections, the run times that do not exceed one hour are given in seconds.

For all examples but 6s258, Ja-verification solved all properties locally. On the other hand, for many examples, in joint verification, only a small fraction of properties were solved by Ic3-db and Abc globally. Let us consider example 6s207 in more detail. Ja-verification solved all properties of 6s207 fairly quickly generating the debugging set of two properties. On the other hand, joint verification by Ic3-db proved that ten properties failed globally within 10 hours. Since Ja-verification showed that only two properties failed locally, eight out of those ten failed properties were true locally. Let be one of those eight properties. The Cex found for by joint verification first falsifies a property of the debugging set. Thus, we do not know if there is a Cex where fails before other properties. Ja-verification does not determine whether fails but guarantees that every Cex for (if any) first fails some other property.

name #latch #pro- Joint verification Ja-verification by Ic3-db
pert. with clause re-use
Abc Ic3-db time #un- total
time time limit solved time
6s124 6,748 630 10 h 2.9 h 0.8 h 0 1.9 h
6s135 2,307 340 123 s 335 s 0.8 h 0 746 s
6s139 16,230 120 4.7 h 1.7 h 2.8 h 2 6.5 h
6s256 3,141 5 10 h 602 s 2.8 h 1 2.9 h
bob12m09 285 85 1,692 s 930 s 0.8 h 0 784 s
6s407 11,379 371 1.3 h 3.4 h 0.8 h 0 2,077 s
6s273 15,544 42 1.8 s 325 s 0.8 h 0 290 s
6s275 3,196 673 334 s 1,154 s 0.8 h 0 1,611 s


TABLE IV: All properties are true

9-C Designs where all properties hold

In this subsection, we describe an experiment with eight designs where all properties were true. The results are given in Table IV. The first three columns are the same as in Table III. The next two columns give run times of Abc and Ic3-db in joint verification. The last three columns provide information about Ja-verification: time limit per property, number of unsolved properties and total run time. The best of the run times obtained in joint verification and Ja-verification based on Ic3-db is given in bold. In three cases, joint verification based on Abc was the fastest but we needed a comparison that uses a uniform setup.

Table IV shows that joint verification performed slightly better. In particular, for benchmarks 6s139 and 6s256, Ja-verification failed to solve some properties with the time limit of 2.8 hours. However, when we verified properties in an order different from the one of design description, both benchmarks were solved in time comparable with joint verification.

10 Studying Ja-verification In More Detail

10-a Comparison of local and global proofs

In this subsection, we describe an experiment where we compared separate verification with global and local proofs. Both versions of separate verification employed clause re-using. (Thus, separate verification with local proofs is Ja-verification). In Table V, we compare global and local proofs on benchmarks with failing properties from Table III. The first two columns provide the name of a benchmark and its number of properties. The next two columns specify the performance of separate verification with global proofs. The first column of the two shows how many properties were solved within a 10-hour time limit for the entire benchmark. The time limit per property was the same as in Table III. The second column gives the overall time for a benchmark. The last two columns provide the same information for separate verification with local proofs. Table V shows that separate verification with local proofs dramatically outperforms the one with global proofs.

name #properties global proofs local proofs
#unsolved time #unsolved time
6s104 124 123 10 h 0 2.5 h
6s260 35 36 10 h 0 1,686 s
6s258 80 70 10 h 7 2.4 h
6s175 3 1 1,070 s 0 554 s
6s207 33 23 7.0 h 0 22 s
6s254 14 0 237 s 0 2 s
6s335 61 3 3,243 s 0 56 s
6s380 897 698 10 h 0 550 s



TABLE V: Separate verification with global and local proofs for examples of Table III. Time limit per property is the same as in Table III. The total time limit per benchmark is 10 hours

In Table VI, we compare global and local proofs on benchmarks from Table IV where all properties are true. The structure of Table VI is the same as that of Table V. Table VI demonstrates that both versions of separate verification show comparable performance. A noticeable difference is observed only on benchmarks 6s256 and 6s407. So one can conclude that it is more likely to see the effect of using local proofs on benchmarks with failed properties. On the other hand, the advantage of using local proofs may become more pronounced even on correct designs if the number of properties is large (see Section 11).

name #properties global proofs local proofs
#unsolved time #unsolved time
6s124 630 0 2.1 h 0 1.9 h
6s135 340 0 764 s 0 746 s
6s139 120 2 8.1 h 2 6.5 h
6s256 5 2 5.7 h 1 2.9 h
bob12m09 85 0 809 s 0 784 s
6s407 317 5 5.4 h 0 2,077 s
6s273 42 0 278 s 0 290 s
6s275 673 0 1.542 s 0 1,611 s


TABLE VI: Separate verification with global and local proofs for examples of Table IV. Time limit per property is the same as in Table IV

10-B Benefit of clause re-using

name #properties without clause re-use with clause re-use
#unsolved time #unsolved time
6s124 630 505 10 h 0 1.9 h
6s135 340 0 2.7 h 0 746 s
6s139 120 116 10 h 2 6.5 h
6s256 5 0 892 s 1 2.9 h
bob12m09 85 0 1.1 h 0 784 s
6s407 317 270 10 h 0 2,077 s
6s273 42 0 1,445 s 0 290 s
6s275 673 0 3,273 s 0 1,611 s


TABLE VII: Re-using strengthening clauses in Ja-verification. Time limit per property is the same as in Table IV. Verification of 6s124, 6s139, and 6s407 (without clause re-use) was aborted after 10 hours

To illustrate the benefit of re-using strengthening clauses, Table VII compares Ja-verification with and without re-using strengthening clauses on the examples of Table IV. Table VII shows that Ja-verification with re-using strengthening clauses significantly outperforms its counterpart. The only exception is 6s256 which has only five properties to check.

name #properties respecting prop. constr. ignoring prop. constr.
#unsolved time #unsolved time
6s104 124 0 2.5 h 0 2.5 h
6s260 35 2 1 h 0 1,686 s
6s258 80 0 69 s 7 2.4 h
6s175 3 0 294 s 0 554 s
6s207 33 0 33 s 0 22 s
6s254 14 0 2 s 0 2 s
6s335 61 0 120 s 0 56 s
6s380 897 0 878 s 0 550 s



TABLE VIII: Ja-verification with lifting respecting or ignoring property constraints for examples of Table III. Time limit per property is the same as in Table III

10-C Ja-verification and state-lifting

As we said in Subsection 7-A, Ic3-db proves property locally by treating all the properties , as constraints. We also mentioned that this may affect the state lifting procedure used by Ic3. In this subsection, we study this issue experimentally by considering two versions of Ic3-db. In the first version, property constraints are respected when lifting a state whereas in the second version these constraints are ignored. Respecting and ignoring property constraints means the following. Let be the cube obtained by lifting state . When proving property , the first version of Ic3-db guarantees that the states of satisfy , whereas this is, in general, not true for the second version.

name #properties respecting prop. constr. ignoring prop. constr.
#unsolved time #unsolved time
6s124 630 618 10 h 0 1.9 h
6s135 340 248 10 h 0 746 s
6s139 120 98 10 h 2 6.5 h
6s256 5 0 1,282 s 1 2.9 h
bob12m09 85 0 1,070 s 0 784 s
6s407 317 0 1.0 h 0 2,077 s
6s273 42 0 537 s 0 290 s
6s275 673 0 6.4 h 0 1,611 s


TABLE IX: Ja-verification with lifting respecting or ignoring property constraints for examples of Table IV. Time limit per property is the same as in Table IV. The total time limit per benchmark is 10 hours

Table VIII shows the results of both versions of Ic3-db on the failed designs of Table III. The first two columns give the name of a benchmark and the total number of properties. The next two columns show the results of the first version of Ic3-db. These columns give the number of unsolved properties and the total run time. The last two columns provide the same information for the second version. The results of Table VIII show that both versions of Ic3-db have comparable performance.

Table IX shows the results of both versions of Ic3-db on the designs of Table IV where all properties are true. The structure of this table is the same as that of Table VIII. Table IX shows that the first version is faster than the second version only on one benchmark. On the other benchmarks the second version outperforms the first version, sometimes quite dramatically.

11 Ja-Verification And Parallel Computing

Intuitively, Ja-verification can significantly benefit from parallel computing. In this section, we give the results of a simple experiment substantiating this intuition. Our interest in a discussion of Ja-verification in the context of parallel computing is based on the following two observations. Let denote the set of properties to prove. The first observation is that the larger , the easier proving locally due to growing number of constraints , . The second observation is that the larger , the smaller an inductive invariant for property . Hence, when proving different properties of locally, the exchange of information (in the form of strengthening clauses) reduces as grows. The observations above suggest that by proving properties in parallel one can significantly decrease verification time.

prop. global proof local proof
index #time frames time #time frames time
20 11 213 s 1 4.0 s
137 13 289 s 1 3.0 s
500 12 370 s 1 7.7 s
1,001 10 10 s 1 2.5 s
1,310 9 30 s 1 2.4 s
2,678 9 21 s 1 2.3 s
4,789 14 32 s 1 3.5 s
6,600 14 75 s 1 2.8 s
10,002 10 9.9 s 1 2.7 s
max 14 370 s 1 7.7 s
TABLE X: Verification of single properties of benchmark 6s289 (10,789 properties) using global and local proofs

In Table X, we report the results of a simple experiment where we randomly picked individual properties of benchmark 6s289 and proved them both locally and globally. (The total number of properties of 6s289 is 10,789.) The proofs were generated independently of each other i.e. there was no exchange of strengthening clauses. The index of selected property is shown in the first column of Table X. The next two columns provide information about the performance of Ic3-db when proving the selected property globally. The first column gives the number of time frames Ic3-db had to unfold. The second column of the two shows the run time taken by Ic3-db. The next pair of columns provide the same information when Ic3-db proved the selected property locally. Table X shows that proving the properties we tried locally was very easy. Assume that finding a local proof for each of the remaining properties of 6s289 takes a very small amount of time as well. Then, if one had, say, 10,789 processors to prove each property on a separate processor, verification would be finished in a matter of seconds.

12 Related Work

We found only a few references to research on multi-property verification. In [9], some modifications of Abc are presented that let it handle multi-property designs. In [10, 8], the idea of grouping similar properties and solving them together is introduced. The similarity of properties is decided based on design structure (e.g., properties with similar cones of influence are considered similar). The main difference of this approach from ours is that the latter is purely semantic. Thus, the optimizations of separate verification we consider (local proofs and re-using strengthening clauses) can be incorporated in any structure-aware approach. One further difference is that the idea of grouping favors correct designs. Grouping may not work well for designs with broken properties that fail for different reasons and thus have vastly different Cexs.

Assume-guarantee reasoning is an important method for compositional verification [11, 12]. It reduces verification of the whole system to checking properties of its components under some assumptions. To guarantee the correctness of verification one needs to prove these assumptions true. As we mentioned earlier, Ja-verification uses yet-unproven properties as assumptions without subsequent justification. This is achieved by our particular formulation of multi-property verification. Instead of proving or refuting every property, Ja-verification builds a subset of failed properties that are the first to break or proves that this subset is empty.

13 Conclusions

We consider the problem of verifying multiple properties of the same design. We make a case for separate verification where properties are proved one by one as opposed to joint verification where the aggregate property is used. Our approach is purely semantic, i.e., we do not rely on any structural features a design may or may not have.

We introduce a novel variant of separate verification called Ja-verification. Ja-verification checks if holds locally, i.e., under the assumption that all other properties are true. We show that if all properties hold locally, they also hold globally, i.e., without any assumptions. Instead of finding the set of all failed properties, Ja-verification identifies a “debugging” subset. The properties in the debugging subset highlight design behaviors that need to be fixed first, which can yield substantial time savings in the design-verification cycle.

We experimentally compare conventional joint verification and Ja-verification. We give examples of designs with failed properties where Ja-verification dramatically outperforms its counterpart, especially for designs where a small debugging set exists. For these designs, one needs to find only Cexs which are typically shallow. Computation of deeper Cexs for false properties that are not in is replaced with proving them true locally. Re-using inductive invariants generated for individual properties that are locally true significantly speeds up Ja-verification. In particular, for correct designs, it makes Ja-verification competitive with joint verification even for benchmarks that favor the latter.

References

  • [1] A. Biere, A. Cimatti, E. Clarke, M. Fujita, and Y. Zhu, “Symbolic model checking using SAT procedures instead of BDDs,” in DAC, 1999, pp. 317–320.
  • [2]

    K. L. Mcmillan, “Interpolation and SAT-based model checking,” in

    CAV.   Springer, 2003, pp. 1–13.
  • [3] A. Bradley, “SAT-based model checking without unrolling,” in VMCAI, 2011, pp. 70–87.
  • [4] N. Eén, A. Mishchenko, and R. Brayton, “Efficient implementation of property directed reachability,” in FMCAD, 2011.
  • [5] B. L. Synthesis and V. Group, “ABC: A system for sequential synthesis and verification,” 2017, http://www.eecs.berkeley.edu/alanmi/abc.
  • [6] H. Chockler, A. Ivrii, A. Matsliah, S. Moran, and Z. Nevo, “Incremental formal verification of hardware,” in FMCAD, 2011.
  • [7] R. Mukherjee, D. Kroening, and T. Melham, “Hardware verification using software analyzers,” in IEEE Computer Society Annual Symposium on VLSI.   IEEE, 2015, pp. 7–12.
  • [8] G. Cabodi and S. Nocco, “Optimized model checking of multiple properties,” in DATE, 2011.
  • [9] Http://people.eecs.berkeley.edu/alanmi/presentations/
    updating_engines00.ppt.
  • [10] P. Camurati, C. Loiacono, P. Pasini, D. Patti, and S. Quer, “To split or to group: from divide-and-conquer to sub-task sharing in verifying multiple properties,” in DIFTS, 2014.
  • [11] C. Jones, “Specification and design of (parallel) programs,” in IFIP 9th World Congress, 1983, pp. 321–332.
  • [12] A. Pnueli, “In transition from global to modular temporal reasoning about programs,” in Logic and Models of Conc. Sys., vol. 13, 1984, pp. 123–144.