Manthan: A Data Driven Approach for Boolean Function Synthesis

05/14/2020 ∙ by Priyanka Golia, et al. ∙ National University of Singapore Indian Institute of Technology Kanpur 0

Boolean functional synthesis is a fundamental problem in computer science with wide-ranging applications and has witnessed a surge of interest resulting in progressively improved techniques over the past decade. Despite intense algorithmic development, a large number of problems remain beyond the reach of the state of the art techniques. Motivated by the progress in machine learning, we propose Manthan, a novel data-driven approach to Boolean functional synthesis. Manthan views functional synthesis as a classification problem, relying on advances in constrained sampling for data generation, and advances in automated reasoning for a novel proof-guided refinement and provable verification. On an extensive and rigorous evaluation over 609 benchmarks, we demonstrate that Manthan significantly improves upon the current state of the art, solving 356 benchmarks in comparison to 280, which is the most solved by a state of the art technique; thereby, we demonstrate an increase of 76 benchmarks over the current state of the art. Furthermore, Manthan solves 60 benchmarks that none of the current state of the art techniques could solve. The significant performance improvements, along with our detailed analysis, highlights several interesting avenues of future work at the intersection of machine learning, constrained sampling, and automated reasoning.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Given an existentially quantified Boolean formula over the set of variables and

, the problem of Boolean functional synthesis is to compute a vector of Boolean functions, denoted by

, and referred to as Skolem function vector, such that . In the context of applications, the sets and are viewed as inputs and outputs, and the formula is viewed as a functional specification capturing the relationship between and , while the Skolem function vector allows one to determine the value of for the given by evaluating . The study of Boolean functional synthesis traces back to Boole [10], and over the decades, the problem has found applications in a wide variety of domains such as certified QBF solving [6, 7, 33, 38], automated program repair [24], program synthesis [43], and cryptography [32].

Theoretical investigations have demonstrated that there exist instances where Boolean functional synthesis takes super-polynomial time. On the other hand, practical applicability has necessitated the development of algorithms with progressively impressive scaling. The algorithmic progress for Boolean functional synthesis has been driven by a diverse set of techniques: (i) the usage of incremental determinization employing the several heuristics in state-of-the-art Conflict Driven Clause Learning (CDCL) solvers 

[38], (ii) usage of decomposition techniques employing the progress in knowledge compilation [3, 17, 25, 44], and (iii) Counter-Example Guided Abstraction Refinement (CEGAR)-based techniques relying on usage of SAT solvers as black boxes [1, 3, 2, 25]. While the state of the art techniques are capable of handling problems of complexity beyond the capability of tools a decade ago, the design of scalable algorithms capable of handling industrial problems remains the holy grail.

In this work, we take a step towards the above goal by proposing a novel approach, called , at the intersection of machine learning, constrained sampling, and automated reasoning. Motivated by the unprecedented advances in machine learning, we view the problem of functional synthesis through the lens of multi-class classification aided by the generation of the data via constrained sampling and employ automated reasoning to certify and refine the learned functions. To this end, the architecture of comprises of the following three novel techniques:

[topsep=0pt]

Data Generation

The state of the art machine learning techniques use training data represented as a set of samples where each sample consists of valuations to features and the corresponding label. In our context, we treat as the features and as labels. Unlike the standard setup of machine learning wherein for each assignment to , there is a unique label, i.e. assignment to , the relationship between and is captured by a relation and not necessarily a function. To this end, we design a weighted sampling strategy to generate a representative data set that can be fitted using a compactly sizedclassifier. The weighted sampling strategy, implemented using state of the constrained sampler, seeks to uniformly sample input variables () while biasing the valuations of output variables towards a particular value.

Dependency-Driven Classifier for Candidates

Given training data viewed as a valuation of features (X) and their corresponding labels (Y), a natural approach from machine learning perspective would be to perform multi-class classification to obtain , where is a symbolic representation of the learned classifier. Such an approach, however, can not ensure that

can be expressed as a vector of Boolean functions. To this end, we design a dependency aware classifier to construct a vector of decision trees corresponding to each

, wherein each decision tree is expressed as a Boolean function.

Proof-Guided Refinement

Since machine learning techniques often produce good but inexact approximations, we augment our method with automated reasoning techniques to verify the correctness of decision tree-based candidate Skolem functions. To this end, we perform a counterexample driven refinement approach for candidate Skolem functions.

To fully utilize the impressive test accuracy attained by machine learning models, we design a proof-guided refinement approach that seeks to identify and apply minor repairs to the candidate functions, in an iterative manner, until we converge to a provably correct Skolem function vector. In a departure from prior approaches utilizing the Shannon expansion and self-substitution, we first use a MaxSAT solver to determine potential repair candidates, and employ unsatisfiability cores obtained from the infeasibility proofs capturing the reason for current candidate functions to meet the specification, to construct a good repair.

Finally, We perform an extensive evaluation over a diverse set of benchmarks with state-of-the-art tools, viz. C2Syn[1], BFSS[2], and CADET[40]. Of 609 benchmarks, is able to solve benchmarks while C2Syn, BFSS, and CADET solve , , and benchmarks respectively. Significantly, can solve benchmarks beyond the reach of all the other existing tools extending the reach of functional synthesis tools. We then perform an extensive empirical evaluation to understand the impact of different design choices on the performance of . Our study reveals several surprising observations arising from the inter-play of machine learning and automated reasoning.

owes its runtime performance to recent advances in machine learning, constrained sampling, and automated reasoning. Encouraged by ’s scalability, we will seek to extend the above approach to related problem domains such as automated program synthesis, program repair, and reactive synthesis.

The rest of the paper is organized as follows: We first introduce notations and preliminaries in Section 2. We then discuss the related work in Section 3. In Section 4 we present an overview of and give an algorithmic description in Section  5. We then describe the experimental methodology and discuss results in Section 6. Finally, we conclude in Section 7.

2 Notations and Preliminaries

We use lower case letters (with subscripts) to denote propositional variables and upper case letters to denote a subset of variables. The formula is existentially quantified in , where and . For notational clarity, we use to refer to when clear from the context. We denote as the set of variables appearing in . A literal is a boolean variable or its negation. We often abbreviate universally (resp. existentially) quantified variables as universal (resp. existential) variables.

A satisfying assignment of a formula is a mapping , on which the formula evaluates to True. For , represents the truth values of variables in in a satisfying assignment of . We denote the set of all witnesses of as . For a formula in conjunctive normal form, the unsatisfiable core(UnsatCore) is a subset of clauses of the formula for which no satisfying assignment exists.

We use to denote substitutions: a formula obtained after substituting every occurrence of in by , where can be a constant ( or ) or a formula. The operator ite(condition,exp1,exp2) is used to represent the if-else case: if the condition is true, then it returns exp1, else it returns exp2.

A variable is considered as a positive unate if and only if is UNSAT and a negative unate if and only if is UNSAT [2].

Given a function vector for the vector of variables such that is the function corresponding to , we say that there exists a partial order over the variables such that if depends on .

In decision tree learning, a fraction of incorrectly assigned labels refer to the impurity. We use Gini Index [37] as a measure of impurity for a class label. The impurity decrease at a node is the difference of its impurity to the mean of impurities of its children. The minimum impurity decrease is a hyper-parameter used to control the maximum allowable impurity at the leaf nodes, thereby providing a lever for how closely the classifier fits the training data.

Given a propositional formula and a weight function assigning non-negative weights to every literal, we refer to the weight of a satisfying assignment , denoted as , as the product of weights of all the literals appearing in , i.e., . A sampler is a probabilistic generator that guarantees .

We use a function that takes a mapping from a sequence of variables to the desired weights of their positive literals, and assigns corresponding weights to each of the positive literals. We use a simpler notation, (a,b) to denote that positive literals corresponding to all universal variables are assigned a weight a and positive literals corresponding to all existential variables are assigned a weight b. For example, (0.5, 0.9) assigns a weight of 0.5 to the positive literals of the universally quantified variables and 0.9 to the positive literals of the existentially quantified variables.

Problem Statement:

Given a Boolean specification between set of inputs and vector of outputs , the problem of Skolem function synthesis is to synthesize a function vector such that and . We refer to as the Skolem function vector and as the Skolem function for .

A variable is called self-substituted variable, if the Skolem function corresponding to is set to  [17].

Given a formula and a Skolem function vector , we refer to as an error formula [25], where , and .

(1)

We use the following theorems from prior work:

Theorem 2.1 ([25])

is a Skolem function if and only if is UNSAT.

Theorem 2.2 ([2])

If is positive(resp negative) unate in , then (resp ) is the Skolem function for .

3 Related Work

The origins of the problem of Boolean functional synthesis traces back to Boole’s seminal work [10], which was subsequently rigorously pursued, albeit focused on decidability, by Lowenheim and Skolem [30]. The complexity theoretic studies have shown that there exist instances where Boolean functional synthesis takes super polynomial time and was also shown that there exist instances for which polynomial size Skolem function vector does not suffice unless Polynomial Hierarchy (PH) collapses [2].

Motivated by the success of the CEGAR (Counter-Example Guided Abstraction Refinement) approach in model checking, CEGAR-based approaches have been pursued in the context of synthesis as well, where the key idea is to use a Conflict-Driven Clause Learning (CDCL) SAT solver to verify and refine the candidate Skolem functions [1, 2, 3, 25].

Another line of work has focused on the representation of specification, i.e., , in representations that are amenable to efficient synthesis for a class of functions. The early approaches focused on ROBDD representation building on the functional composition approach proposed by Balabanov and Jiang [6]. Building on Tabajara and Vardi’s ROBDD-based approach [44], Chakraborty et al. extended the approach to factored specifications [12]. It is worth mentioning that factored specifications had earlier been pursued in the context of CEGAR-based approaches. Motivated by the success of knowledge compilation in the field of probabilistic reasoning, Akshay et al. achieved a significant breakthrough over a series of papers [2, 3, 25] to propose a new negation normal form, SynNNF [1]. The generalization and a functional specification presented in SynNNF is amenable to efficient functional synthesis [1]. Another line of work focused on the usage of incremental determinization to incrementally construct the Skolem functions [22, 27, 33, 40, 38].

Several approaches have been proposed for the particular case when the specification, is valid, i.e., is True. Inspired by the sequential relational decomposition, Chakraborty et al. [12] recently proposed an approach focused on viewing each CNF clause of the specification consisting of input and output clauses and employing a cooperation-based strategy. The progress in modern CDCL solvers has led to an exploration of usage of heuristics for problems in complexity classes beyond NP. This has led to work on the extraction of Skolem functions from the proofs constructed for the formulas expressed as  [6, 7].

The performance of Manthan crucially depends on its ability to employ constrained sampling, which has witnessed a surge of interest with approaches ranging from those based on hashing-based techniques [13], knowledge compilation [21, 41], augmentation of SAT solvers with heuristics [42].

The recent success of machine learning has led to several attempts to the usage of machine learning in several related synthesis domains such as program synthesis [4], invariant generation, decision-tree for functions in Linear Integer Arithmetic theory using pre-specified examples [16], strategy synthesis for QBF [23]. Use of data-driven approaches for invariant synthesis has been investigated in the ICE learning framework [15, 18, 19] aimed with data about the program behavior from test executions, it proposes invariants by learning from data, checks for inductiveness and, on failure, extend the data by the generated counterexamples. The usage of proof-artifacts such as unsat cores has been explored in verification since early 2000s [20] and in program repair in Wolverine [45], while MaxSAT has been used in program debugging in [8, 26].

4 : An overview

In this section, we provide an overview of our proposed framework, , before divulging into core algorithmic details in the following section. takes in a function specification, represented as , and returns a Skolem function vector such that . As shown in Figure 1 consists of following three phases:

  1. employs state-of-the-art pre-processing techniques on to compute a partial Skolem function vector.

  2. takes in the pre-processed formula and uses constrained samplers, and classification techniques to compute candidate Skolem functions for all the variables in .

  3. performs verification and proof-guided refinement procedure wherein a SAT solver is employed to verify the correctness of candidate functions and a MaxSAT solver in conjunction with a SAT solver is employed to refine the candidate functions until the entire candidate Skolem function vector passes the verification check.

Figure 1: Overview of

We now provide a high-level description of different phases to highlight the technical challenges, which provides context for several algorithmic design choices presented in the next section.

4.1 Phase 1:

focuses on pre-processing of the formula to search for unates among the variables in ; if is positive (resp. negative) unate, then (resp. ) suffices as a Skolem function. We employ the algorithmic routine proposed by Akshay et al. [2] to drive this preprocessing.

4.2 Phase 2:

views the problem of functional synthesis through the lens of machine learning where the learned machine learning model for classification of a variable can be viewed as a candidate Skolem function for . We gather training data about the function’s behavior by exploiting the progress in constrained sampling to sample solutions of . Recall that defines a relation (and not necessarily a function) between and , and the machine learning techniques typically assume the existence of function between features and labels, necessitating the need for sophisticated sampling strategy as discussed below. Moving on to features and labels, since we want to learn in terms of , we view as a set of features while assignments to as a set of class labels.

The off-the-shelf classification techniques typically require that the size of training data is several times larger than the size of possible class labels, which would be prohibitively large for the typical problems involving more than thousand variables. To mitigate the requirement of large training data, we make note of two well-known observations in functional synthesis literature: (1) the Skolem function for a variable typically does not depend on all the variables in , (2) A Skolem function vector where depends on variable is a valid vector if the Skolem function is not dependent on (i.e., acyclic dependency), i.e., there exists a partial order over .

The above observations lead us to design an algorithmic procedure where we learn candidate Skolem functions as decision trees in an iterative manner, i.e., one at a time, thereby allowing us to constrain ourselves to the binary classification. The learned classifier can then be represented as the disjunction of all the paths from the root to the leaves in the learnt decision tree. We update the set of possible features for a given depending on the candidate functions generated so far, i.e., valuation of variables and variables, which are not dependent on . Finally, we compute the candidate Skolem function for as the disjunction of labels along edges for all the paths from the root to leaf nodes with label 1. Once, we have the candidate Skolem function vector , we obtain a valid linear extension, TotalOrder, of the partial order in accordance to .

Before moving on to the next phase, we return to the formulation of sampling. The past few years have witnessed the design of uniform [13, 41], and weighted samplers [21], and one wonders what kind of sampler should we choose to generate samples for training data. A straightforward choice would be to perform uniform sampling over and , but the relational nature of specification, , between and offers interesting challenges and opportunities. Recall while specifies a relation between and , we are interested in a Skolem function, and we would like to tailor our sampling subroutines to allow discovery of Skolem functions with small description given the relationship between description and sample complexity. To this end, consider and , and let . Note that has 7 solutions over , out of which appears in solutions while appears in 4. Also, note that there are several possible Skolem functions such as . Now, if we uniformly sample solutions of over , i.e. (0.5, 0.5), we would see (almost) equal number of samples with and . A closer look at reveals that it is possible to construct a Skolem function by knowing that the only case where cannot be assigned is when . To encode this intuition, we propose a novel idea of collecting samples with weighted sampling, i.e., (0.5, q) where is chosen in a multi-step process of first drawing a small set of samples with both and , and then drawing rest of the samples by fixing the value of following analysis of an initial set of samples. To the best of our knowledge, this is the first application of weighted sampling in the context of synthesis, and our experimental results point to several interesting avenues of future work.

4.3 Phase 3:

The candidate Skolem functions generated in may not always be the actual Skolem functions. Hence, we require a verification check to see if candidate Skolem functions are indeed correct; if not, the generated counterexample can be used to repair it. The verification query constructs an error formula (Formula 1): if unsatisfiable, the candidate Skolem function vector is indeed a Skolem function vector and the procedure can terminate; else, when is SAT, the solution of is used to identify and refine the erring functions among the candidate Skolem function vector.

In contrast to prior techniques that apply Shannon expansion or self-substitution, the refinement strategy in is guided by the view that the candidate function vector from the phase is almost correct, and hence, attempts to identify and apply a series of minor repairs to the erring functions to arrive at the correct Skolem function vector. To this end, uses two key techniques: fault localization and repair synthesis. Let us assume that is a satisfying assignment of and referred to as counterexample for the current candidate Skolem function vector .

Fault Localization

In order to identify the initial candidates to repair for the counterexample , attempts to identify a small number of Skolem functions (correspondingly variables) whose outputs must undergo a change for the formula to behave correctly on ; in other words, it makes a best-effort attempt to ensure that most of the Skolem functions (correspondingly variables) can retain their current output on while satisfying the formula. encodes this problem as a partial MaxSAT query with as a hard constraint and as soft constraints. All variables whose valuation constraint does not hold in the MaxSAT solution are identified as erring Skolem functions that may need to be repaired.

Repair Synthesis

Let be the variable corresponding to the erring function, , identified in the previous step. To synthesize a repair for the function, applies a proof-guided strategy: it constructs a formula , such that if is unsatisfiable then must undergo a change. The UnsatCore of provides a reason that explains the discrepancy between the specification and the current Skolem function.

(2)

uses the UnsatCore to constructs a repair formula, say , as a conjunction over literals in the unsatisfiable core; if is true with the current valuation of and , updates the function by conjoining it with the negation of repair formula (); otherwise, updates the function , by disjoining it with the repair formula (.

4.3.1 Self-substitution for poorly learnt functions

Some Skolem functions are difficult to learn through data. In our implementation, the corresponding variables escape the phase with poor candidate functions, thereby requiring a long sequence of incremental repairs for convergence. To handle such scenarios, we make the following observation: though synthesizing Skolem functions via self-substitution[17] can lead to an exponential blowup in the worst case, it is inexpensive if the number of variables synthesized via this technique is small. We use this observation to quickly synthesize a Skolem function for an erring variable if we detect its candidate function is poor (detected by comparing the number of times it enters refinement against an empirically determined threshold). Of course, this heuristic does not scale well if the number of such variables is large; in our experiments, we found less than 20% of the instances solved required self-substitution, and for over 75% of these instances, only one variable needed self-substitution. We elaborate more on the empirical evidence on the success of this heuristic in section 6. A theoretical understanding of the learnability of Boolean functions from data seems to be an interesting direction for future work.

5 : Algorithmic Description

1 () foreach   do
2        ()
3 () repeat
4        () if ret = SAT then
5              
until ret = UNSAT () return
Algorithm 1 ()

In this section, we present a detailed algorithmic description of , whose pseudocode is presented in Algorithm 1. takes in a formula as input and returns a Skolem vector . The algorithm starts off by preprocessing (line 1) the formula to get the unates () and their corresponding Skolem functions (). Next, it invokes the sampler (line 1) to collect a set of samples as training data for the learning phase.

For each of the existential variables that are not unates, attempts to learn candidate Skolem functions (lines 1-1). To generate a variable order, uses a collection of sets , such that indicates that depends on . Next, the routine (line 1) construct TotalOrder of the variables in accordance to the dependencies in . The verification and refinement phase (line 1) commences by constructing the error formula and launching the verification check (line 1). If the error formula is satisfiable, the counterexample model () is used to refine the formula. Once the verification check is successful, the refinement phase ends and the subroutine is invoked to recursively substitute all appearing in Skolem functions with their corresponding Skolem functions such that only variables entirely describe all Skolem functions. The strict variable ordering enforced above ensures that always succeeds and does not get stuck in a cycle. Finally, the Skolem function vector is returned.

It is worth noting that can successfully solve an instance without having to necessarily execute all the phases. In particular, if , then terminates after (i.e., line 1). Similarly, if the return UNSAT during the first iteration of loop (lines 11), then does not invoke . We now discuss each subroutine in detail.

:
1 foreach  do
2        if  = UNSAT then
3                  /* is positive unate */
4       else
5               if  = UNSAT then
6                         /* is negative unate */
return
Algorithm 2 ()

We perform the pre-processing step as described in [2], which performs SAT queries on the formulas constructed as specified in Theorem 2.2. As described in Algorithm 2, sequentially detects unate for every of variable. first checks for the satisfiability of positive unate formula (line 2) for . If is a positive unate, then is (line 2), and is added to set . If is not a positive unate, then checks for the satisfiability of negative unate formula (lines 2). If is a negative unate, then is (line 2), and is added to set .

:
1 () () foreach  do
2        Count( Count( if   then
3              
4       else
5              
() return
Algorithm 3 ()

, presented in Algorithm 3, takes as input and returns, , a subset of satisfying assignments of . first generates samples each with (line 3), and with (line 3). Then, in line 3, calculates for all , is a ratio of number of samples with being to the total number of samples, i.e. 500. Similarity, in line 3, it calculates for all , is a ratio of number of samples with being to the total number of samples. Finally, generates required number of samples with ; for a , is if both and are in range to , else is .

:
1 foreach   do
2        if   then
3               /* if is not dependent on */
4 foreach  n (t) do
5        if (n) = 1  then
6              
7 foreach  do
8       
return
Algorithm 4

, presented in Algorithm 4, assumes access to following three subroutines:

  1. [topsep=0pt]

  2. takes the feature and label sets as input (training data) and returns a decision tree . We use the ID3 algorithm [37] to construct a decision tree where the internal node of represents a feature on which a decision is made, the branches represent partitioning of the training data on the decision, and the leaf nodes represent the classification outcomes (i.e class labels). The ID3 algorithm iterates over the training data, and in each iteration, it selects a new attribute to extend the tree by a new decision node: the selected attribute is one that causes the maximum drop in the impurity of the resulting classes; we use Gini Index [37] as the measure of impurity. The algorithm, then, extends the tree by the selected decision and continues extending building the tree. The algorithm terminates on a path if either it exhausts all attributes for decisions, or the impurity of the resulting classes drop below a (user-specified) impurity decrease parameter.

  3. takes a leaf node of the decision tree as input and returns the class label corresponding to the node.

  4. takes a tree and two nodes of (node and node ) as input and outputs a conjunction of literals in the path from node to node in .

As we seek to learn Boolean functions, we employ binary classifiers with class labels 0 and 1. shows our algorithm for extracting a Boolean function from the decion trees: lines 4-4 find a feature set (featset) to predict . The feature set includes all variables and the subset of variables that are not dependent on . creates decision tree using samples over the feature set. Lines 4-4 generate candidate Skolem function by iterating over all the leaf nodes of . In particular, if a leaf node is labeled with 1, the candidate function is updated by disjoining with the formula returned by subroutine . also updates in , is set of all variables on which, depends. If depends on , then by transitivity also depends on ; in line 4, updates  accordingly.

:

takes as an input to output a valid linear extension of the partial order defined over with respect to the candidate Skolem function vector .

:
1; () foreach  do
2        if  then
3              
4       else
5               if   then
6                     
7              else
8                      foreach   do
9                             if   then
10                                   
11                     
12              
return
Algorithm 5 ()

is invoked with a counterexample . first performs fault localization to find the initial set of erring candidate functions; to this end, it calls the subroutine (line 5) with as hard-constraints and as soft-constraints. employs a MaxSAT solver to find the solution that satisfies all the hard constraints and maximizes the number of satisfied soft constraints, and then returns a list (Ind) of variables such that for each of the variables appearing in (Ind) the corresponding soft-constraint was not satisfied by the optimal solution returned by MaxSAT solver.

Since candidate Skolem function corresponding to the variables in Ind needs to refine, now attempts to synthesize a repair for each of these candidate Skolem functions. Repair synthesis loop (lines 55) starts off by collecting the set of variables, , on which of Ind can depend on as per the ordering constraints (line 5). Next, it invokes the subroutine , which returns True if the candidate function corresponding to has been refined more than a chosen threshold times (fixed to 10 in our implementation), and the corresponding decision tree constructed during execution has exactly one node. If returns true, calls to perform self-substitution. takes a formula , an existentially quantified variable and a list of variables which depends on and performs self substitution of with constant in the formula [25].

If returns false, attempts a proof-guided repair for . calls in line 5 on , which corresponds to formula 2: if is SAT, then returns a satisfying assignment of in , else returns unsatisfiable in the result, .

  1. If is UNSAT, we proceed to refine such that for . Ideally, we would like to apply a refinement that generalizes to potentially other counter-examples, i.e. solutions of . To this end, calls with ; returns the list of variables () that occur in the clauses of UnsatCore of . Accordingly, the algorithm constructs a repair formula as a conjunction of literals in corresponding to variables in (line 5). If is , then is with conjunction of negation of and if is , then is with disjunction of .

  2. If is SAT and is a satisfying assignment of , then there exists a Skolem function vector such that the value of agrees with for the valuation of and set to and . However, for any if , then for such a , the Skolem function corresponding to may need to refine . Therefore, adds to list of candidates to refine, . Note that since , there exists at least one iteration of the loop (lines 5– 5) where is UNSAT.

:
1foreach  do
2        foreach  do
3              
return
Algorithm 6 ()

To return the Skolem functions in terms of only , invokes subroutine. As described in Algorithm 6, for each of variable, finds existentially quantified variables(), on which depends. Then, for each of ; it substitutes corresponding Skolem function in the Skolem function of .

5.1 Example

We now illustrate our algorithm though an example.

Example 1

Let , in where is

:

As appears only as positive literal in , discovers as a positive unate, synthesizing a Skolem function() for as .

:

generates training data through sampling (Figure 2) and attempts to learn a candidate Skolem function of by creating a decision tree (Figure 3). The decision tree construction uses the samples of as features and samples of as labels. This decision tree is compiled down to a propositional formula by taking a disjunction over all paths that end in leaf nodes with label : in Figure 3, is synthesized as .

As is not dependent on , the samples of can be used to predict . Considering the path to the leaf nodes of the tree (Figure 4) with label 1, is .

:

Error formula is UNSAT with . returns as skolem functions for .

To illustrate ; let us assume that returns the candidate Skolem functions . We construct the error formula, , which turns out SAT with counterexample , , , , , , , .

calls with , which returns as is an unsatisfiable constraint.

Repair synthesis commences for with a query for satisfiability on . As the formula is unsatisfiable, calls , which returns variable (as and are not satisfiable in ).

As the repaired output of on current must change from 0 to 1, is updated by a disjunction of the old with , yielding the refined formula . For this case, the formula is UNSAT, thus passing the verification check.

Figure 2: Samples of Figure 3: Decision tree for Figure 4: Decision tree for

6 Experimental Results

We evaluate the performance of on the union of all the benchmarks employed in the most recent works [1, 2],which includes benchmarks from different sources: Prenex-2QBF track of QBFEval-17[35], QBFEval-18[36], disjunctive[3], arithmetic[44] and factorization[3]. We ran all the tools as per the specification laid out by their authors. We used Open-WBO [31] for our MaxSAT queries and PicoSAT [9] to compute UnsatCore. We used PicoSAT for its ease of usage and we expect further performance improvements by upgrading to one of the state of the art SAT solvers. We have used the Scikit-Learn[34] to create decision trees in phase of . We have also used ABC [28] to represent and manipulate Boolean functions. To allow for the input formats supported by the different tools, we use the utility scripts available with the BFSS distribution [2] to convert each of the instances to both QDIMACS and Verilog formats. For , unless otherwise specified, we set the number of samples according to heuristic based on as described in Section 6.3 and minimum impurity decrease to . All our experiments were conducted on a high-performance computer cluster with each node consisting of a E5-2690 v3 CPU with cores and GB of RAM, with a memory limit set to GB per core. All tools were run in a single-threaded mode on a single core with a timeout of seconds.

The objective of our experimental evaluation was two-fold: to understand the impact of various design choices on the runtime performance of and to perform an extensive comparison of runtime performance vis-a-vis state of the art synthesis tools. In particular, we sought to answer the following questions:

  1. How does the performance of compare with state of the functional synthesis engines?

  2. How do the usage of different sampling schemes and the quality of samplers impact the performance of ?

  3. What is the impact of on the performance of ?

  4. What is the distribution of the time spent in each phase of ?

  5. How does using MaxSAT solver to identify the potential erring Skolem functions impacts on the performance of ?

  6. How does employing self-substitution for some Skolem functions impact ?

We observe that