Proving Unrealizability for Syntax-Guided Synthesis

05/14/2019 ∙ by Qinheping Hu, et al. ∙ 0

Proving Unrealizability for Syntax-Guided Synthesis We consider the problem of automatically establishing that a given syntax-guided-synthesis (SyGuS) problem is unrealizable (i.e., has no solution). Existing techniques have quite limited ability to establish unrealizability for general SyGuS instances in which the grammar describing the search space contains infinitely many programs. By encoding the synthesis problem's grammar G as a nondeterministic program P_G, we reduce the unrealizability problem to a reachability problem such that, if a standard program-analysis tool can establish that a certain assertion in P_G always holds, then the synthesis problem is unrealizable. Our method can be used to augment any existing SyGus tool so that it can establish that a successfully synthesized program q is optimal with respect to some syntactic cost -- e.g., q has the fewest possible if-then-else operators. Using known techniques, grammar G can be automatically transformed to generate exactly all programs with lower cost than q -- e.g., fewer conditional expressions. Our algorithm can then be applied to show that the resulting synthesis problem is unrealizable. We implemented the proposed technique in a tool called NOPE. NOPE can prove unrealizability for 59/134 variants of existing linear-integer-arithmetic SyGus benchmarks, whereas all existing SyGus solvers lack the ability to prove that these benchmarks are unrealizable, and time out on them.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The goal of program synthesis is to find a program in some search space that meets a specification—e.g., satisfies a set of examples or a logical formula. Recently, a large family of synthesis problems has been unified into a framework called syntax-guided synthesis (SyGuS). A SyGuS problem is specified by a regular-tree grammar that describes the search space of programs, and a logical formula that constitutes the behavioral specification. Many synthesizers now support a specific format for SyGuS problems [1], and compete in annual synthesis competitions [2]. Thanks to these competitions, these solvers are now quite mature and are finding a wealth of applications [9].

Consider the SyGuS problem to synthesize a function that computes the maximum of two variables and , denoted by . The goal is to create —an expression-tree for —where is in the language of the following regular-tree grammar :

and is valid, where denotes the meaning of , and

SyGuS solvers can easily find a solution, such as

Although many solvers can now find solutions efficiently to many SyGuS problems, there has been effectively no work on the much harder task of proving that a given SyGuS problem is unrealizable—i.e., it does not admit a solution. For example, consider the SyGuS problem , where is the more restricted grammar with if-then-else operators and conditions stripped out:

This SyGuS problem does not have a solution, because no expression generated by meets the specification.111Grammar only generates terms that are equivalent to some linear function of and ; however, the maximum function cannot be described by a linear function. However, to the best of our knowledge, current SyGuS solvers cannot prove that such a SyGuS problem is unrealizable.222The synthesis problem presented above is one that is generated by a recent tool called QSyGuS, which extends SyGuS with quantitative syntactic objectives [10]. The advantage of using quantitative objectives in synthesis is that they can be used to produce higher-quality solutions—e.g., smaller, more readable, more efficient, etc. The synthesis problem arises from a QSyGuS problem in which the goal is to produce an expression that (i) satisfies the specification , and (ii) uses the smallest possible number of if-then-else operators. Existing SyGuS solvers can easily produce a solution that uses one if-then-else operator, but cannot prove that no better solution exists—i.e., is unrealizable.

A key property of the previous example is that the grammar is infinite. When such a SyGuS problem is realizable, any search technique that systematically explores the infinite search space of possible programs will eventually identify a solution to the synthesis problem. In contrast, proving that a problem is unrealizable requires showing that every program in the infinite search space fails to satisfy the specification. This problem is in general undecidable [6]. Although we cannot hope to have an algorithm for establishing unrealizability, the challenge is to find a technique that succeeds for the kinds of problems encountered in practice. Existing synthesizers can detect the absence of a solution in certain cases (e.g., because the grammar is finite, or is infinite but only generate a finite number of functionally distinct programs). However, in practice, as our experiments show, this ability is limited—no existing solver was able to show unrealizability for any of the examples considered in this paper.

In this paper, we present a technique for proving that a possibly infinite SyGuS problem is unrealizable. Our technique builds on two ideas.

  1. We observe that unrealizability can often be proven using finitely many input examples. In IllustrativeExample, we show how the example discussed above can be proven to be unrealizable using four input examples—, , , and .

  2. We devise a way to encode a SyGuS problem over a finite set of examples as a reachability problem in a recursive program . In particular, the program that we construct has an assertion that holds if and only the given SyGuS problem is unrealizable. Consequently, unrealizability can be proven by establishing that the assertion always holds. This property can often be established by a conventional program-analysis tool.

The encoding mentioned in EncodingOfGrammarAndExamples is non-trivial for three reasons. The following list explains each issue, and sketches how they are addressed

1) Infinitely many terms. We need to model the infinitely many terms generated by the grammar of a given synthesis problem .

To address this issue, we use non-determinism and recursion, and give an encoding such that (i) each non-deterministic path in the program corresponds to a possible expression that can generate, and (ii) for each expression that can generate, there is a path in . (There is an isomorphism between paths and the expression-trees of )

2) Nondeterminism. We need the computation performed along each path in to mimic the execution of expression . Because the program uses non-determinism, we need to make sure that, for a given path in the program , computational steps are carried out that mimic the evaluation of for each of the finitely many example inputs in .

We address this issue by threading the expression-evaluation computations associated with each example in through the same non-deterministic choices.

3) Complex Specifications. We need to handle specifications that allow for nested calls of the programs being synthesized.

For instance, consider the specification . To handle this specification, we introduce a new variable and rewrite the specification as . Because is now also used as an input to , we will thread both the computations of and through the non-deterministic recursive program.

Our work makes the following contributions:

  • We reduce the SyGuS unrealizability problem to a reachability problem to which standard program-analysis tools can be applied (IllustrativeExample and verification).

  • We observe that, for many SyGuS problems, unrealizability can be proven using finitely many input examples, and use this idea to apply the Counter-Example-Guided Inductive Synthesis (CEGIS) algorithm to the problem of proving unrealizability (cegis).

  • We give an encoding of a SyGuS problem over a finite set of examples as a reachability problem in a nondeterministic recursive program , which has the following property: if a certain assertion in always holds, then the synthesis problem is unrealizable (verification).

  • We implement our technique in a tool nope using the ESolver synthesizer [2] as the SyGuS solver and the SeaHorn tool [8] for checking reachability. nope is able to establish unrealizability for 59 out of 132 variants of benchmarks taken from the SyGuS competition. In particular, nope solves all benchmarks with no more than 15 productions in the grammar and requiring no more than 9 input examples for proving unrealizability. Existing SyGuS solvers lack the ability to prove that these benchmarks are unrealizable, and time out on them.

RelatedWork discusses related work. Some additional technical material, proofs, and full experimental results are given in AdditionalMaterialproofssupp-eval, respectively.

2 Illustrative Example

In this section, we illustrate the main components of our framework for establishing the unrealizability of a SyGuS problem.

Consider the SyGuS problem to synthesize a function that computes the maximum of two variables and , denoted by . The goal is to create —an expression-tree for —where is in the language of the following regular-tree grammar :

and is valid, where denotes the meaning of , and

SyGuS solvers can easily find a solution, such as

Although many solvers can now find solutions efficiently to many SyGuS problems, there has been effectively no work on the much harder task of proving that a given SyGuS problem is unrealizable—i.e., it does not admit a solution. For example, consider the SyGuS problem , where is the more restricted grammar with if-then-else operators and conditions stripped out:

This SyGuS problem does not have a solution, because no expression generated by meets the specification.333Grammar generates all linear functions of and , and hence generates an infinite number of functionally distinct programs; however, the maximum function cannot be described by a linear function. However, to the best of our knowledge, current SyGuS solvers cannot prove that such a SyGuS problem is unrealizable. As an example, we use the problem discussed in Introduction, and show how unrealizability can be proven using four input examples: , , , and .

width=.75center

1int I_0;
2void Start(int x_0,int y_0){
3  if(nd()){  // Encodes ‘‘Start ::= Plus(Start, Start)’’ 
4    Start(x_0, y_0);   
5    int tempL_0 = I_0;
6    Start(x_0, y_0);   
7    int tempR_0 = I_0;
8    I_0 = tempL_0 + tempR_0;  
9  }
10  else if(nd())  I_0 = x_0;  // Encodes ‘‘Start ::= x’’  
11  else if(nd())  I_0 = y_0;  // Encodes ‘‘Start ::= y’’  
12  else if(nd())  I_0 = 1;    // Encodes ‘‘Start ::= 1’’  
13  else         I_0 = 0;    // Encodes ‘‘Start ::= 0’’
14}
15
16bool spec(int x, int y, int f){
17  return (f>=x && f>=y && (f==x || f==y))
18}
19
20void main(){
21  int x_0 = 0; int y_0 = 1;   // Input example (0,1)  
22  Start(x_0,y_0);               
23  assert(!spec(x_0,y_0,I_0));   
24} \end{lstlisting}
25  \end{adjustbox}
26\caption{Program $\pge{G_2}{E_1}$ created during the course of proving the unrealizability
27  of $(\psi_{\texttt{max2}}(f,x,y), G_2)$ using the set of input examples $E_1=\{(0,1)\}$.
28}
29\label{Fi:cexample0}
30\end{figure}
31
32Our method can be seen as a variant of Counter-Example-Guided
33Inductive Synthesis (CEGIS), in which the goal is to create a program $P$
34in which a certain assertion always holds.
35Until such a program is created, each round of the algorithm returns
36a counter-example, from which we extract an additional input example for the
37original \sygus problem.
38On the $i^{\textrm{th}}$ round, the current set of input examples $E_i$ is used,
39together with the grammar---in this case $G_2$---and the specification of the
40desired behavior---$\psi_{\texttt{max2}}(f,x,y)$, to create a candidate program $\pge{G_2}{E_i}$.
41The program $\pge{G_2}{E_i}$ contains an assertion, and a standard program analyzer is used
42to check whether the assertion always holds.
43
44Suppose that for the \sygus problem $(\psi_{\texttt{max2}}(f,x,y), G_2)$
45we start with just the one example input $(0,1)$---i.e., $E_1=\{(0,1)\}$.
46\figref{cexample0} shows the initial program $\pge{G_2}{E_1}$ that our method creates.
47The function \texttt{spec} implements the predicate $\psi_{\texttt{max2}}(f,x,y)$.
48(All of the programs $\{ \pge{G_2}{E_i} \}$ use the same function \texttt{spec}.)
49The initialization statements ‘‘\texttt{int x\_0 = 0; int y\_0 = 1;}’’
50at \lineref{Initialization0} in procedure \texttt{main} correspond to
51the input example $(0,1)$.
52The recursive procedure \texttt{Start} encodes the productions of grammar $G_2$.
53\texttt{Start} is non-deterministic;
54it contains four calls to an external function \nd{}, which returns a
55non-deterministically chosen Boolean value.
56The calls to \nd{} can be understood as controlling whether or not a
57production is selected from $G_2$ during a top-down, left-to-right
58generation of an expression-tree:
59\lineseqref{PlusBegin0}{PlusEnd0} correspond to
60‘‘Start ::= Plus(Start, Start),’’ and
61\linerefspp{StartX0}{StartY0}{StartOne0}{StartZero0} correspond to
62‘‘Start ::= x,’’ ‘‘Start ::= y,’’ ‘‘Start ::= 1,’’ and ‘‘Start ::= 0,’’ respectively.
63The code in the five cases in the body of \texttt{Start} encodes
64the semantics of the respective production of $G_2$;
65in particular, the statements that are executed along any execution
66path of $\pge{G_2}{E_1}$ implement the \emph{bottom-up evaluation of
67some expression-tree that can be generated by $G_2$}.
68For instance, consider the path that visits statements in the
69following order (for brevity, some statement numbers have been
70elided):
71\begin{equation}
72  \label{Eq:PathPlusXOne0}
73  \ref{Li:Initialization0}~~\,
74  \ref{Li:MainCallsStart0}~~\,(_{\texttt{Start}}~~\,
75    \ref{Li:PlusBegin0}~~\,
76    \ref{Li:StartRecursiveOne0}~~\,(_{\texttt{Start}}~~\,
77      \ref{Li:StartX0}~~\,
78    )_{\texttt{Start}}~~\,
79    \ref{Li:StartRecursiveTwo0}~~\,(_{\texttt{Start}}~~\,
80      \ref{Li:StartOne0}~~\,
81    )_{\texttt{Start}}~~\,
82    \ref{Li:PlusEnd0}~~\,
83  )_{\texttt{Start}}~~\,
84  \ref{Li:Assertion0},
85\end{equation}
86where $(_{\texttt{Start}}$ and $)_{\texttt{Start}}$ indicate entry to,
87and return from, procedure \texttt{Start}, respectively.
88Path (\ref{Eq:PathPlusXOne0}) corresponds to the top-down, left-to-right generation
89of the expression-tree \texttt{Plus(x,1)}, interleaved with the tree’s bottom-up evaluation.
90
91\begin{figure}[!t]
92  \begin{adjustbox}{width=.85\textwidth,center}
93  \begin{lstlisting}[style=CStyle]
94int I_0, I_1, I_2, I_3;
95void Start(int x_0,int y_0,...,int x_3,int y_3){
96  if(nd()){  // Encodes ‘‘Start ::= Plus(Start, Start)’ 
97    Start(x_0, y_0, x_1, y_1, x_2, y_2, x_3, y_3);    
98    int tempL_0 = I_0; int tempL_1 = I_1;
99    int tempL_2 = I_2; int tempL_3 = I_3;
100    Start(x_0, y_0, x_1, y_1, x_2, y_2, x_3, y_3);     
101    int tempR_0 = I_0; int tempR_1 = I_1;
102    int tempR_2 = I_2; int tempR_3 = I_3;
103    I_0 = tempL_0 + tempR_0;
104    I_1 = tempL_1 + tempR_1;
105    I_2 = tempL_2 + tempR_2;
106    I_3 = tempL_3 + tempR_3;}    
107  else if(nd()) {  // Encodes ‘‘Start ::= x’
108    I_0 = x_0; I_1 = x_1; I_2 = x_2; I_3 = x_3;}   
109  else if(nd()) {  // Encodes ‘‘Start ::= y’
110    I_0 = y_0; I_1 = y_1; I_2 = y_2; I_3 = y_3;}   
111  else if(nd()) {  // Encodes ‘‘Start ::= 1’
112    I_0 = 1;   I_1 = 1;   I_2 = 1;   I_3 = 1;}           
113  else {          // Encodes ‘‘Start ::= 0’  
114    I_0 = 0;   I_1 = 0;   I_2 = 0;   I_3 = 0;}
115}
116
117bool spec(int x, int y, int f){
118  return (f>=x && f>=y && (f==x || f==y))
119}
120
121void main(){
122  int x_0 = 0; int y_0 = 1;  // Input example (0,1)     
123  int x_1 = 0; int y_1 = 0;  // Input example (0,0)
124  int x_2 = 1; int y_2 = 1;  // Input example (1,1)
125  int x_3 = 1; int y_3 = 0;  // Input example (1,0)     
126  Start(x_0,y_0,x_1,y_1,x_2,y_2,x_3,y_3);               
127  assert(   !spec(x_0,y_0,I_0) || !spec(x_1,y_1,I_1)    
128         || !spec(x_2,y_2,I_2) || !spec(x_3,y_3,I_3));       
129} \end{lstlisting}
130  \end{adjustbox}
131\caption{Program $\pge{G_2}{E_4}$ created during the course of proving the unrealizability
132  of $(\psi_{\texttt{max2}}(f,x,y), G_2)$ using the set of input examples $E_4=\{(0,0), (0,1), (1,0), (1,1)\}$.
133}
134\label{Fi:cexample}
135\end{figure}
136
137Note that with path (\ref{Eq:PathPlusXOne0}), when control returns to \texttt{main},
138variable \texttt{I\_0} has the value 1, and thus the assertion at
139\lineref{Assertion0} fails.
140
141A sound program analyzer will discover that some such path exists in the program,
142and will return the sequence of non-deterministic choices required to follow
143one such path.
144Suppose that the analyzer chooses to report path
145(\ref{Eq:PathPlusXOne0}); the sequence of choices would be $t, f, t,
146f, f, f, t$, which can be decoded to create the expression-tree
147\texttt{Plus(x,1)}.
148At this point, we have a candidate definition for $f$: $f = x + 1$.
149This formula can be checked using an SMT solver to see whether it
150satisfies the behavioral specification $\psi_{\texttt{max2}}(f,x,y)$.
151In this case, the SMT solver returns ‘‘false.’
152One counter-example that it could return is $(0,0)$.
153
154
155
156At this point, program $\pge{G_2}{E_2}$ would be constructed using both of the example inputs
157$(0,1)$ and $(0,0)$.
158Rather than describe $\pge{G_2}{E_2}$, we will describe the final program constructed, $\pge{G_2}{E_4}$
159(see \figref{cexample}).
160%The structure of $\pge{G_2}{E_2}$ will be obvious from the structure of $\pge{G_2}{E_4}$.
161
162As can be seen from the comments in the two programs, program $\pge{G_2}{E_4}$ has the same
163basic structure as $\pge{G_2}{E_1}$.
164\begin{itemize}
165  \item
166    \texttt{main} begins with initialization statements for
167    the four example inputs.
168  \item
169    \texttt{Start} has five cases that correspond to the
170    five productions of $G_2$.
171\end{itemize}
172The main difference is that because the encoding of $G_2$ in \texttt{Start}
173uses non-determinism, we need to make sure that along \emph{each}
174path $p$ in $\pge{G_2}{E_4}$, each of the example inputs is used to evaluate the
175\emph{same} expression-tree.
176We address this issue by threading the expression-evaluation
177computations associated with each of the example inputs through
178the \emph{same} non-deterministic choices.
179That is, each of the five ‘‘production cases’ in \texttt{Start} has four
180encodings of the production’s semantics---one for each of the four
181expression evaluations.
182By this means, the statements that are executed along path $p$ perform
183\emph{four simultaneous bottom-up evaluations} of the expression-tree
184from $G_2$ that corresponds to $p$.
185
186Programs $\pge{G_2}{E_2}$ and $\pge{G_2}{E_3}$ are similar to $\pge{G_2}{E_4}$, but their paths carry
187out two and three simultaneous bottom-up evaluations, respectively.
188The actions taken during rounds 2 and 3 to generate a new
189counter-example---and hence a new example input---are similar to what
190was described for round 1.
191On round 4, however, the program analyzer will determine that the
192assertion on \lineseqref{AssertionBegin}{AssertionEnd} always holds,
193which means that there is no path through $\pge{G_2}{E_4}$ for which the
194behavioral specification holds for all of the input examples.
195%Because there is an isomorphism between the set of paths in $\pge{G_2}{E_4}$ and
196%the expression-trees generated by $G_2$,
197This property means that
198there is no expression-tree that satisfies the specification---i.e.,
199the \sygus problem $(\psi_{\texttt{max2}}(f,x,y), G_2)$ is unrealizable.
200
201Our implementation uses the program-analysis tool \seahorn\ \cite{seahorn}
202as the assertion checker.
203In the case of $\pge{G_2}{E_4}$, \seahorn takes only 0.5 seconds to establish
204that the assertion in $\pge{G_2}{E_4}$ always holds.
205
206%%% Local Variables:
207%%% mode: latex
208%%% TeX-master: "main.tex"
209%%% End:

3 SyGuS, Realizability, and Cegis

3.1 Background

Trees and Tree Grammars.

A ranked alphabet is a tuple where is a finite set of symbols and associates a rank to each symbol. For every , the set of all symbols in with rank is denoted by . In our examples, a ranked alphabet is specified by showing the set and attaching the respective rank to every symbol as a superscript—e.g., . (For brevity, the superscript is sometimes omitted.) We use to denote the set of all (ranked) trees over —i.e., is the smallest set such that (i, (ii) if and , then . In what follows, we assume a fixed ranked alphabet .

In this paper, we focus on typed regular tree grammars, in which each nonterminal and each symbol is associated with a type. There is a finite set of types . Associated with each symbol , there is a type assignment , where is called the left-hand-side type and are called the right-hand-side types. Tree grammars are similar to word grammars, but generate trees over a ranked alphabet instead of words.

Definition 1 (Regular Tree Grammar)

A typed regular tree grammar (RTG) is a tuple , where is a finite set of non-terminal symbols of arity 0; is a ranked alphabet; is an initial non-terminal; is a type assignment that gives types for members of ; and is a finite set of productions of the form , where for , each is a non-terminal such that if then .

In a SyGuS problem, each variable, such as and in the example RTGs in Introduction, is treated as an arity- symbol—i.e., and .

Given a tree , applying a production to produces the tree resulting from replacing the left-most occurrence of in with the right-hand side . A tree is generated by the grammar —denoted by —iff it can be obtained by applying a sequence of productions to the tree whose root is the initial non-terminal .

Syntax-Guided Synthesis.

A SyGuS problem is specified with respect to a background theory —e.g., linear arithmetic—and the goal is to synthesize a function that satisfies two constraints provided by the user. The first constraint, , describes a semantic property that should satisfy. The second constraint limits the search space of , and is given as a set of expressions specified by an RTG that defines a subset of all terms in .

Definition 2 (SyGuS)

A SyGuS problem over a background theory is a pair where is a regular tree grammar that only contains terms in —i.e., —and is a Boolean formula constraining the semantic behavior of the synthesized program .

A SyGuS problem is realizable if there exists a expression such that is true. Otherwise we say that the problem is unrealizable.

Theorem 3.1 (Undecidability [6])

Given a SyGuS problem , it is undecidable to check whether is realizable.

Counterexample-Guided Inductive Synthesis

The Counterexample-Guided Inductive Synthesis (CEGIS) algorithm is a popular approach to solving synthesis problems. Instead of directly looking for an expression that satisfies the specification on all possible inputs, the CEGIS algorithm uses a synthesizer that can find expressions that are correct on a finite set of examples . If finds a solution that is correct on all elements of , CEGIS uses a verifier to check whether the discovered solution is also correct for all possible inputs to the problem. If not, a counterexample obtained from is added to the set of examples, and the process repeats. More formally, CEGIS starts with an empty set of examples and repeats the following steps:

  1. Call the synthesizer to find an expression such that holds and go to step 2; return unrealizable if no expression exists.

  2. Call the verifier to find a model for the formula , and add to the counterexample set ; return as a valid solution if no model is found.

Because SyGuS problems are only defined over first-order decidable theories, any SMT solver can be used as the verifier to check whether the formula is satisfiable. On the other hand, providing a synthesizer to find solutions such that holds is a much harder problem because is a second-order term drawn from an infinite search space. In fact, checking whether such an exists is an undecidable problem [6].

The main contribution of our paper is a reduction of the unrealizability problem—i.e., the problem of proving that there is no expression such that holds—to an unreachability problem (verification). This reduction allows us to use existing (un)reachability verifiers to check whether a SyGuS instance is unrealizable.

3.2 Cegis and Unrealizability

The CEGIS algorithm is sound but incomplete for proving unrealizability. Given a SyGuS problem and a finite set of inputs , we denote with the corresponding SyGuS problem that only requires the function to be correct on the examples in .

Lemma 1 (Soundness)

If is unrealizable then is unrealizable.

Even when given a perfect synthesizer —i.e., one that can solve a problem for every possible set —there are SyGuS problems for which the CEGIS algorithm is not powerful enough to prove unrealizability.

Lemma 2 (Incompleteness)

There exists an unrealizable SyGuS problem such that for every finite set of examples the problem is realizable.

Despite this negative result, we will show that a CEGIS algorithm can prove unrealizability for many SyGuS instances (evaluation).

4 From Unrealizability to Unreachability

In this section, we show how a SyGuS problem for finitely many examples can be reduced to a reachability problem in a non-deterministic, recursive program in an imperative programming language.

4.1 Reachability Problems

A program takes an initial state as input and outputs a final state , i.e., where denotes the semantic function of the programming language. As illustrated in IllustrativeExample, we allow a program to contain calls to an external function nd(), which returns a non-deterministically chosen Boolean value. When program contains calls to nd(), we use to denote the program that is the same as except that takes an additional integer input n, and each call nd() is replaced by a call to a local function nextbit() defined as follows:

In other words, the integer parameter n of formalizes all of the non-deterministic choices made by in calls to nd().

For the programs used in our unrealizability algorithm, the only calls to nd() are ones that control whether or not a production is selected from grammar during a top-down, left-to-right generation of an expression-tree. Given n, we can decode it to identify which expression-tree n represents.

Example 1

Consider again the SyGuS problem discussed in IllustrativeExample. In the discussion of the initial program (cexample0), we hypothesized that the program analyzer chose to report path (LABEL:Eq:PathPlusXOne0) in , for which the sequence of non-deterministic choices is . That sequence means that for , the value of n is (or ). The s, from low-order to high-order position, represent choices of production instances in a top-down, left-to-right generation of an expression-tree. (The s represent rejected possible choices.) The rightmost in n corresponds to the choice in PlusBegin0 of “Start ::= Plus(Start, Start)”; the in the third-from-rightmost position corresponds to the choice in StartX0 of “Start ::= x” as the left child of the Plus node; and the in the leftmost position corresponds to the choice in StartOne0 of “Start ::= 1” as the right child. By this means, we learn that the behavioral specification holds for the example set for . ∎

Definition 3 (Reachability Problem)

Given a program , containing assertion statements and a non-deterministic integer input n, we use to denote the corresponding reachability problem. The reachability problem is satisfiable if there exists a value that, when bound to n, falsifies any of the assertions in . The problem is unsatisfiable otherwise.

4.2 Reduction to Reachability

The main component of our framework is an encoding enc that given a SyGuS problem over a set of examples , outputs a program such that is realizable if and only if is satisfiable. In this section, we define all the components of , and state the correctness properties of our reduction.

Remark: In this section, we assume that in the specification every occurrence of has as input parameter. We show how to overcome this restriction in complex-invocations. In the following, we assume that the input has type , where could be a complex type—e.g., a tuple type.

Program construction. Recall that the grammar is a tuple . First, for each non-terminal , the program contains global variables of type that are used to express the values resulting from evaluating expressions generated from non-terminal on the examples. Second, for each non-terminal , the program contains a function

We denote by the set of production rules of the form in . The body of funcA has the following structure:

The encoding of a production is defined as follows ( denotes the type of the term ):

Note that if is of arity —i.e., if —the construction yields assignments of the form .

The function interprets the semantics of on the input example. We take Linear Integer Arithmetic as an example to illustrate how works.

We now turn to the correctness of the construction. First, we formalize the relationship between expression-trees in , the semantics of , and the number n. Given an expression-tree , we assume that each node in is annotated with the production that has produced that node. Recall that is the set of productions with head (where the subscripts are indexes in some arbitrary, but fixed order). Concretely, for every node , we assume there is a function , which associates with a pair that indicates that non-terminal produced using the production (i.e., is the production whose left-hand-side non-terminal is ).

We now define how we can extract a number for which the program will exhibit the same semantics as that of the expression-tree . First, for every node in such that , we define the following number:

The number indicates what suffix of the value of n will cause funcA to trigger the code corresponding to production . Let be the sequence of nodes visited during a pre-order traversal of expression-tree . The number corresponding to , denoted by

, is defined as the bit-vector

.

Finally, we add the entry-point of the program, which calls the function funcS corresponding to the initial non-terminal , and contains the assertion that encodes our reachability problem on all the input examples .

Correctness. We first need to show that the function captures the correct language of expression-trees. Given a non-terminal , a value , and input values , we use to denote the values of the variables at the end of the execution of with the initial value of and input values . Given a non-terminal , we write to denote the set of terms that can be derived starting with .

Lemma 3

Let be a non-terminal, an expression, and an input set. Then, .

Each procedure that we construct has an explicit dependence on variable n, where n controls the non-deterministic choices made by the funcA and procedures called by funcA. As a consequence, when relating numbers and expression-trees, there are two additional issues to contend with:

Non-termination.

Some numbers can cause to fail to terminate. For instance, if the case for “Start ::= Plus(Start, Start)” in program from cexample0 were moved from the first branch (PlusBegin0PlusEnd0) to the final else case (StartZero0), the number would cause Start to never terminate, due to repeated selections of Plus nodes. However, note that the only assert statement in the program is placed at the end of the main procedure. Now, consider a value of such that is satisfiable. reachability implies that the flow of control will reach and falsify the assertion, which implies that terminates. 444If the SyGuS problem deals with the synthesis of programs for a language that can express non-terminating programs, that would be an additional source of non-termination, different from that discussed in item Non-termination. That issue does not arise for LIA SyGuS problems. Dealing with the more general kind of non-termination is postponed for future work.

Shared suffixes of sufficient length.

In TermDecoding, we showed how for program (cexample0) the number corresponds to the top-down, left-to-right generation of Plus(x,1). That derivation consumed exactly seven bits; thus, any number that, written in , shares the suffix —e.g., —will also generate Plus(x,1).

The issue of shared suffixes is addressed in the following lemma:

Lemma 4

For every non-terminal and number n such that (i.e., funcA terminates when the non-deterministic choices are controlled by n), there exists a minimal that is a () suffix of n for which (i) there is an such that , and (ii) for every input , we have .

We are now ready to state the correctness property of our construction.

Theorem 4.1

Given a SyGuS problem over a finite set of examples , the problem is realizable iff is satisfiable.

5 Implementation and Evaluation

nope is a tool that can return two-sided answers to unrealizability problems of the form . When it returns unrealizable, no expression-tree in satisfies ; when it returns realizable, some satisfies ; nope can also time out. nope incorporates several existing pieces of software.

  1. The (un)reachability verifier SeaHorn is applied to the reachability problems of the form created during successive CEGIS rounds.

  2. The SMT solver Z3 is used to check whether a generated expression-tree satisfies . If it does, nope returns realizable (along with ); if it does not, nope creates a new input example to add to .

It is important to observe that SeaHorn, like most reachability verifiers, is only sound for unsatisfiability—i.e., if SeaHorn returns unsatisfiable, the reachability problem is indeed unsatisfiable. Fortunately, SeaHorn’s one-sided answers are in the correct direction for our application: to prove unrealizability, nope only requires the reachability verifier to be sound for unsatisfiability.

There is one aspect of nope that differs from the technique that has been presented earlier in the paper. While SeaHorn is sound for unreachability, it is not sound for reachability—i.e., it cannot soundly prove whether a synthesis problem is realizable. To address this problem, to check whether a given SyGuS problem is realizable on the finite set of examples , nope also calls the SyGuS solver ESolver [2] to synthesize an expression-tree that satisfies .555We chose ESolver because on the benchmarks we considered, ESolver outperformed other SyGuS solvers (e.g., CVC4 [3]).

In practice, for every intermediate problem generated by the CEGIS algorithm, nope runs the ESolver on and SeaHorn on in parallel. If ESolver returns a solution , SeaHorn is interrupted, and Z3 is used to check whether satisfies . Depending on the outcome, nope either returns realizable or obtains an additional input example to add to . If SeaHorn returns unsatisfiable, nope returns unrealizable.

Modulo bugs in its constituent components, nope is sound for both realizability and unrealizability, but because of Lemma 2 and the incompleteness of SeaHorn, nope is not complete for unrealizability.

Benchmarks. We perform our evaluation on 132 variants of the 60 LIA benchmarks from the LIA SyGuS competition track [2]. We do not consider the other SyGuS benchmark track, Bit-Vectors, because the SeaHorn verifier is unsound for most bit-vector operations–e.g., bit-shifting. We used three suites of benchmarks. LimitedIf (resp. LimitedPlus) contains 57 (resp. 30) benchmarks in which the grammar bounds the number of times an IfThenElse (resp. Plus) operator can appear in an expression-tree to be less than the number required to solve the original synthesis problem. We used the tool Quasi to automatically generate the restricted grammars. LimitedConst

contains 45 benchmarks in which the grammar allows the program to contain only constants that are coprime to any constants that may appear in a valid solution—e.g., the solution requires using odd numbers, but the grammar only contains the constant

. The numbers of benchmarks in the three suites differ because for certain benchmarks it did not make sense to create a limited variant—e.g., if the smallest program consistent with the specification contains no IfThenElse operators, no variant is created for the LimitedIf benchmark. In all our benchmarks, the grammars describing the search space contain infinitely many terms.

Our experiments were performed on an Intel Core i7 4.00GHz CPU, with 32GB of RAM, running Lubuntu 18.10 via VirtualBox. We used version 4.8 of Z3, commit 97f2334 of SeaHorn, and commit d37c50e of ESolver. The timeout for each individual SeaHorn/ESolver call is set at 10 minutes.

Experimental Questions. Our experiments were designed to answer the questions posed below.

[ breakable, left=0pt, right=0pt, top=0pt, bottom=-1pt, colback=gray!20, colframe=gray!20, width=boxsep=4pt, arc=0pt,outer arc=0pt, ] EQ 1. Can nope prove unrealizability for variants of real SyGuS benchmarks, and how long does it take to do so?

Finding: nope can prove unrealizability for benchmarks. For the benchmarks solved by nope, the average time taken is s. The time taken to perform the last iteration of the algorithm—i.e., the time taken by SeaHorn to return unsatisfiable—accounts for of the total running time.

nope can solve all of the LimitedIf benchmarks for which the grammar allows at most one IfThenElse operator. Allowing more IfThenElse operators in the grammar leads to larger programs and larger sets of examples, and consequently the resulting reachability problems are harder to solve for SeaHorn.

For a similar reason, nope can solve only one of the LimitedPlus benchmarks. All other LimitedPlus benchmarks allow or more Plus statements, resulting in grammars that have at least productions.

nope can solve all LimitedConst benchmarks because these require few examples and result in small encoded programs.

[ breakable, left=0pt, right=0pt, top=0pt, bottom=-1pt, colback=gray!20, colframe=gray!20, width=boxsep=4pt, arc=0pt,outer arc=0pt, ] EQ 2. How many examples does nope use to prove unrealizability and how does the number of examples affect the performance of nope? Note that Z3 can produce different models for the same query, and thus different runs of NOPE can produce different sequences of example. Hence, there is no guarantee that NOPE finds a good sequence of examples that prove unrealizability. One measure of success is whether nope is generally able to find a small number of examples, when it succeeds in proving unrealizability.

Finding: Nope used 1 to 9 examples to prove unrealizability for the benchmarks on which it terminated. Problems requiring large numbers of examples could not be solved because either ESolver or SeaHorn timeouts—e.g., on the problem max4, nope gets to the point where the CEGIS loop has generated 17 examples, at which point ESolver exceeds the timeout threshold.

examples

time (s)
Figure 1: Time vs examples.

Finding: The number of examples required to prove unrealizability depends mainly on the arity of the synthesized function and the complexity of the grammar. The number of examples seems to grow quadratically with the number of bounded operators allowed in the grammar. In particular, problems in which the grammar allows zero IfThenElse operators require 2–4 examples, while problems in which the grammar allows one IfThenElse operator require 7–9 examples.

Figure 1 plots the running time of nope against the number of examples generated by the CEGIS algorithm. Finding: The solving time appears to grow exponentially with the number of examples required to prove unrealizability.

6 Related Work

The SyGuS formalism was introduced as a unifying framework to express several synthesis problems [1]. Caulfield et al. [6] proved that it is undecidable to determine whether a given SyGuS problem is realizable. Despite this negative result, there are several SyGuS solvers that compete in yearly SyGuS competitions [2] and can efficiently produce solutions to SyGuS problems when a solution exists. Existing SyGuS synthesizers fall into three categories: (i) Enumeration solvers enumerate programs with respect to a given total order [7]. If the given problem is unrealizable, these solvers typically only terminate if the language of the grammar is finite or contains finitely many functionally distinct programs. While in principle certain enumeration solvers can prune infinite portions of the search space, none of these solvers could prove unrealizability for any of the benchmarks considered in this paper. (ii) Symbolic solvers reduce the synthesis problem to a constraint-solving problem [3]. These solvers cannot reason about grammars that restrict allowed terms, and resort to enumeration whenever the candidate solution produced by the constraint solver is not in the restricted search space. Hence, they also cannot prove unrealizability. (iii) Probabilistic synthesizers randomly search the search space, and are typically unpredictable [14], providing no guarantees in terms of unrealizability.

Synthesis as Reachability. CETI [12] introduces a technique for encoding template-based synthesis problems as reachability problems. The CETI encoding only applies to the specific setting in which (i) the search space is described by an imperative program with a finite number of holes—i.e., the values that the synthesizer has to discover—and (ii) the specification is given as a finite number of input-output test cases with which the target program should agree. Because the number of holes is finite, and all holes correspond to values (and not terms), the reduction to a reachability problem only involves making the holes global variables in the program (and no more elaborate transformations).

In contrast, our reduction technique handles search spaces that are described by a grammar, which in general consist of an infinite set of terms (not just values). Due to this added complexity, our encoding has to account for (i) the semantics of the productions in the grammar, and (ii) the use of non-determinism to encode the choice of grammar productions. Our encoding creates one expression-evaluation computation for each of the example inputs, and threads these computations through the program so that each expression-evaluation computation makes use of the same set of non-deterministic choices.

Using the input-threading, our technique can handle specifications that contain nested calls of the synthesized program (e.g., ). ( complex-invocations.)

The input-threading technique builds a product program that perform multiple executions of the same function as done in relational program verification [4]. Alternatively, a different encoding could use multiple function invocations on individual inputs and require the verifier to thread the same bit-stream for all input evaluations. In general, verifiers perform much better on product programs [4], which motivates our choice of encoding.

Unrealizability in Program Synthesis. For certain synthesis problems—e.g., reactive synthesis [5]—the realizability problem is decidable. The framework tackled in this paper, SyGuS, is orthogonal to such problems, and it is undecidable to check whether a given SyGuS problem is realizable [6].

Mechtaev et al. [11] propose to use a variant of SyGuS to efficiently prune irrelevant paths in a symbolic-execution engine. In their approach, for each path in the program, a synthesis problem is generated so that if is unrealizable, the path is infeasible. The synthesis problems generated by Mechtaev et al. (which are not directly expressible in SyGuS) are decidable because the search space is defined by a finite set of templates, and the synthesis problem can be encoded by an SMT formula. To the best of our knowledge, our technique is the first one that can check unrealizability of general SyGuS problems in which the search space is an infinite set of functionally distinct terms.

Acknowledgment This work was supported, in part, by a gift from Rajiv and Ritu Batra; by AFRL under DARPA MUSE award FA8750-14-2-0270 and DARPA STAC award FA8750-15-C-0082; by ONR under grant N00014-17-1-2889; by NSF under grants CNS-1763871 and CCF-1704117; and by the UW-Madison OVRGE with funding from WARF.

References

  • [1] R. Alur, R. Bodik, G. Juniwal, M. M. Martin, M. Raghothaman, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa. Syntax-guided synthesis. In Formal Methods in Computer-Aided Design (FMCAD), pages 1–8. IEEE, 2013.
  • [2] R. Alur, D. Fisman, R. Singh, and A. Solar-Lezama. Sygus-comp 2016: results and analysis. arXiv preprint arXiv:1611.07627, 2016.
  • [3] C. Barrett, C. L. Conway, M. Deters, L. Hadarean, D. Jovanović, T. King, A. Reynolds, and C. Tinelli. Cvc4. In International Conference on Computer Aided Verification, (CAV), pages 171–177. Springer-Verlag, 2011.
  • [4] G. Barthe, J. M. Crespo, and C. Kunz. Relational verification using product programs. In International Symposium on Formal Methods (FM), pages 200–214. Springer, 2011.
  • [5] R. Bloem. Reactive synthesis. In Formal Methods in Computer-Aided Design (FMCAD), pages 3–3, 2015.
  • [6] B. Caulfield, M. N. Rabe, S. A. Seshia, and S. Tripakis. What’s decidable about syntax-guided synthesis? arXiv preprint arXiv:1510.08393, 2015.
  • [7] ESolver. https://github.com/abhishekudupa/sygus-comp14.
  • [8] A. Gurfinkel, T. Kahsai, and J. A. Navas. SeaHorn: A framework for verifying C programs (competition contribution). In Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pages 447–450, 2015.
  • [9] Q. Hu and L. D’Antoni. Automatic program inversion using symbolic transducers. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation, (PLDI), pages 376–389, 2017.
  • [10] Q. Hu and L. D’Antoni. Syntax-guided synthesis with quantitative syntactic objectives. In Computer Aided Verification - 30th International Conference, (CAV), pages 386–403, 2018.
  • [11] S. Mechtaev, A. Griggio, A. Cimatti, and A. Roychoudhury. Symbolic execution with existential second-order constraints. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 389–399, 2018.
  • [12] T. Nguyen, W. Weimer, D. Kapur, and S. Forrest. Connecting program synthesis and reachability: Automatic program repair using test-input generation. In A. Legay and T. Margaria, editors, Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pages 301–318, 2017.
  • [13] H. Qinheping, B. Jason, C. John, D. Loris, and T. Reps. Proving unrealizability for syntax-guided synthesis. arXiv preprint arXiv:1905.05800, 2019.
  • [14] E. Schkufza, R. Sharma, and A. Aiken. Stochastic program optimization. Commun. ACM, 59(2):114–122, 2016.

Appendix 0.A Additional Material

0.a.1 Encoding in the Presence of Nested Function-Invocations

In EncodingAlgorithm, we presented a simplified encoding that relied on the specification to only involve function invocations of the form , where represents the input parameter of the function to be synthesized. In this section, we show with a simple example how such a restriction can be overcome.

Consider the following semantic specification that involves multiple invocations of the function on different arguments, as well as nested function calls:

By introducing new input variables and performing the proper refactoring, we can rewrite as the following specification, where is always called on a single input variable:

It is now easy to adapt our encoding to operate over this new specification. First, the program will now operate over input examples of the form , where each example is a tuple corresponding to the values of variables . Second, the program will need to compute the values of all possible calls of on the various input parameters. Hence, for every expression in