Formal verification can be a time-consuming task that requires significant manual effort. Especially for complex systems, users often need to manually provide, for example, loop invariants, function summaries, or environment models. Synthesis has the potential to alleviate some of this manual burden (Seshia, ). For example, prior work has used synthesis to reason about program loops (David et al., 2015), and to automate program repair (Le et al., 2017). We believe this is a promising direction, but, for it to make a real impact, verification tools need to offer flexible synthesis integration, generic support for proof procedures, and a capable synthesis engine back-end.
In this work, we primarily address the first two requirements. Specifically, we integrate program synthesis into the Uclid5 (Seshia and Subramanyan, 2018) formal modelling and verification tool by allowing users to declare functions to synthesize and to use these functions freely. While Uclid5 has previously supported use of synthesis, it only supported invariant synthesis through a special command that was independent of verification. We use the new synthesis integration to generate 25 benchmarks from existing verification tasks. These benchmarks have small solutions, but are out of reach for current synthesis engines. We hope that they will help the synthesis engine development effort, particularly for syntax-guided synthesis (Alur et al., 2013).
Consider the Uclid5 model in Fig. 1, which represents a Fibonacci sequence. The (hypothetical) user wants to prove by induction that the invariant a_le_b at line 13 always holds. Unfortunately, the proof fails because the invariant is not inductive. Without synthesis, the user would need to manually strengthen the invariant until it became inductive. However, the user can ask Uclid5 to automatically do this for them. Fig. 1 demonstrates this on lines 16, 17 and 18. Specifically, the user specifies a function to synthesize called h at lines 16 and 17, and then uses h at line 18 to strengthen the existing set of invariants. Given this input, Uclid5, using e.g. cvc4 (Barrett et al., 2011) as a synthesis engine, will automatically generate the function h(x, y) = x >= 0, which completes the inductive proof.
In this example, the function to synthesize represents an inductive invariant. However, functions to synthesize are treated exactly like any interpreted function in Uclid5: the user could have called h anywhere in the code. Furthermore, this example uses induction and a global invariant, however, the user could also have used a linear temporal logic (LTL) specification and bounded model checking (BMC). In this sense, our integration is fully flexible and generic.
We present an integration of synthesis into the verification tool Uclid5, allowing users to generate program synthesis queries for unknown parts of a system they wish to verify. The integration is a natural extension of the existing Uclid5 language, and to the best of our knowledge, is the first to support program synthesis with bounded model checking, k-induction, sequential program verification, and hyperproperty verification. The synthesis queries Uclid5 generates are in the standard sygus-if (Raghothaman and Udupa, 2014) specification language. We use this tool to generate a 25 sygus-if synthesis benchmarks from existing verification queries and release these benchmarks to the community.
2. Related work
Program sketching (Solar-Lezama, 2009) synthesizes expressions to fill holes in programs, and has subsequently been applied to program repair (Le et al., 2017; Hua et al., 2018). Uclid5 aims to be more flexible than this work, allowing users to declare unknown functions even in the verification annotations, as well as supporting multiple verification algorithms and types of properties. Rosette (Torlak and Bodík, 2013) provides support for synthesis and verification, but the synthesis is limited to bounded specifications of sequential programs, whereas Uclid5 can also synthesize programs that satisfy unbounded specifications, by using proof procedures like induction. Formal synthesis algorithms have been used to assist in verification tasks, such as safety and termination of loops (David et al., 2015), and generating invariants (Fedyukovich and Bodík, 2018; Zhang et al., 2020), but none of this work to-date integrates program synthesis fully into an existing verification tool. Before this new synthesis integration, Uclid5 supported synthesis of inductive invariants. The key insight of this work is to generalize the synthesis support, and to unify all synthesis tasks in Uclid5 by re-using the verification back-end.
3. From Verification to Program Synthesis
In this section, we give the necessary background on program synthesis, and the existing verification techniques inside of Uclid5. We then describe how we combine the two to realize synthesis in Uclid5.
3.1. Program Synthesis
The program synthesis problem corresponds to the second-order query
where is the function to synthesize, is the set of all possible inputs, and is the specification to be satisfied.
3.2. Verification in Uclid5
At at high level, Uclid5 takes in a model, generates a set of verification conditions, asks a satisfiability modulo theory (SMT) solver (Barrett et al., 2009) to check the verification conditions, and then returns the results to the user. This process is the same regardless of the proof procedure used. The important point, is that Uclid5 encodes the violation of each independent verification condition as a separate smt-lib query.
Let encode the verification condition, and take the smt-lib query to be checking the validity of , where contains no free variables. We say that there is a counterexample to the verification query if the query is valid. Verification of a model with verification conditions succeeds iff there are no counter-examples:
3.3. Synthesis Encoding in Uclid5
Given a Uclid5 model in which the user has declared a function to synthesize, , we wish to construct a synthesis query that is satisfied iff there is an for which all the verification conditions pass for all possible inputs. We build this synthesis query by taking the conjunction of the negation of all the verification queries. Specifically, we check the validity of
where each encodes a verification condition that may refer to the function to synthesize, . Note the similarity between this query and the standard program synthesis formulation: the specification for synthesis, , from the equation in Sec. 3.1, is now replaced with the conjunction of all the verification conditions. With this observation, to enable synthesis for any verification procedure in Uclid5, all we do is let users declare and use functions to synthesize.
The Uclid5 verification tool is constructed as shown in Figure 2. An input Uclid5 model is parsed by the front-end into an abstract syntax tree. From this abstract syntax tree, a symbolic simulator generates an assertion stack that contains an assertion for each verification condition. Prior to our work, assertions were then passed to an smt-lib interface which converted the assertions to smt-lib and called a solver. The new Uclid5 instead uses a new intermediate representation, synth-lib, that is easily passed to either an SMT solver or a synthesis engine. This architecture allows us to use the same code that generates verification queries for synthesis.
where is the name of the function, is the name of an argument, is the sort of the corresponding argument, there are zero or more arguments, is the sort returned by the function, and is an optional syntactic specification for the function body. Intuitively, a synth-lib query with a single synth-blocking -fun declaration asks “is there a function that makes this underlying smt-lib query unsatisfiable?”
Fig. 3 shows the synth-lib query corresponding to the Fibonacci model in Fig. 1. A synthesis engine might solve the query in Fig. 3 by finding the function h(x, y) = x >= 0. This is a correct solution because the corresponding smt-lib query—which we can get by commenting out line 1 of Fig. 3 and uncommenting line 2—is unsatisfiable.
The semantics of synth-lib is exactly that of smt-lib when no function to synthesize is on the assertion stack, and assertions are passed directly to the SMT solver. When the assertion stack contains a function to synthesize, Uclid5 applies the following four rewrite rules to convert synth-lib into sygus-if:
(assert ) (constraint (not ))
(declare-fun () (declare-var -¿…-¿ )
The first rewrite rule is the most important: it implements the following equivalence
where the left hand side is the form of queries in synth-lib, and the right hand side is the corresponding query in sygus-if. The source code for Uclid5 is available online (Seshia and Subramanyan, 2020).
5. Benchmark Suite
The integration of synthesis into Uclid5 allows us to generate synthesis benchmarks from any Uclid5 verification task. We thus present a set of 25 benchmarks with known, small solutions that are out of reach of existing synthesis solvers. These benchmarks use induction, BMC, LTL specifications, and sequential code. To conform to the sygus-if
language, we limited ourselves to bit-vector, integer, array, and boolean data-types, and did not use verification tasks that required quantifiers. All benchmarks are available online(Mora, 2020).
The benchmarks come from four different sources. Four benchmarks come from a simplified model of the Two Phase Commit protocol, written in P (Desai et al., 2013); three benchmarks come from Sahai at al’s (Sahai et al., 2020) work on hyperproperty verification; six benchmarks come from Uclid5’s documentation; and the remaining 12 benchmarks come from models used in UC Berkeley’s EECS 219C course. In all cases, we constructed the benchmarks by replacing small parts of either auxiliary invariants or parts of existing code with functions to synthesize. 12 benchmarks come from models that use induction, and 13 from models that use LTL specifications and BMC. All 25 benchmarks are difficult for existing state-of-the-art engines, but are a reasonable target for synthesis engines.
6. Conclusions and Future Work
We have presented an integration of synthesis into the Uclid5 verification tool, allowing users to generate synthesis queries for unknown parts of a system they wish to verify. This integration is compatible with all verification algorithms currently supported by Uclid5, and generates synthesis queries in the standard sygus-if format.
In the future, we intend to apply synthesis in Uclid5 to the verification of distributed systems written in P. Prior work has been successfully in finding invariants for bounded distributed systems, and then generalizing the invariants to the unbounded setting (Ma et al., 2019). We plan to explore these approaches with Uclid5 now that we can easily switch between synthesis using e.g. BMC and k-induction.
Acknowledgements.This work was supported in part by NSF grants 1739816 and 1837132, a gift from Intel under the SCAP program, SRC Task 2867.001, and the iCyPhy center.
- Syntax-guided synthesis. In FMCAD, pp. 1–17. Cited by: §1.
- CVC4. In CAV, pp. 171–177. Cited by: §1.
- The Satisfiability Modulo Theories Library (SMT-LIB). Note: www.SMT-LIB.org Cited by: §4.
- Satisfiability modulo theories. In Handbook of Satisfiability, A. Biere, H. van Maaren, and T. Walsh (Eds.), Vol. 4. Cited by: §3.2.
- Using program synthesis for program analysis. In LPAR, pp. 483–498. Cited by: §1, §2.
- P: safe asynchronous event-driven programming. PLDI, pp. 321–332. Cited by: §5.
- Accelerating syntax-guided invariant synthesis. In TACAS (1), pp. 251–269. Cited by: §2.
- Towards practical program repair with on-demand candidate generation. In ICSE, pp. 12–23. Cited by: §2.
- S3: syntax- and semantic-guided repair synthesis via programming by examples. In ESEC/SIGSOFT FSE, pp. 593–604. Cited by: §1, §2.
- I4: incremental inference of inductive invariants for verification of distributed protocols. In OSDI, pp. 370–384. Cited by: §6.
- UCLID5 Synthesis Benchmarks. External Links: Cited by: §5.
- Language to Specify Syntax-Guided Synthesis Problems. Note: https://sygus.org/assets/pdf/SyGuS-IF.pdf Cited by: §1, §4.
- Verification of quantitative hyperproperties using trace enumeration relations. In CAV, Cited by: §5.
-  Combining induction, deduction, and structure for verification and synthesis. Cited by: §1.
- UCLID5: integrating modeling, verification, synthesis, and learning. In MEMOCODE, Cited by: §1.
- UCLID5: a system for modeling, verification, and synthesis of computational systems. External Links: Cited by: §4.
- The sketching approach to program synthesis. In Asian Symposium on Programming Languages and Systems, pp. 4–13. Cited by: §2.
- Growing solver-aided languages with rosette. In Onward!, pp. 135–152. Cited by: §2.
- Synthesizing environment invariants for modular hardware verification. In VMCAI, pp. 202–225. Cited by: §2.