Synthesis in Uclid5

07/14/2020
by   Federico Mora, et al.
0

We describe an integration of program synthesis into Uclid5, a formal modelling and verification tool. To the best of our knowledge, the new version of Uclid5 is the only tool that supports program synthesis with bounded model checking, k-induction, sequential program verification, and hyperproperty verification. We use the integration to generate 25 program synthesis benchmarks with simple, known solutions that are out of reach of current synthesis engines, and we release the benchmarks to the community.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

09/12/2018

Proceedings 5th Workshop on Horn Clauses for Verification and Synthesis

Many Program Verification and Synthesis problems of interest can be mode...
03/17/2021

Toward Neural-Network-Guided Program Synthesis and Verification

We propose a novel framework of program and invariant synthesis called n...
02/07/2018

Recent Advances in Neural Program Synthesis

In recent years, deep learning has made tremendous progress in a number ...
11/29/2017

CTL* synthesis via LTL synthesis

We reduce synthesis for CTL* properties to synthesis for LTL. In the con...
01/08/2019

Computational Register Analysis and Synthesis

The study of register in computational language research has historicall...
12/30/2020

Solving Interactive Fiction Games via Partial Evaluation and Bounded Model Checking

We present a case study on using program verification tools, specificall...
07/16/2021

Enhancing Loop-Invariant Synthesis via Reinforcement Learning

Loop-invariant synthesis is the basis of every program verification proc...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Formal verification can be a time-consuming task that requires significant manual effort. Especially for complex systems, users often need to manually provide, for example, loop invariants, function summaries, or environment models. Synthesis has the potential to alleviate some of this manual burden (Seshia, ). For example, prior work has used synthesis to reason about program loops (David et al., 2015), and to automate program repair (Le et al., 2017). We believe this is a promising direction, but, for it to make a real impact, verification tools need to offer flexible synthesis integration, generic support for proof procedures, and a capable synthesis engine back-end.

In this work, we primarily address the first two requirements. Specifically, we integrate program synthesis into the Uclid5 (Seshia and Subramanyan, 2018) formal modelling and verification tool by allowing users to declare functions to synthesize and to use these functions freely. While Uclid5 has previously supported use of synthesis, it only supported invariant synthesis through a special command that was independent of verification. We use the new synthesis integration to generate 25 benchmarks from existing verification tasks. These benchmarks have small solutions, but are out of reach for current synthesis engines. We hope that they will help the synthesis engine development effort, particularly for syntax-guided synthesis (Alur et al., 2013).

Illustrative Example

Consider the Uclid5 model in Fig. 1, which represents a Fibonacci sequence. The (hypothetical) user wants to prove by induction that the invariant a_le_b at line 13 always holds. Unfortunately, the proof fails because the invariant is not inductive. Without synthesis, the user would need to manually strengthen the invariant until it became inductive. However, the user can ask Uclid5 to automatically do this for them. Fig. 1 demonstrates this on lines 16, 17 and 18. Specifically, the user specifies a function to synthesize called h at lines 16 and 17, and then uses h at line 18 to strengthen the existing set of invariants. Given this input, Uclid5, using e.g. cvc4 (Barrett et al., 2011) as a synthesis engine, will automatically generate the function h(x, y) = x >= 0, which completes the inductive proof.

In this example, the function to synthesize represents an inductive invariant. However, functions to synthesize are treated exactly like any interpreted function in Uclid5: the user could have called h anywhere in the code. Furthermore, this example uses induction and a global invariant, however, the user could also have used a linear temporal logic (LTL) specification and bounded model checking (BMC). In this sense, our integration is fully flexible and generic.

Contributions

We present an integration of synthesis into the verification tool Uclid5, allowing users to generate program synthesis queries for unknown parts of a system they wish to verify. The integration is a natural extension of the existing Uclid5 language, and to the best of our knowledge, is the first to support program synthesis with bounded model checking, k-induction, sequential program verification, and hyperproperty verification. The synthesis queries Uclid5 generates are in the standard sygus-if (Raghothaman and Udupa, 2014) specification language. We use this tool to generate a 25 sygus-if synthesis benchmarks from existing verification queries and release these benchmarks to the community.

2. Related work

Program sketching (Solar-Lezama, 2009) synthesizes expressions to fill holes in programs, and has subsequently been applied to program repair (Le et al., 2017; Hua et al., 2018). Uclid5 aims to be more flexible than this work, allowing users to declare unknown functions even in the verification annotations, as well as supporting multiple verification algorithms and types of properties. Rosette (Torlak and Bodík, 2013) provides support for synthesis and verification, but the synthesis is limited to bounded specifications of sequential programs, whereas Uclid5 can also synthesize programs that satisfy unbounded specifications, by using proof procedures like induction. Formal synthesis algorithms have been used to assist in verification tasks, such as safety and termination of loops (David et al., 2015), and generating invariants (Fedyukovich and Bodík, 2018; Zhang et al., 2020), but none of this work to-date integrates program synthesis fully into an existing verification tool. Before this new synthesis integration, Uclid5 supported synthesis of inductive invariants. The key insight of this work is to generalize the synthesis support, and to unify all synthesis tasks in Uclid5 by re-using the verification back-end.

3. From Verification to Program Synthesis

In this section, we give the necessary background on program synthesis, and the existing verification techniques inside of Uclid5. We then describe how we combine the two to realize synthesis in Uclid5.

1module main {
2  // Part 1: System Description.
3  var a, b : integer;
4  init {
5    a, b = 0, 1;
6  }
7  next {
8    a’, b = b, a + b;
9  }
10
11  // Part 2: System Specification.
12  invariant a_le_b: a <= b;
13
14  // Part 3: (NEW) Synthesis Integration
15  synthesis function
16    h(x : integer, y : integer): boolean;
17  invariant hole: h(a, b);
18
19  // Part 4: Proof Script.
20  control {
21    induction;
22    check;
23    print_results;
24  }
25}
Figure 1. Uclid5 Fibonacci model. Part 3 shows the new synthesis syntax, and how to find an auxiliary invariant.

3.1. Program Synthesis

The program synthesis problem corresponds to the second-order query

where is the function to synthesize, is the set of all possible inputs, and is the specification to be satisfied.

3.2. Verification in Uclid5

At at high level, Uclid5 takes in a model, generates a set of verification conditions, asks a satisfiability modulo theory (SMT) solver (Barrett et al., 2009) to check the verification conditions, and then returns the results to the user. This process is the same regardless of the proof procedure used. The important point, is that Uclid5 encodes the violation of each independent verification condition as a separate smt-lib query.

Let encode the verification condition, and take the smt-lib query to be checking the validity of , where contains no free variables. We say that there is a counterexample to the verification query if the query is valid. Verification of a model with verification conditions succeeds iff there are no counter-examples:

3.3. Synthesis Encoding in Uclid5

Given a Uclid5 model in which the user has declared a function to synthesize, , we wish to construct a synthesis query that is satisfied iff there is an for which all the verification conditions pass for all possible inputs. We build this synthesis query by taking the conjunction of the negation of all the verification queries. Specifically, we check the validity of

where each encodes a verification condition that may refer to the function to synthesize, . Note the similarity between this query and the standard program synthesis formulation: the specification for synthesis, , from the equation in Sec. 3.1, is now replaced with the conjunction of all the verification conditions. With this observation, to enable synthesis for any verification procedure in Uclid5, all we do is let users declare and use functions to synthesize.

4. Implementation

The Uclid5 verification tool is constructed as shown in Figure 2. An input Uclid5 model is parsed by the front-end into an abstract syntax tree. From this abstract syntax tree, a symbolic simulator generates an assertion stack that contains an assertion for each verification condition. Prior to our work, assertions were then passed to an smt-lib interface which converted the assertions to smt-lib and called a solver. The new Uclid5 instead uses a new intermediate representation, synth-lib, that is easily passed to either an SMT solver or a synthesis engine. This architecture allows us to use the same code that generates verification queries for synthesis.

front-end

Symbolic Simulator

synth-lib interface

smt-lib interface

sygus-if interface

Figure 2. Overview of synthesis in Uclid5. Dashed blocks indicate blocks introduced for the new synthesis integration.

The synth-lib representation is smt-lib (Barrett et al., 2016), but with one extra command borrowed from sygus-if (Raghothaman and Udupa, 2014). The syntax for the new command is

where is the name of the function, is the name of an argument, is the sort of the corresponding argument, there are zero or more arguments, is the sort returned by the function, and is an optional syntactic specification for the function body. Intuitively, a synth-lib query with a single synth-blocking -fun declaration asks “is there a function that makes this underlying smt-lib query unsatisfiable?”

Fig. 3 shows the synth-lib query corresponding to the Fibonacci model in Fig. 1. A synthesis engine might solve the query in Fig. 3 by finding the function h(x, y) = x >= 0. This is a correct solution because the corresponding smt-lib query—which we can get by commenting out line 1 of Fig. 3 and uncommenting line 2—is unsatisfiable.

1(synth-blocking-fun h ((x Int) (y Int)) Bool)
2;(define-fun h ((x Int) (y Int)) Bool (>= x 0))
3(declare-fun initial_b () Int)
4(declare-fun initial_a () Int)
5(declare-fun new_a () Int)
6(declare-fun new_b () Int)
7(assert (or
8 (not (and (<= initial_a initial_b) (h 0 1)))
9 (and
10  (and (<= initial_a initial_b) (h initial_a initial_b))
11  (= new_a initial_b)
12  (= new_b (+ initial_a initial_b))
13  (not (and (<= new_a new_b) (h new_a new_b))))))
14(check-sat)
Figure 3. synth-lib induction query of Fig. 1

The semantics of synth-lib is exactly that of smt-lib when no function to synthesize is on the assertion stack, and assertions are passed directly to the SMT solver. When the assertion stack contains a function to synthesize, Uclid5 applies the following four rewrite rules to convert synth-lib into sygus-if:

  1. [leftmargin=*]

  2. (assert ) (constraint (not ))

  3. (declare-fun () (declare-var -¿…-¿ )

  4. synth-blocking-fun synth-fun

  5. check-sat check-synth

The first rewrite rule is the most important: it implements the following equivalence

where the left hand side is the form of queries in synth-lib, and the right hand side is the corresponding query in sygus-if. The source code for Uclid5 is available online (Seshia and Subramanyan, 2020).

5. Benchmark Suite

The integration of synthesis into Uclid5 allows us to generate synthesis benchmarks from any Uclid5 verification task. We thus present a set of 25 benchmarks with known, small solutions that are out of reach of existing synthesis solvers. These benchmarks use induction, BMC, LTL specifications, and sequential code. To conform to the sygus-if

language, we limited ourselves to bit-vector, integer, array, and boolean data-types, and did not use verification tasks that required quantifiers. All benchmarks are available online

(Mora, 2020).

The benchmarks come from four different sources. Four benchmarks come from a simplified model of the Two Phase Commit protocol, written in P (Desai et al., 2013); three benchmarks come from Sahai at al’s (Sahai et al., 2020) work on hyperproperty verification; six benchmarks come from Uclid5’s documentation; and the remaining 12 benchmarks come from models used in UC Berkeley’s EECS 219C course. In all cases, we constructed the benchmarks by replacing small parts of either auxiliary invariants or parts of existing code with functions to synthesize. 12 benchmarks come from models that use induction, and 13 from models that use LTL specifications and BMC. All 25 benchmarks are difficult for existing state-of-the-art engines, but are a reasonable target for synthesis engines.

6. Conclusions and Future Work

We have presented an integration of synthesis into the Uclid5 verification tool, allowing users to generate synthesis queries for unknown parts of a system they wish to verify. This integration is compatible with all verification algorithms currently supported by Uclid5, and generates synthesis queries in the standard sygus-if format.

In the future, we intend to apply synthesis in Uclid5 to the verification of distributed systems written in P. Prior work has been successfully in finding invariants for bounded distributed systems, and then generalizing the invariants to the unbounded setting (Ma et al., 2019). We plan to explore these approaches with Uclid5 now that we can easily switch between synthesis using e.g. BMC and k-induction.

Acknowledgements.
This work was supported in part by NSF grants 1739816 and 1837132, a gift from Intel under the SCAP program, SRC Task 2867.001, and the iCyPhy center.

References

  • R. Alur, R. Bodik, G. Juniwal, M. M. K. Martin, M. Raghothaman, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa (2013) Syntax-guided synthesis. In FMCAD, pp. 1–17. Cited by: §1.
  • C. Barrett, C. L. Conway, M. Deters, L. Hadarean, D. Jovanovic, T. King, A. Reynolds, and C. Tinelli (2011) CVC4. In CAV, pp. 171–177. Cited by: §1.
  • C. Barrett, P. Fontaine, and C. Tinelli (2016) The Satisfiability Modulo Theories Library (SMT-LIB). Note: www.SMT-LIB.org Cited by: §4.
  • C. Barrett, R. Sebastiani, S. A. Seshia, and C. Tinelli (2009) Satisfiability modulo theories. In Handbook of Satisfiability, A. Biere, H. van Maaren, and T. Walsh (Eds.), Vol. 4. Cited by: §3.2.
  • C. David, D. Kroening, and M. Lewis (2015) Using program synthesis for program analysis. In LPAR, pp. 483–498. Cited by: §1, §2.
  • A. Desai, V. Gupta, E. Jackson, S. Qadeer, S. Rajamani, and D. Zufferey (2013) P: safe asynchronous event-driven programming. PLDI, pp. 321–332. Cited by: §5.
  • G. Fedyukovich and R. Bodík (2018) Accelerating syntax-guided invariant synthesis. In TACAS (1), pp. 251–269. Cited by: §2.
  • J. Hua, M. Zhang, K. Wang, and S. Khurshid (2018) Towards practical program repair with on-demand candidate generation. In ICSE, pp. 12–23. Cited by: §2.
  • X. D. Le, D. Chu, D. Lo, C. Le Goues, and W. Visser (2017) S3: syntax- and semantic-guided repair synthesis via programming by examples. In ESEC/SIGSOFT FSE, pp. 593–604. Cited by: §1, §2.
  • H. Ma, A. Goel, J. Jeannin, M. Kapritsos, B. Kasikci, and K. A. Sakallah (2019) I4: incremental inference of inductive invariants for verification of distributed protocols. In OSDI, pp. 370–384. Cited by: §6.
  • F. Mora (2020) UCLID5 Synthesis Benchmarks. External Links: Link Cited by: §5.
  • M. Raghothaman and A. Udupa (2014) Language to Specify Syntax-Guided Synthesis Problems. Note: https://sygus.org/assets/pdf/SyGuS-IF.pdf Cited by: §1, §4.
  • S. Sahai, R. Sinha, and P. Subramanyan (2020) Verification of quantitative hyperproperties using trace enumeration relations. In CAV, Cited by: §5.
  • [14] S. A. Seshia Combining induction, deduction, and structure for verification and synthesis. Cited by: §1.
  • S. Seshia and P. Subramanyan (2018) UCLID5: integrating modeling, verification, synthesis, and learning. In MEMOCODE, Cited by: §1.
  • S. Seshia and P. Subramanyan (2020) UCLID5: a system for modeling, verification, and synthesis of computational systems. External Links: Link Cited by: §4.
  • A. Solar-Lezama (2009) The sketching approach to program synthesis. In Asian Symposium on Programming Languages and Systems, pp. 4–13. Cited by: §2.
  • E. Torlak and R. Bodík (2013) Growing solver-aided languages with rosette. In Onward!, pp. 135–152. Cited by: §2.
  • H. Zhang, W. Yang, G. Fedyukovich, A. Gupta, and S. Malik (2020) Synthesizing environment invariants for modular hardware verification. In VMCAI, pp. 202–225. Cited by: §2.