An AI problem is generally solved by defining a model equivalent to a production system, with one or more initial states, one ore more goal (final) states and some transitions rules, and by providing a search strategy which is able to find a path between an initial state and a final state within the problem space described by the model.
From the STRIPS [Fikes and Nilsson1971] to more recent efforts [Russell and Norvig2016] and [Hopgood2016], various methods to formalize AI problems (described in natural language) have been made. Although significant variations exists, a general procedure has been cemented, as described initially in [Amarel1981] and further detailed in [Russell and Norvig2016]. We follow four steps: we find a representation for a state of the problem, we identify initial and goal (final) states, we described the available transitions (and means to validate them) and we apply a search strategy within the problem space which is able to find a solution as a path between an initial and a final state. The resulting transition system can, however, be difficult to check, since the problem can hide many details (assumed data, hidden references), can be incorrectly interpreted and can have solutions difficult (or impossible) to recover. Checking whether a model can solve an AI problem has been a task considered since the original STRIPS language, with solutions ranging from mathematical demonstrations [Bylander1994] to computer aided efforts [Huang and Van Der Meyden2014] and [Edelkamp and Helmert2001]. In [Clarke et al.2001], bounded model checking is tested in real scenarios with good results, which was one of the motivations for the effort presented in this paper.
A model can allow a solution (path) to be discovered, but it can also fail to do so, even if the problem has a solution. The reasons why a model can fail are varied, starting from an insufficiently detailed representation for a state, not including all required validations and transitions, even up to not considering particular search strategies which may be unable to find a solution even if it exists. We propose validating a model by checking whether two claims (implemented as logical formulas) can be proven as valid:
Valid final state (VFS): a valid state exists which checks the conditions for a final state. This would mean that a final state exists within the problem space, as defined by the model.
Path to a final state (PFS): using valid transitions, as described in the model, a path can be found between the initial state and a final state.
If both claims are proven, a search strategy can be used to recover one or more solutions using the tested model. Checking PFS has the added benefit of providing such a solution, which could be used to choose or build a better search heuristic for this particular problem and model.
Indeed, checking these properties is not an easy task. The usual way to tackle this is to explore the solution space of the model using various strategies. In this paper we propose a different methodology for checking the desired properties of AI models. First, we encode the model using logical formulas, and thus, we keep an abstract representation of the solution space. Then, we use an existing tool called Z3 [de Moura and Bjørner2008] which is able to automatically reason about logical formulas. Reasoning at an abstract level is not as expensive as exploring the entire solution space. Finally, we analyze the results returned by Z3.
Satisfiability Modulo Theories
Satisfiability Modulo Theories, shorthanded as SMT, are proposed solution to the problem of finding whether a (classical first-order) logical formula is satisfiable with respect to logical theories. More precisely, provided with a logical formula with variables, the task is to find an assignment for the variables that make the formula true. Solving the SMT problem is useful when it is restricted to logical theories: actual tools can be implemented to search the solutions of the problem. Such tools are called SMT solvers and some of the more well-known are: Z3 [de Moura and Bjørner2008], CVC4 [Barrett et al.2011] and AltErgo [Bobot et al.2008].
The list of theories and procedures that SMT solvers are based on is extensive and includes simple linear arithmetic [Kroening et al.2010], array theory [Bruttomesso et al.2012], uninterpreted functions to model-based theory combination [de Moura and Bjørner2008], model based quantifier instantiation [Reynolds et al.2013], simplex (linear real arithmetic) [Dutertre and de Moura2006], extensional array theory [Stump et al.2001] and SAT abstraction/refinement [Clarke et al.2004].
To give the reader a glimpse of how SMT solvers work, we provide a classic example [de Moura and Bjørner2009] of a logical formula that involves integer arithmetic, arrays (encoded using the and axioms), and uninterpreted functions:
Here, returns the array which stores value at position , and returns the value stored in at position . The formula also contains an uninterpreted function (we are not aware of what computes).
First, an SMT solver performs a substitution of inside the formula, and thus, the first equality dissapears:
Second, some simple arithmetic is performed:
Then, the SMT solver uses an axiom about the and , which is included in a theory of arrays:
Therefore, after using this axiom, we obtain:
Indeed, this formula is not satisfiable for any choice of the function .
What is important to mention here is that the reasoning that Z3 performs is based on theories which guarantee that the results obtained by Z3 are formally correct. By encoding VFS and PFS in Z3 as logical formulas and checking their satisfiability, we obtain formal guarantees of correctness.
For a given input formula, say , an SMT solver has three possible outputs:
sat: when the input is satisfiable;
unsat: when the input is not satisfiable;
unknown: might be sat or unsat, but determining this is beyond the capabilities of the decision procedures.
SMT solvers scale in practice. For instance, Z3 is used at Microsoft for test case generation (e.g., PEX [Tillmann and de Halleux2008]), program verification (e.g., Spec# [Barnett et al.2005]), static driver verification (e.g., SLAM [Ball et al.2004]). Our choice for the Z3 SMT solver is motivated by several reasons: Z3 is used at an industrial scale; Z3 has native support for recursive functions (a quite recent addition to Z3, since November 20, 2018); and Z3 can be trusted: when it answers sat then it produces a witness (an assignment for variables) that make the formula satisfiable; when it answers unsat it produces a proof that the formula is not satisfiable.
We propose a systematic approach to tackle the verification of the VFS and PFS properties for AI models. First, we show how to encode these properties as logical formulas. Then, we explain how to use these encodings inside the Z3 SMT solver in order to obtain useful answers from it. Next, we exemplify our approach on a model for a well-known AI toy problem and we provide an implementation in Z3 which is also available online. Finally, we analyze the results that we get when running the implemented Z3 code.
1.2 Paper organization.
Section 2 of this paper shows how the two tests (VFS and PFS) are defined as logical formulas and implemented in an SMT solver. Section 3 exemplifies the proposed method on a model for a classic AI problem and the results of testing the corresponding formulas in Z3, as well as the impact some changes have on the obtained results. Section 4 concludes the above sections and talks about possible future developments.
2 Encoding PFS and VFS as logical formulas.
We start by explaining our notations. We denote by the predicate which is true when the state is valid, and by the predicate which is true when the state is final. Transition functions are denoted by , and they receive as arguments a state and some parameters, and return a state. Indeed, the passed parameters should produce a valid state. Multiple applications of a transition function to a state is denoted by , where is the number of applications and is a generic list of parameters:
By checking this property we essentially check the model with respect to predicates valid () and final (). If there is no state which is both valid and final then a final state could never be reached, since it doesn’t exists within the searcheable problem space.
In SMT, we analyse VFS by checking the satisfiability of the quantified conjunction: . If the solver returns sat then it means that our model is consistent and it includes at least one final state in the problem space. On the other hand, if the SMT solver returns unsat then there is no point of implementing any transitions or search strategies, as no path to a goal state could exist in the current model. If the solver returns unknown, then the only information we have is that the procedures implemented by the solver were not sufficient to decide satisfiability. This usually means that finding a valid final state is very difficult, even if it exists, a difficulty which transfers to an eventual working implementation of that model.
Encoding PFS as a logical formula is a little bit more complex. The problem is that we have to check if there exists a sequence of applications of the transition function (checking also the validity of the transition for its parameters) which starts with an initial state and ends with a final state. If we assume that is the predicate which is true when is an initial state, then PFS is equivalent to the following formula:
Basically, the formula above says that there exists an initial state and there is a sequence of transitions which produces a final state starting with . Moreover, the transitions to the final state are given by the list of parameters .
When Z3 answers unsat for this formula, then there is no initial state such that a final state can be reached from in the model. This information is a quite significant, since it tells us that, whatever strategy we use, we will never reach a final state.
On the other hand, if Z3 answers sat then there is certainly a path to a final state starting from an initial state. Therefore, we are sure that the model allows us to find such paths. However, there is no guarantee that a particular strategy will recover solutions within this model. But Z3 also provides a solution as a set of values for witch the checked formula is satisfiable, including the list of parameters P. Having an example of a solution will allow an informed choice with regards to a search strategy. Also, we can adjust the problem’s input parameters by adding various constraints over them and then use Z3 to look for solutions for particular instances. We discuss this further in Section 3.2.
The SMTLib language.
The above properties can be encoded into a language called SMT-LIB [Barrett et al.2017] which is accepted by the most important SMT solvers. This language is meant to describe logical theories and logical formulas which are meant to be checked for satisfiability. The language has a syntax very similar to S-expressions or Lisp. Examples of SMT-LIB code is shown in Section 3.2.
3 Missionaries and Cannibals
3.1 Problem description
This classic AI problem is well known, its generalization can be formulated as: On the shore of a river there are nm missionaries and nc cannibals. There is a boat with bcap capacity on the same shore. Find, if it exists, a way to move all people on the initial shore to the other shore, using the boat. Consider that the boat moves only with 1 to bcap people in it, and on neither shore there can be more cannibals than missionaries, if there is at least one missionary there.
3.2 Model #1: The valid variant
A first example of a possible model for this problem will be described below. The model is an adaptation of the one proposed in [Amarel1981]. Consider a state of the problem a list of 6 values: , where:
is the capacity of the boat (maximum number of passengers)
and are the numbers of missionaries and cannibals on the left (initial) shore
is the current location of the boat (either shore 1 or 2)
and are the numbers of missionaries and cannibals on the right (goal) shore
Considering the problem statement in Section 3.1, the following have to be true for a state to be valid (within the problem space):
The initial state would be , and the final state .
A transition is configured by two parameters, - the number of missionaries to be moved and - the number of cannibals to be moved. The only transition available is
which is valid if:
and and and
and and and
A solution (sequence of transitions) can be described as a list , where each element denotes a valid transition, first one from the initial state, last one to the final state.
The model in SMT-LIB.
The model discussed in Section 3.2 can be easily encoded as an SMT-LIB specification. For precision, we use version 2.0 of this language. Recall that SMT-LIB is a language accepted by many SMT solvers, including Z3.
Let us recall the VFS and PFS properties: and . Encoding these existentially quantified formulas in SMT-LIB is done as follows:
, and are declared as uninterpreted functions/constants of their corresponding type:
(declare-const state State)
(declare-const n Int)
(declare-const p Parameters)
It is worth noting that State and Parameters are just aliases for arrays of integers.
The input parameters are uninterpreted constants too:
(declare-const nm Int)
(declare-const nc Int)
The predicates , , and are implemented using functions that return a boolean value; here, we show only :
(define-fun final ((s State)) Bool
(= (nm2 s) nm)
(= (nc2 s) nc))
(= 2 (bp s))))
We use additional helper functions that extract information from the state: the call (bp s) returns the boat position in the current state s, (nm2 s) returns the number of missionaries on shore 2, and (nc2 s) returns the number of missionaries on shore 2. Therefore, a state is final if all the missionaries and cannibals are on shore 2.
The code corresponding to is available at https://github.com/andreiarusoaie/z3-ai-model-verification/blob/master/experiments/cannibals_and_missionaries/correct-model/pfs.smt2#L65, while the code corresponding to is available at https://github.com/andreiarusoaie/z3-ai-model-verification/blob/master/experiments/cannibals_and_missionaries/correct-model/pfs.smt2#L36.
The function is defined as an ordinary function which takes a state and some parameters (e.g., missionaries and cannibals to be moved) and returns a new state:
((s State) (mm Int) (mc Int))
The implementation in SMT-LIB for the transition function is available at https://github.com/andreiarusoaie/z3-ai-model-verification/blob/master/experiments/cannibals_and_missionaries/correct-model/pfs.smt2#L85.
The function is defined as a recursive function which repeatedly applies :
(params Parameters) <additional params> )
<function body with recursive call>
The corresponding code is available here: https://github.com/andreiarusoaie/z3-ai-model-verification/blob/master/experiments/cannibals_and_missionaries/correct-model/pfs.smt2#L129.
VFS is now encoded as a simple conjunction:
(and (valid state) (final state))
PFS is a little bit more complex:
(final (tran n state p state (* 2 n)))
Here, is a list of length which corresponds to a solution sequence . The last two parameters are helper parameters that hold the last valid state and the size of , respectively. The task of the SMT solver is to find values for , , and .
To check VFS or PFS we need to include the corresponding assertion in the implemented model. Consider that VFS and PFS can be satisfiable or not independent of the other.
In order to check our SMT-LIB specification for satisfiability we have to append the following command at the end of the specification: (check-sat). If Z3 returns sat, then it means that there are some values for (for VFS) or , and (for PFS, respectively) that satisfy the specification. In order to obtain them we add (get-model) right after (check-sat). If Z3 returns unsat, then there are no values that satisfy the specification. This means that our model does not have the property we are currently checking and it might be something wrong with the model. If Z3 returns unsat or unknown, we can try to adjust the parameters of our model by adding various constraints (e.g., limit the boat capacity).
In addition to all these, we can add problem specific restrictions to our specification. For instance, we should add an assertion that limits the range of the parameters of the transition function, i.e., parameters should be between 0 and the boat size. This is acceptable, since exceeding the capacity of the boat in a transition will never lead us to a valid state.
3.3 Other model variants
In order to see how Z3 can indicate failures in a model, we tried two alternatives to the model described above.
3.3.1 Model #2
First variation was to change these tests from the valid state checks:
This variation assumes that we check that the number of missionaries is larger than the number of cannibals while ignoring the requirement for this to be true only when the number of missionaries is non-zero. This is a likely assumption to make by a person developing the model when he/she is not experienced/attentive enough, especially since it requires less written code.
This change should make VFS invalid. Also, for most instances of this problem, a solution, unless is larger than , requires that, at least at one point, the number of missionaries on a shore be zero and the number of cannibals non-zero, thus we should also expect PFS to fail for most problem instances.
3.3.2 Model #3
The second alternative model added an additional check to validate a transition: . The requirement to always move more missionaries than cannibals could be a misinterpretation of the problem requirements that on all shores, if the number of missionaries is non-zero, it is larger than the number of cannibals. Someone might assume that this requirement also applies to the boat.
This change would still allow a valid final state (thus VFS is satisfiable), but PFS should fail, as a solution for which always more missionaries than cannibals are moved shouldn’t exist unless for particular instances in which and .
3.4 Analysis with Z3.
The entire SMT-LIB specification discussed in Section 3.2 is available on Github: https://github.com/andreiarusoaie/z3-ai-model-verification.git. Follow the accompanying instructions to run the code. Implementations for all variation discussed above and both VFS and PFS checks (performed separately) are included.
Typically, for every scenario that we created, we have two files – one for each property: pfs.smt2 and vfs.smt2.
In every scenario, we constrained the boat size to be greater than 2, and we asserted that the number of missionaries and cannibals should be greater than 2 as well (to avoid trivial solutions):
(assert (< 2 missionaries))
(assert (< 2 cannibals))
(assert (< 2 (bcap state)))
For each scenario we present the elapsed time, the memory consumption, and the number of basic resource-consuming operations within the solver.
We have performed the following tests on a machine with an Intel i7 8700K CPU at 3.7 GHz, and 32 GB of RAM. The system runs an Ubuntu based operating system called PopOS.
Tests for model #1.
In the first scenario we used the model described in Section 3.2. Z3 returns sat for both PFS and VFS, indicating that each property is satisfiable, and thus proving that the model can describe at least one solution. The results are summarized in Table 1.
|# 1||(s)||(MB)||(no of ops.)|
The (get-model) command also gives us solutions. For VFS, Z3 finds the state , where the number of missionaries is 7722 and the number of cannibals is 3. Indeed, this state is final and valid. Note that for a different version of Z3 the returned solution could be different. A drawback for Z3 is the output which is not easy to read when it prints arrays or functions. This is common to all SMT solvers, and we recommend the use of an external tool to postprocess the output.
For PFS, Z3 returns sat as well, and, when asked, it provides a solution. What is interesting to notice is that the number resource-consuming operations for PFS is much bigger than for VFS, and this is explained by the use of a recursive function. Solving the SMT problem when recursive functions are used is expensive, as pointed out by the values for memory consumption and elapsed time.
The almost two minutes time required to validate the existence of a sequence of recursive calls of the transition function is low, considering the unrestricted nature of the test. and are not restricted (except as being Integers and positive), is restricted to values between 2 and and the parameters for the transitions ( and ) are restricted between 0 and . No other heuristics are used as to keep the results relevant for any possible search heuristic applied on the model.
Tests for model #2.
In the second test we reproduce the first failure discussed in Section 3.3. The error that we introduce in the model is meant to emphasize a common mistake: the lack of precision, i.e., we only say that the number of missionaries is bigger than the number of cannibals. This is not always true, since the problem constraints allow us to have zero missionaries on shores. This change makes both the VFS and PFS invalid. The corresponding code for both VFS and PFS can be found here: https://github.com/andreiarusoaie/z3-ai-model-verification/tree/master/experiments/cannibals_and_missionaries/variation1-bad-valid-function. For this scenario, Z3 returns unsat for both properties.
|# 1||(s)||(MB)||(no of ops.)|
In Table 2 we show the statistics for this test. Both VFS and PFS fail in this case. Z3 is slower for PFS again, but it is able to decide in a fairly reasonable amount of time that this alteration of the model is not feasible. Since Z3 is sound, the result indicates the fact that we will never be able to discover a solution. Moreover, this applies to any possible strategy. This is a very powerful result, which can be obtained only by reasoning about logical formulas.
Tests for model #3.
Finally, in the third test we experiment the second failure that we discuss in Section 3.3. In this case, the transition function is enriched with a new constraint: the number of missionaries in the boat is always bigger than the number of cannibals. Also, we limited as equal to . As explained in Section 3.3, for VFS Z3 should return sat and it does, while for PFS, Z3 returns unsat.
The behavior of Z3 for VFS should be the same as in Table 1. The change affects only PFS, whose code is available here: https://github.com/andreiarusoaie/z3-ai-model-verification/blob/master/experiments/cannibals_and_missionaries/variation2-bad-transition/pfs.smt2.
For the statistics shown in Table 3.3, we added a constraint for the depth of the recursion to be less than 100. Considering the number of variables (even the instance is not set, parameters for the transitions are not constant), the number of variations attempted at each step is significant, nearly reaching the maximum available RAM for our test system.
|# 1||(s)||(MB)||(no of ops.)|
The statistics shown in Table 3 shows that the time spent by Z3 to decide unsatisfiability is much larger than for the satisfiable model #1. For PFS, the intended error in the model is significantly harder to prove and Z3 needs to almost 2000 times more operations to reach this conclusion.
A positive sign is that for no test the result was ”unknown”, as that would assume the insufficiency of Z3’s decision procedure for this model. In additional tests we found that ”unknown” is concluded extremely rarely and was always a consequence of errors in the SMT implementation of the model.
Z3 allows various tricks and tweaks. For instance, not only that you can use it to find whether the AI model has certain properties, but you can add various constraints to check more powerful properties. For instance, in the first scenario, one can search for solutions where , and are bigger than, less than, or equal to some specified value. Knowing that a model is valid can lead to checking the existence of solutions for particular instances, even discovering limits or correlations between the existence of a solution and the values of these parameters. Constraints set on the values in - for example considering that people only of a single type can use the boat at once - can allow for testing various other variations. By setting an upper bound or a lower bound for the recursion depth we can test the existence of solutions of length between those bounds.
More powerful tools for proving logical formulas are interactive theorem provers like Coq [Bertot and Castéran2010], or Isabelle [Nipkow et al.2002]. Unfortunately, these are not automatic and require a lot of professional training. On the other hand, other SMT solvers like CVC4 [Barrett et al.2011] or AltErgo [Bobot et al.2008], can be used in parallel with Z3. Note that SMT solvers cannot contradict each other: it is impossible for one to return sat and another one to return unsat for the same input formula. What can be different is the amount of time spent to compute the result and the complexity of the implementation. Also, only Z3 has native support for recursive functions which are required to test PFS or similar properties.
4 Conclusion and Future work
We showed that the benefits of a state-of-the-art SMT solver used to validate transition systems (as models for AI problems) can be significant. The added value of SMT solvers consists in the fact that the entire model is encoded using logical formulas. These formulas allow reasoning at an abstract level, and thus, it decreases the search space. The very recent addition of support for recursive functions in Z3 allowed us to look for and recover solutions, a new capability for bounded model checking.
The proposed application of an established SMT solver in AI model validations comes with the minimal overhead of having to describe that model as SMT logical inferences. This effort proved to be manageable, the main difficulty being maintaining the semantic accuracy of the transcription. A potential automated transcription between STRIPS and Z3 code is planned as a future development for our system. PDDL [McDermott et al.1998] or other action languages could be considered later on as well.
- [Amarel1981] Saul Amarel. On representations of problems of reasoning about actions. In Readings in artificial intelligence, pages 2–22. Elsevier, 1981.
- [Ball et al.2004] Tom Ball, Byron Cook, Vladimir Levin, and Sriram Rajamani. Slam and static driver verifier: Technology transfer of formal methods inside microsoft. Technical report, January 2004.
- [Barnett et al.2005] Mike Barnett, Rustan Leino, and Wolfram Schulte. The spec# programming system: An overview. In CASSIS 2004, Construction and Analysis of Safe, Secure and Interoperable Smart devices, volume 3362 of Lecture Notes in Computer Science, pages 49–69. Springer, January 2005.
- [Barrett et al.2011] Clark Barrett, Christopher L. Conway, Morgan Deters, Liana Hadarean, Dejan Jovanović, Tim King, Andrew Reynolds, and Cesare Tinelli. Cvc4. In Ganesh Gopalakrishnan and Shaz Qadeer, editors, Computer Aided Verification, pages 171–177, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg.
- [Barrett et al.2017] Clark Barrett, Pascal Fontaine, and Cesare Tinelli. The SMT-LIB Standard: Version 2.6. Technical report, Department of Computer Science, The University of Iowa, 2017. Available at www.SMT-LIB.org.
- [Bertot and Castéran2010] Yves Bertot and Pierre Castéran. Interactive Theorem Proving and Program Development: Coq’Art The Calculus of Inductive Constructions. Springer, 1st edition, 2010.
- [Bobot et al.2008] François Bobot, Sylvain Conchon, Évelyne Contejean, Mohamed Iguernelala, Stéphane Lescuyer, and Alain Mebsout. The Alt-Ergo automated theorem prover, 2008. http://alt-ergo.lri.fr/.
[Bruttomesso et al.2012]
Roberto Bruttomesso, Silvio Ghilardi, and Silvio Ranise.
Quantifier-free interpolation of a theory of arrays.Logical Methods in Computer Science, 8, 04 2012.
- [Bylander1994] Tom Bylander. The computational complexity of propositional strips planning. Artificial Intelligence, 69(1-2):165–204, 1994.
- [Clarke et al.2001] Edmund Clarke, Armin Biere, Richard Raimi, and Yunshan Zhu. Bounded model checking using satisfiability solving. Formal methods in system design, 19(1):7–34, 2001.
- [Clarke et al.2004] E. M. Clarke, A. Gupta, and O. Strichman. Sat-based counterexample-guided abstraction refinement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 23(7):1113–1123, July 2004.
- [de Moura and Bjørner2008] Leonardo Mendonça de Moura and Nikolaj Bjørner. Z3: an efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems, 14th International Conference, TACAS 2008, ETAPS, 2008, Budapest, March 29-April 6, 2008. Proceedings, pages 337–340, 2008.
- [de Moura and Bjørner2009] Leonardo de Moura and Nikolaj Bjørner. Satisfiability modulo theories: An appetizer. In Formal Methods: Foundations and Applications, 12th Brazilian Symposium on Formal Methods, SBMF, volume 5902 of LNCS, pages 23–36. Springer, 2009.
- [de Moura and Bjørner2008] Leonardo de Moura and Nikolaj Bjørner. Model-based theory combination. Electronic Notes in Theoretical Computer Science, 198:37–49, 05 2008.
- [Dutertre and de Moura2006] Bruno Dutertre and Leonardo de Moura. Integrating simplex with dpll(t ). 01 2006.
- [Edelkamp and Helmert2001] Stefan Edelkamp and Malte Helmert. Mips: The model-checking integrated planning system. AI magazine, 22(3):67–67, 2001.
- [Fikes and Nilsson1971] Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189–208, 1971.
- [Hopgood2016] Adrian A Hopgood. Intelligent systems for engineers and scientists. CRC press, 2016.
- [Huang and Van Der Meyden2014] Xiaowei Huang and Ron Van Der Meyden. Symbolic model checking epistemic strategy logic. In Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.
- [Kroening et al.2010] Daniel Kroening, Jérôme Leroux, and Philipp Rümmer. Interpolating quantifier-free presburger arithmetic. pages 489–503, 10 2010.
- [McDermott et al.1998] Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Pddl-the planning domain definition language. 1998.
- [Nipkow et al.2002] Tobias Nipkow, Markus Wenzel, and Lawrence C. Paulson. Isabelle/HOL: A Proof Assistant for Higher-order Logic. Springer-Verlag, Berlin, 2002.
- [Reynolds et al.2013] Andrew Reynolds, Cesare Tinelli, Amit Goel, Sava Krstić, Morgan Deters, and Clark Barrett. Quantifier instantiation techniques for finite model finding in smt. In Maria Paola Bonacina, editor, Automated Deduction – CADE-24, pages 377–391, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.
- [Russell and Norvig2016] Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Malaysia; Pearson Education Limited,, 2016.
- [Stump et al.2001] Aaron Stump, Clark W. Barrett, David L. Dill, and Jeremy Levitt. A decision procedure for an extensional theory of arrays. In Proceedings of the 16th Annual IEEE Symposium on Logic in Computer Science, LICS ’01, pages 29–, Washington, DC, USA, 2001. IEEE Computer Society.
- [Tillmann and de Halleux2008] Nikolai Tillmann and Jonathan de Halleux. Pex-white box test generation for .net. In Tests and Proofs, Second International Conference, TAP 2008, Prato, Italy, April 9-11, 2008. Proceedings, pages 134–153, 2008.