The last two decades have witnessed the rise of Satisfiability Modulo Theories (SMT) [BSST09HBSAT] as efficient tool for dealing with several applications of industrial interest, in particular in the contexts of Formal Verification (FV). SMT is the problem of finding value assignments satisfying some formula in first-order logic wrt. some background theory. Optimization Modulo Theories (OMT) [nieuwenhuis_sat06, st-ijcar12, li_popl14, LarrazORR14, st_tacas15, bjorner_tacas15, NadelR16, kovas18] is a more-recent extension of SMT searching for the optimal value assignment(s) w.r.t. some objective function(s), by means of a combination of SMT and optimization procedures. (Since OMT extends SMT, and that all OMT tools are built on top of SMT solvers, hereafter we often simply say “OMT” for both SMT and OMT.)
Several distinctive traits of OMT solvers –like, e.g., the efficient combination of Boolean and arithmetical reasoning, incrementality, the availability of decision procedures for infinite-precision arithmetic and the ability to produce conflict explanations– are a direct consequence of their tight relationship with the FV domain and its practical needs. On the whole, it appears that OMT can be a potentially interesting and efficient technology for dealing with Constraint Programming (CP) problems as well. At the same time, modeling CP problems for OMT solvers requires a higher-level of expertise, because the same CP instance can have many possible alternative formulations, s.t. the performance of SMT solvers on each encoding are hardly predictable [frisch09, frisch10, elgabou14].
On the other hand, the availability, efficiency and expressiveness of CP tools makes them of potential interest as backend engines also for FV applications (e.g., [Collavizza2006, Collavizza2010, Grinchtein2015]), in particular with SW verification, where currently SMT is the dominating backend technology, s.t. a large amount of SMT-encoded FV problems are available [brst10_smtlib].
In this paper we aim at bridging the gap between CP and OMT, in both directions.
In the CP-to-OMT direction, we have extended the state-of-the-art OMT solver OptiMathSAT [st_jar18] with a FlatZinc interface (namely “fzn2omt”). In combination with the standard mzn2fzn encoder [minizinc_url], this new interface can be used to either (i) solve CP models with OptiMathSAT directly or (ii) generate OMT formulas encoded in the SMT-LIB [cts_cpaior20_extended] format with optimization extensions, to be fed to other OMT solvers, such as bclt [Bofilletal2008CAV] and Z3 [bjorner_tacas15]. This allows state-of-the-art OMT technology to be used on MiniZinc problems coming from the CP community.
In the OMT-to-CP direction, we have introduced a tool for translating SMT and OMT problems on the theories of linear arithmetic over the integers and rational () and bit-vector () into MiniZinc models (hereafter “omt2mzn”). This allows MiniZinc solvers to be used on OMT problems, giving them access to a large amount of OMT problems, mostly coming from formal verification.
With both directions, we first present and discuss the challenges we encountered and the solutions we adopted to address the differences between the two formalisms. Then we present an extensive empirical evaluation comparing three OMT tools with many state-of-the-art CP tools on (i) CP problems coming from the MiniZinc challenge, and (ii) OMT problems coming mostly from formal verification. This analysis allowed us to identify some criticalities, in terms of efficiency and correctness, one has to cope with when addressing CP problems with OMT tools, and vice versa.
Overall, our new compilers fzn2omt and omt2mzn in combination with the standard compiler mzn2fzn [minizinc_url] provide a framework for translating problems encoded in the SMT-LIB or the MiniZinc format in either direction. This framework enables also for a comparison between OMT solvers and CP tools on problems that do not belong to their original application domain. To the best of our knowledge, this is the first time that such a framework has been proposed, and that the OMT and CP technologies have been extensively compared on problems coming from both fields.
The tight connection between SMT and Constraint Programming (CP) has been known for a relatively long period of time [nieuwenhuisetal2007rta] and it has previously been subject to investigation. Some works considered a direct encoding of CP [frisch09, frisch10] and weighted CP [ansotegui13] into SMT and MaxSMT, or an automatic framework for translating MiniZinc –a standard CP modeling language [stuckey_mzn_07]– into SMT-LIB –the standard SMT format– [bof10sys, bof12]. Other works explored the integration of typical SAT and SMT techniques within CP solvers [ohriSC09, feydy09]. Nowadays, several MiniZinc solvers –like, e.g., HaifaCSP [vekslerS16] and Picat [zhou17]– are at least partially based on SAT technology.
To this extent, our first contribution fzn2omt also obviates the loss, due to obsolescence, of the fzn2smt compiler proposed by Bofill et al. in [bof10sys, bof12]. fzn2smt is not compatible with the changes that have been introduced to the MiniZinc and FlatZinc standards starting from version of the MiniZinc distribution. Since some of these changes are not backward compatible, it is also not possible to use fzn2smt in conjunction with an older version of the mzn2fzn compiler when dealing with recent MiniZinc models. Furthermore, fzn2smt translates satisfaction problems into the Version of the SMT-LIB standard and produces no SMT-LIB output in the case of optimization problems, that are solved directly. However, the optimization interface of modern OMT solvers is based on the Version of the SMT-LIB standard. This makes it difficult to use it together with OMT solvers. Unfortunately, the fzn2smt compiler is closed source, with only the binaries being freely distributed, and seemingly no longer maintained. This made it necessary to provide a new alternative solution to fzn2smt. To this extent, our new FlatZinc interface of OptiMathSAT, fzn2omt, translates both satisfaction and optimization problems in the Version of the SMT-LIB standard enriched with the optimization extensions for OMT described in [st_jar18].
The rest of the paper is organized as follows. In §2 we provide some background on OMT, MiniZinc and FlatZinc. In §3 we describe the process from MiniZinc to OMT. In §4 we describe the process from OMT to MiniZinc. In §5 we describe an empirical evaluation comparing a OMT-based tool with many state-of-the-art CP tools. Finally, in §6 we conclude and point out some further research directions.
A longer and more detailed version of this paper is publicly available as [cts_cpaior20_extended].
Satisfiability Modulo Theories (SMT) is the problem of deciding the satisfiability of a first-order formula with respect to a combination of decidable first-order theories. Typical theories of SMT interest are (the theory of) linear arithmetic over the rationals (), the integers () or their combination (), non-linear arithmetic over the rationals () or the integers (), arrays (), bit-vectors (), and their combinations thereof. (See [nieot-jacm-06, sebastiani07, BSST09HBSAT] for an overview.). The last two decades have witnessed the development of very efficient SMT solvers based on the so-called lazy-SMT schema [sebastiani07, BSST09HBSAT]. This has brought previously-intractable problems to the reach of state-of-the-art SMT solvers.
Optimization Modulo Theories (OMT), [nieuwenhuis_sat06, st-ijcar12, li_popl14, LarrazORR14, st_tacas15, bjorner_tacas15, NadelR16, ts_cade19], is an extension to SMT that allows for finding a model of a first-order formula that is optimal with respect to some objective function expressed in some background theory, by means of a combination of SMT and optimization procedures. State-of-the art OMT tools allow optimization in a variety of theories, including linear arithmetic over the rationals () [st-ijcar12] and the integers () [bjorner_tacas15, st_tacas15], bit-vectors () [NadelR16] and floating-point numbers (OMT ()) [ts_cade19].
A relevant strict subcase of is OMT with Pseudo-Boolean objective functions () in the form s.t. are rational values and are Boolean variables whose values are interpreted as . Notice that is also equivalent to (partial weighted) MaxSMT, the SMT extension of MaxSAT, and that and MaxSMT can be encoded into but not vice versa [st_tocl14]. Encoding /MaxSMT into , however, is not the most efficient way to solve them, so that modern OMT solvers such as bclt [Bofilletal2008CAV], OptiMathSAT [st_jar18] and Z3 [bjorner_tacas15] implement specialized /MaxSMT procedures which are much more efficient than general-purpose ones [bjorner_tacas15, st_tacas17, st_jar18].
We stress the fact that —unlike with purely-combinatorial problems, which are encoded into SAT or MaxSAT and are thus solved by purely-Boolean search– typically OMT problems involve the interleaving of both Boolean and arithmetical search: search not only for the best truth-value assignment to the atomic subformulae, but also for the best values to the numerical variables compatible with such truth-value assignment [st_tocl14].
To this date, few OMT solvers exist, namely bclt [Bofilletal2008CAV], Cegio [araujo16], Hazel [NadelR16], OptiMathSAT [st_jar18], Puli [kovas18], Symba [li_popl14] and Z3 [bjorner_tacas15]. To this aim, we observe that (i) some of these solvers are quite recent, (ii) most of these solvers focus on different, partially overlapping, niche subsets of Optimization Modulo Theories, and (iii) the lack of an official Input/Output interface for OMT makes it hard to compare some of these tools with one another. OMT finds applications in the context of static analysis [candeago16, karpenkov17], formal verification and model checking [liutbt17, ratschan17], scheduling and planning with resources [kovas18, leofante18, roselli18, oliver18], software security and requirements engineering [nguyensgm16], workflow analysis [bertolissi18]teso17], and quantum computing [bian17_frocos17].
A distinctive trait of SMT (and OMT) solvers is the trade-off of speed against the ability to certify the correctness of the result of any computation, which is particularly important in the contexts of Formal Verification (FV) and Model Checking (MC). When dealing with linear arithmetic in particular, SMT solvers employ infinite-precision arithmetic software libraries to avoid numerical errors and overflows.
SMT-LIB [cts_cpaior20_extended] is the standard input format by SMT solvers, it provides a standardized definition of the most prominent theories supported by SMT solvers and the corresponding language primitives to use these features. At present, there is no standard input format for modeling optimization problems targeting OMT solvers, although there exist only minor syntactical differences between the major OMT solvers. The tools presented in this paper conform to the extended SMT-LIB format for OMT presented in [st_jar18], that includes language primitives for modeling objectives.
OptiMathSAT [st-ijcar12, st_tocl14, st_cav15, st_tacas15, st_tacas17, st_jar18] is a state-of-the-art OMT solver based on the MathSAT5 SMT solver [mathsat5-url, mathsat5_tacas13]. OptiMathSAT features both single- and multi-objective optimization over arbitrary sets of , , , , , Pseudo-Boolean () and MaxSMT cost functions. Multiple objective functions can be combined with one another into a Lexicographic or a Pareto optimization problem, or independently solved in a single run (for the best efficiency).
MiniZinc [stuckey_mzn_07, minizinc_url] is a widely adopted high-level declarative language for modeling Constraint Satisfaction Problems (CSP) and Constraint Optimization Problems (COP). The MiniZinc format defines three scalar types (bool, int and float) and two compound types (sets and fixed-size arrays of some scalar type). The standard provides an extensive list of predefined global constraints, a class of high-level language primitives that allows one to encode complex constraints in a compact way. Furthermore, MiniZinc supports useful language constructs such as if-then-else, let expressions, universal and existential comprehensions over finite domains, user-defined predicates and more.
FlatZinc is a lower-level language whose purpose is to bridge the gap between the high-level modeling in MiniZinc and the need for a fixed, and easy-to-parse, format that can simplify the implementation of the input interface of MiniZinc solvers. A MiniZinc model is typically flattened into a FlatZinc instance using the mzn2fzn compiler [minizinc_url], and then solved with some MiniZinc tool.
3 From MiniZinc to Omt
We consider the problem of translating MiniZinc models into OMT problems first. Similarly to other MiniZinc solvers, we assume that the MiniZinc model is first translated into FlatZinc using the mzn2fzn standard compiler, as depicted in Figure 1. We describe the main aspects of fzn2omt, focusing on the challenges we have encountered and on the solutions we have adopted.
The first challenge is to find a suitable representation of the data-types supported by FlatZinc in SMT-LIB.
One possible choice for modeling the three basic scalar types of FlatZinc –namely bool, int and float– with SMT-LIB are the Boolean, bit-vector and floating-point theories respectively. However, the decision procedures for the bit-vector and floating-point numbers theories can be significantly more resource demanding than the decision procedure for the linear arithmetic theory (), in particular when dealing with a substantial amount of arithmetic computations. For this reason, we have opted to model FlatZinc int and float data-types with the SMT-LIB integer and rational types respectively, by default. For the case in which no substantial linear arithmetic computation is performed, we also optionally allow for encoding the FlatZinc int data-type as a SMT-LIB bit-vector.
For what concerns the two compound types of FlatZinc, that is the set and array data-types, we have chosen to proceed as follows. Given that OptiMathSAT lacks a decision procedure for the theory of finite sets [sets_url], we model a set using the Boolean and integer theories, similarly to what has been done in [bof12]. The basic idea is to introduce a fresh Boolean variable for each element in the domain of a set, and use such variable as a placeholder for the membership of an integer element to the set instance. Differently from [bof12], we make an extensive use of cardinality networks [Asin2011] to encode constraints over the sets because they are handled more efficiently, for their nice arc-consistency properties. No action is required to encode a FlatZinc array into SMT-LIB, because it is used only as a container for other variables.
A consequence of encoding the FlatZinc int and float data-types with the linear arithmetic theory is that all of our computation is performed with infinite-precision arithmetic. This can result in a performance disadvantage wrt. other MiniZinc solvers using finite-precision arithmetic, due to the increased cost of each operation, but it has the benefit of guaranteeing the correctness of the final result of the computation.
Currently, the MiniZinc language does not allow one to express a certain quantity as an infinite-precision fraction between two constant numbers. Instead, the mzn2fzn compiler computes on-the-fly the result of any division operation between two constant integers or floating-point numbers applying the rules corresponding to the type of the operands. However, there are some instances in which we really need to be able to both express quantities and perform computation with infinite-precision arithmetic. One of such situations is to double-check the correctness of the MiniZinc models generated by the omt2mzn compiler described in Section §4 (we have done this for the experimental evaluations in Section §5.2). In order to get around this limitation we developed a simple wrapper around the mzn2fzn compiler, called emzn2fzn [emzntofzn_url], that replaces any fraction among two constant floats with a fresh variable, and then sets its value using the FlatZinc constraint float_div().
The SMT-LIB encoding of the majority of FlatZinc constraints in OptiMathSAT follows their definition in the FlatZinc Standard, with the exception of Pseudo-Boolean constraints, which we examine in detail later on. Several global constraints are also supported in the same way, because the OMT-solver currently lacks ad hoc and efficient decision procedures for dealing with them. Constant values and alias variables (e.g. those arising from the definition of some arrays) are propagated through the formula, so as to keep the set of problem variables as compact as possible. Those constraints requiring non-linear arithmetic –like, e.g., trigonometric, logarithmic and exponential functions– are currently not supported; this situation may change soon due to the recent extension of MathSAT5 with a procedure for it [ac_tocl18].
When dealing with Pseudo-Boolean sums of the form , where is a Boolean variable and is a numerical weight, the mzn2fzn compiler associates a fresh / variable to each , and encodes the sum as . Notice that the original s may not be eliminated from the FlatZinc model, because they typically occur elsewhere in the problem, i.e. as part of a Boolean formula. From our own experience, this situation arises frequently, because Pseudo-Boolean sums are typically used to express cardinality constraints that have a variety of uses. As described in [st_tacas17], one limitation of this naive approach is that SMT and OMT solvers do not typically handle this encoding efficiently. The main reason is that the pruning power of the conflict clause resulting from a conflicting assignment is typically limited to one specific Boolean assignment at a time, meaning that a large number of conflict clauses (possibly exponential) has to be generated along the search. As shown in [st_tacas17], SMT and OMT solvers can benefit from encoding Pseudo-Boolean constraints with cardinality networks.
fzn2omt goes through some effort in order to recognize Pseudo-Boolean sums over the integers, and replace the naive encoding with one based on cardinality networks. We note that using this technique generally results in a trade-off between solving time and the overhead of generating cardinality networks prior to starting the search, especially when dealing with a large number of variables.
fzn2omt allows for multiple optimization goals, of heterogeneous type, being defined within the same FlatZinc model. This is a non-standard extension to the FlatZinc format. Multiple objectives can be solved independently from one another, or combined into a Lexicographic or Pareto optimization goal. We refer the reader to [st_jar18] for details on the input encoding and the solver configuration.
Given a satisfiability or optimization problem encoded in the FlatZinc format, OptiMathSAT can be used in the following ways (Figure 1):
to directly solve the problem, optionally enumerating any sub-optimal solution found during the search or all possible solutions with the same optimal value;
to produce an OMT problem encoded with the extended SMT-LIB format described in [st_jar18]. This problem can be directly solved with OptiMathSAT or, with minor transformations111To make this step as easy as possible, we collected our scripts into a public repository [fzntoomt_url]., fed as input to other OMT solvers such as bclt and Z3.
4 From Omt to MiniZinc
In this section, we consider the problem of translating OMT formulas, encoded in the optimization-extended SMT-LIB format of [st_jar18], into MiniZinc models. Hereafter, we describe the main challenges we have faced and the solutions we have adopted. Further details about this conversion are available in [omt2mzn].
General Translation Approach.
The main challenge is to design an encoding from OMT to MiniZinc that is correct (i.e., it preserves in full the semantics of the input OMT problems), effective (i.e., it produces as output MiniZinc models which are as compact and easy-to-solve as possible), and efficient (i.e. it does it with the least consumption of time and memory). For best effectiveness, the internal representation of the input OMT formula adopted by the compiler is that of a Directed Acyclic Graph (DAG), that represents repeating sub-formula elements with the same node. A simple, tree-like, traversal of this data-structure can result in some node –and its entire sub-tree– being visited multiple times during the translation of the OMT formula, and hence in a potentially exponential blow-up of the generated MiniZinc model (left-most tree in Figure 2). The converse approach, that is, to assign an explicit label to each node of the DAG and visit each node exactly once, can suffer from the creation of a large number of fresh variables that do not exist in the original problem, even for those nodes that would be traversed only once (center DAG in Figure 2). To cope with this problem, we adopted the idea of assigning a label only to those DAG nodes that have at least two father nodes (right-most DAG in Figure 2). This limits the number of artificial variables introduced in the encoding, and at the same time produces the most compact representation of the input problem. In our experiments, the latter idea showed to be essential to produce much more compact and easy-to-solve problems.
The SMT-LIB standard describes a wide number of SMT theories, most of which have no direct counterpart in MiniZinc due to the few data-types supported (see §2). Hence, hereafter we deal only with the theory of linear rational and integer arithmetic, their combination, and the theory of bit-vectors. We leave the handling of other SMT theories to future work.
Linear Arithmetic Theory.
On the surface, encoding linear arithmetic constraints coming from OMT in MiniZinc, using the int and float data-types, looks like a trivial task. In reality, this poses several challenges and it is subject to several limitations, due to a couple of facts.
First, in SMT-LIB the linear arithmetic theory requires the capability to perform infinite-precision computations. Unfortunately, to the best of our knowledge, no MiniZinc solver provides infinite-precision arithmetic reasoning, and the mzn2fzn compiler itself prevents representing arbitrarily-large and arbitrarily-precise quantities (e.g. the fine-grained decimal weights of the machine learning application in [teso17]).
Second, in OMT linear arithmetic variables are not required to be bounded and have quite often no explicit domain (i.e. they lack a lower-bound, an upper-bound or both), because it is not necessary for the problem at hand or it is implied by other constraints. This is in contrast with MiniZinc, whereby linear arithmetic variables are expected to have a finite domain and, when they lack one, their domain appears to be capped with some solver-dependent pair of values.
These restrictions are currently part of the MiniZinc language and the target application domain, and we do not see any obvious work-around solving them. We note that although there exist methods for bounding all variables in a given LP problem (e.g. [papadimitriou81]), these have been deemed too impractical at this stage of our investigation. Nonetheless, we have chosen to translate SMT-LIB linear arithmetic constraints with a corresponding MiniZinc encoding based on the int and float data-types. Although the encoding is not always applicable, it does still allow one to correctly translate a number of interesting OMT problems into MiniZinc, as witnessed by our experimental evaluation in Section §5.2. More in detail, the translation is done as follows. We declare each integer variable as unbounded, and then extend the MiniZinc model with the appropriate constraints bounding its domain when the input OMT formula contains any such information. Our empirical observation is that MiniZinc models generated in this way are correctly handed by all MiniZinc solvers which we have tried, with the exception of Gurobi, which returns an “unsupported” message. Floating-Point variables, instead, are always declared with a user-defined domain. This is because all of the MiniZinc solvers we have tried, among those that can handle floating-point constraints, require such information.
Dealing with bit-vectors is also challenging. In essence, an OMT bit-vector can be seen as a vector of bits of fixed size, that can be represented as a Boolean array in MiniZinc. However, because bit-vectors are typically used to encode machine registers and numbers at the binary-level, the bit-vectors theory provides both signed and unsigned arithmetic operators, as well as those based on Boolean logic. Performing arithmetic computation over a Boolean array in MiniZinc requires one to define and instantiate of the Boolean circuit encoding the given operation as a set of constraints. As far as we can see, this has two main disadvantages. First, it can result in a massive blow-up of the formula size, due to the nature of some operations (e.g. multiplication of two bit-vectors). Second, heavily relying on Boolean reasoning may put MiniZinc solvers at a disadvantage when trying to solve the generated formulas.
An alternative solution is to represent SMT-LIB bit-vectors using the MiniZinc int data-type treated as a binary sequence of bits. On the one hand, this approach provides the most-efficient handling of arithmetic operations over bit-vectors (both signed and unsigned). On the other hand, this encoding can only be applied to bit-vectors whose width is less than the int data-type used by the target MiniZinc application (i.e. less than bits). All things considered, we opted for this approach because it resulted in the most compact, and easy to handle, encoding of the problem.
Other Omt Functionalities.
Several problems of OMT interest require the capability of dealing with soft-constraints (i.e. Weighted MaxSMT) and also with multiple objectives, that are either considered independent goals or combined in a Lexicographic or Pareto-like fashion. To the best of our knowledge, the MiniZinc standard does not allow for an explicit encoding of soft-constraints, nor to deal with more than one objective function at a time.
We encode (weighted) MaxSMT problems using a standard Pseudo-Boolean encoding, such as the one used in [st-ijcar12]. When dealing with OMT problems that contain goals , for , we use the following approach. If these objectives are independent targets, we generate MiniZinc models, each with a different goal , and separately solve each model. If instead the multiple objectives belong to a Lexicographic OMT problem, then we generate a unique MiniZinc model that leverages the lexicographic-optimization functionality provided by MiniSearch [MiniSearch15]. (In all other cases, MiniSearch is not used). We do not have any encoding for dealing with Pareto-optimization, yet.
5 Experimental Evaluations
In this section we present an extensive empirical evaluation comparing OMT tools with many state-of-the-art CP tools on CP problems coming from the MiniZinc challenge (§5.1), and on OMT problems coming mostly from formal verification (§5.2).
The OMT solvers under evaluation are bclt, OptiMathSAT (v. 1.6.0) and Z3 (v. 4.8.5). These are compared with some of the top-scoring solvers that participated at recent editions of the MiniZinc challenge, including Choco (v. 4.0.4), Chuffed, G12(fd) (v. 1.6.0), Gecode (v. 6.0.1), Gurobi (v. 8.0.1), HaifaCSP (v. 1.3.0), JaCoP (v. 4.5.0), iZplus (v. 3.5.0), OR-Tools (v. 6.7.4981) and Picat (v. 2.4).
We could not include fzn2smt [bof10sys, bof12] in our experimental evaluation because it is not compatible with the features of MiniZinc that have been added since version 2.0.
We run all these experimental evaluations on two identical 8-core 2.20Ghz Xeon machines with of RAM and running Ubuntu Linux. All the benchmark-sets, the tools and the scripts used to run these experiments, and some of the plots for the results in Tables 1-3 which could not fit into this paper, can be downloaded from [expeval_url].
We stress the fact that the goal of these experiments is not to establish a winner among OMT and MiniZinc tools; rather, it is to assess the correctness, effectiveness and efficiency of our OMT-to-CP and CP-to-OMT encoders and, more generally, to investigate the feasibility of solving MiniZinc problems with OMT tools and vice versa, and to identify the criticalities in terms of efficiency and correctness in these processes.
5.1 Evaluation on MiniZinc Benchmark Sets
We consider the benchmark-sets used at the MiniZinc Challenge of (MC16) and (MC19), each comprised by instances. For compatibility reasons, the version of mzn2fzn used to convert the problems to the FlatZinc format differs between the two benchmark-sets. We use version and (with patches) for the problems in MC16 and MC19 respectively. Due to recent changes in the FlatZinc format that affect the benchmarks in MC19, the version of some MiniZinc tools differs from what described in Section §5 (see Table 1). In some cases, we had to download and compile the latest source available for the tool, i.e. the “nightly” version.
We run each MiniZinc solver with the corresponding directory of global constraints, and we run each MiniZinc and OMT tool with the default options. We consider two OMT encodings of the original FlatZinc problems, la and bv. The first encodes the FlatZinc int type with the theory of linear integer arithmetic, whereas the second is based on the theory of bit-vectors. We evaluate each OMT solver on both SMT-LIB encodings, except for bclt that has no support for bit-vector optimization. For uniformity reasons with the other OMT solvers, we evaluate OptiMathSAT using its SMT-LIB interface only, using thus its fzn2omt interface as an external tool, like with the other OMT solvers. We note that the solving time for all OMT solvers includes the time required for translating the formula from the FlatZinc to the SMT-LIB format. Each solver, either OMT or MiniZinc, is given up to to solve each problem, not including the time taken by mzn2fzn to flatten it.
We verify the correctness of the results by automatically checking that all terminating solvers agree on the (possibly optimal) solution and, when this is not the case, we manually investigate the inconsistency.
|tool, configuration & encoding||s|||||o||s|||||o||s|||||o||s|||||o||s|||||o||s|||||o||s|||||o||s|||||o||s|||||o|
|MiniZinc Challenge 2016|
|MiniZinc Challenge 2019|
The results of this experiment are shown in Table 1, with separate numbers for satisfiability (s) and optimization (o) instances in each benchmark-set. Using the experimental data, we separately computed the virtual best configuration among all MiniZinc solvers (i.e. virtual best(MiniZinc)), all OMT solvers (i.e. virtual best(OMT)), and also the virtual best among all tools considered in the experiment (i.e. virtual best(all)). The last two columns in the table list the number of problems solved by the given configuration in the same amount of time as the virtual best() of each group (col. BT1) and as the virtual best(all) (col. BT2).
We start by looking at the MiniZinc solvers in Table 1. The performance ladder is dominated by OR-Tools(sat) and Picat(sat), closely followed by Gurobi, HaifaCSP and Chuffed (in MC19). By looking at column BT1, we observe that the top-performing MiniZinc solvers tend to dominate over all the others. Looking at the results of the MC19 experiment, we notice a significant increase in the number of errors with respect to the benchmark-set of the MC16 edition, as well as a handful of problems solved incorrectly. In the case of Gurobi and Picat(sat), the mzn2fzn compiler encountered an error over a few instances. As a consequence, the total number of problems is smaller than for both tools. After taking a closer look, we ascribe this phenomenon to the recent changes in the MiniZinc/FlatZinc format, that has created some minor issues with some tools that have not been adequately updated.
Looking at the OMT tools only, we observe that Z3 has leading performance over the other solvers. When compared to the MiniZinc solvers, the OMT solvers place themselves in the middle of the rank on both benchmark-sets. Given the fact that none of the OMT solvers has specialized procedures or encodings for dealing with global constraints, we consider this an interesting result.
5.2 Evaluation on Omt Benchmark Sets
In this experimental evaluation we use OMT formulas taken from well-known, publicly available, repositories. We characterize these benchmark-sets as follows:
SAL [integers]: SMT-based Bounded Model Checking and K-Induction parametric problems created with the SAL model checker [salurl];
SAL [rationals]: as above, with problems on the rationals;
Symba [rationals]: bounded222 We discarded any unbounded instance in the original benchmark-set in [li_popl14]. software verification instances derived from a set of C programs used in the Software Verification Competition of 2013 [li_popl14];
Jobshop and Strip Packing [rationals]: problems taken from [DBLP:journals/cce/SawayaG05, st-ijcar12];
Machine Learning [rationals]: OMT instances generated with the pyLMT tool [pylmt_url] based on Machine Learning Modulo Theories [teso17].
The first benchmark-set is on the integers, whereas the other four are on the rationals. We stress the fact that all formulas contained in all benchmark-sets are satisfiable.
Although there exists a repository of multi-objective OMT formulas (e.g. [li_popl14, st_tacas15]), we have chosen to not include these in our experimental evaluation. The reason for this is twofold. First, such comparison would likely be unfair wrt. CP tools because that the workaround for dealing with multi-independent OMT formulas described in Section §4 is not competitive with the integrated optimization schema provided by OMT solvers [li_popl14, st_tacas15]. In fact, the experimental evidence in [li_popl14, st_tacas15] collected on a group of OMT solvers indicates that the latter approach can be an order of magnitude faster than the former one. Second, the workaround for dealing with lexicographic-optimization is limited by the fact that MiniSearch is not fully compatible with recent versions of MiniZinc, and it only works with a restricted set of tools.
We have used the omt2mzn tool described in Section §4 to translate each OMT formula to the MiniZinc format. omt2mzn is written in Python and it is built on top of pySMT [PySMT], a general-purpose Python library for solving SMT problems, and it is available at [omt2mzn]. During this step, it has been necessary to impose a finite domain to any unconstrained SMT-LIB rational variable, because otherwise none of the MiniZinc solvers would have been able to deal with them. We have experimented with two different domains: the largest feasible domain for floating-point variables of bits (i.e. ) for the first two benchmark-sets, and the largest feasible domain for integer variables (i.e. ) for the last two.
We consider two OptiMathSAT configurations: OptiMathSAT(smt), solving the original OMT formulas, and OptiMathSAT(fzn), executed on the generated MiniZinc instances. The benefits of this choice is two-fold. First, we can double-check the correctness of such encoding, by comparing the optimum models generated in the two cases. Second, we can verify whether there is any performance loss caused by the encoding of the formula.
Only four of the MiniZinc solvers listed in Section §5 support floating-point reasoning. This limited the number of tools that could be used with some OMT benchmark-sets. The running-time of each MiniZinc solver reported in these experiments (including OptiMathSAT(fzn)) is comprehensive of the time taken by the mzn2fzn compiler, because the latter can sometime solve the input formulas on its own. The overall timeout is set to .
Notice that the optimal solutions found by OptiMathSAT(smt) have been previously independently verified with a third-party SMT tool as reported in previous publications [st_cav15, st_tacas15, st_tacas17].333 For every OMT problem s.t. OptiMathSAT(smt) returns a minimum value for on the formula , we say is correct iff is satisfiable and is unsatisfiable. (Dual for maximization.) Therefore, we verify the correctness of the results found by any other configuration by comparing them with those found by OptiMathSAT(smt), and otherwise mark the result as “unverified”.
|Tools & Configuration||inst.||timeout||tool-er.||unsupp.||incor.||correct||time(s)|
Experimental Results over the Integers.
In this experiment, we evaluate the SAL (over integers) benchmark-set. The results are collected in Table 2.
We notice first that OptiMathSAT(fzn) always produces correct results and it shows comparable performances in terms on number of problems solved wrt. the baseline OptiMathSAT(smt), solving even 4 problems more. (We conjecture that the latter fact should be attributed to the limited, but effective, deduction capabilities of the mzn2fzn compiler, that may have helped OptiMathSAT in solving the input formulas.) This suggests that, at least on problems on the integers, omt2mzn is efficient and effective and does not affect correctness.
In general, MiniZinc solvers do not seem to deal efficiently with this benchmark-set. Some tools have experienced some internal error (e.g. dumped-core, segmentation fault), some others have been killed to to a high memory consumption (over GB), whereas the majority of the remaining tools had a timeout.
We explain this behavior with the fact that the given benchmark set is characterized by the presence of a heavy Boolean structure combined with arithmetical constraints, which requires the efficient combination of strong Boolean-reasoning capabilities (e.g., efficiently handling chains of unit propagations) with strong arithmetical-solving&optimization capabilities, which is a typical feature of OMT solvers.
None of the input formulas was initially supported by Gurobi. After restricting the bound of every integer variable to , Gurobi(l) was able to solve instances within the timeout. Among the MiniZinc solvers, the best result is obtained by Picat(sat), that solved problems out of .
|Tools & Configuration||
|SAL over rationals|
|Software Verification (Symba)|
|JobShop + Strippacking|
Experimental Results Over the Rationals.
We consider first the first three benchmark-sets over the rationals: SAL over rationals, Symba, JobShop&Strip-Packing. Of all MiniZinc solvers we have tried, only three are able to deal with floating-point constraints. The results are shown in Table 3. Since each of the input formulas is satisfiable, we consider a result incorrect either when it is equal to unsat, or when the relative error exceeds , s.t.: and being the optimal value found by OptiMathSAT(smt) and the optimal value found by the MiniZinc solver under test respectively. (Recall that the former was previously checked to be correct.)
Similarly to the previous experiment on the integers, OptiMathSAT(fzn) always produces correct results, and display comparable performance wrt. OptiMathSAT(omt) in terms of number of instances being solved, solving somewhat fewer problems. This is not the case of the other three MiniZinc solvers. Among these, Gecode experienced a timeout on the majority of the formulas being considered, G12(mip) returned mostly incorrect answers, whereas Gurobi seems to have the best performance, in particular on the third benchmark-set.
We attribute the large number of incorrect results returned by all three MiniZinc solvers to the fact that these tools use finite-precision floating-point arithmetic internally. The incorrect behavior of some of these solvers (e.g. Gurobi) can also be partially explained with the large domain of floating-point variables in these problems. However, given the nature of these input instances, it was not possible for us to assign a smaller domain to each variable in the problem a priori.
We analyze separately the results for the last benchmark-set reported in Table 3. The peculiar aspect of the Machine Learning benchmark-set [teso17] is that it is characterized by Pseudo-Boolean sums over rational weights, and by very fine-grained rational values444For example, , which expands to , is a sample weight value from problems in [teso17].. Unfortunately, these fine-grained rational values are rounded by the standard mzn2fzn compiler, which causes the incorrect results even of OptiMathSAT(fzn) in Table 3, despite the fact that OptiMathSAT uses infinite-precision arithmetic.
In order to overcome this issue, we leverage the emzn2fzn compiler described in Section §3 to preserve the original fractional values, and show that with this approach OptiMathSAT does not produce incorrect results any longer (configuration OptiMathSAT(fzn+e) in Table 3), solving correctly 33 problems more than OptiMathSAT(fzn).
Despite this improvement, however, OptiMathSAT(fzn+e) still solves much less problems than OptiMathSAT(smt). We have analyzed this fact, and we noticed that the performance gap is due to the structural changes introduced by the mzn2fzn compiler, which prevents OptiMathSAT(fzn+E) to use some efficient solving techniques when dealing with the same problem given to OptiMathSAT(smt). 555For example, an equality like where are float values, in an unbounded float variable and are float variables, is rewritten by mzn2fzn into the equality , where the Pseudo-Boolean sum is added to an arithmetic term, preventing OptiMathSAT(fzn+E) from recognizing it as the definition of a Pseudo-Boolean sum, and hence from activating the specialized solving technique for Pseudo-Boolean equalities which are instead used by OptiMathSAT(smt).
Overall, since there are at least formulas affected by the above issue with the mzn2fzn compiler, we avoid an in-depth discussion of the results obtained by the other MiniZinc solvers. However, at a first glance the situation does not seem to differ from the other benchmark-sets over the rationals.
On the whole, from our experiments, OMT tools appear to be still at some disadvantage when dealing with MiniZinc problems wrt. specific tools, and vice versa.
On the one hand, OMT solvers seem to be penalized by their lack of efficient ad hoc decision procedures for dealing with global constraints. Moreover, the approach taken by the mzn2fzn compiler, that creates lots of alias Boolean, integer and floating-point variables for dealing with Pseudo-Boolean constraints, is particularly challenging to deal with efficiently by an OMT solver.
On the other hand, MiniZinc solvers seem to suffer with problems needing an arithmetic-reasoning component combined with heavy Boolean-reasoning component. Even more importantly, the lack of infinite-precision linear arithmetic procedures causes a number of incorrect results when dealing with OMT problems over the rationals. Both of these points need to be addressed in order to deal with the vast number of Formal Verification and Model Checking applications in the SMT/OMT domain.
6 Conclusions & Future Work
In this paper we have taken a first step forward towards bridging the MiniZinc and the OMT communities. The ultimate goal is to obtain a correct, effective and efficient fully-automated system for translating problems from one community to the other, so as to extend the application domain of both communities. With our experimental evaluation, we have identified some criticalities that need to be addressed by each community in order to solidify this union.
We plan to push this investigation forward as follows. In the short term, we plan to address the inefficient handling of Pseudo-Boolean constraints over the rationals revealed by the experimental evaluation in Section §5.2. In order to deal with those FlatZinc constraints that require non-linear arithmetic, we envisage an opportunity to either extend OptiMathSAT with proper handling of the non-linear arithmetic theory [ac_tocl18] or to experiment with an encoding based on the floating-point theory [ts_cade19]. This objective goes hand in hand with the extension of omt2mzn to deal with other SMT theories. In the long term, OMT solving may also benefit from adopting efficient ad hoc decision procedures for frequently used global constraints. Finally, we plan to broaden the scope of our investigation and include other OMT solvers in our study.