1 Introduction
The last two decades have witnessed the rise of Satisfiability Modulo Theories (SMT) [BSST09HBSAT] as efficient tool for dealing with several applications of industrial interest, in particular in the contexts of Formal Verification (FV). SMT is the problem of finding value assignments satisfying some formula in firstorder logic wrt. some background theory. Optimization Modulo Theories (OMT) [nieuwenhuis_sat06, stijcar12, li_popl14, LarrazORR14, st_tacas15, bjorner_tacas15, NadelR16, kovas18] is a morerecent extension of SMT searching for the optimal value assignment(s) w.r.t. some objective function(s), by means of a combination of SMT and optimization procedures. (Since OMT extends SMT, and that all OMT tools are built on top of SMT solvers, hereafter we often simply say “OMT” for both SMT and OMT.)
Several distinctive traits of OMT solvers –like, e.g., the efficient combination of Boolean and arithmetical reasoning, incrementality, the availability of decision procedures for infiniteprecision arithmetic and the ability to produce conflict explanations– are a direct consequence of their tight relationship with the FV domain and its practical needs. On the whole, it appears that OMT can be a potentially interesting and efficient technology for dealing with Constraint Programming (CP) problems as well. At the same time, modeling CP problems for OMT solvers requires a higherlevel of expertise, because the same CP instance can have many possible alternative formulations, s.t. the performance of SMT solvers on each encoding are hardly predictable [frisch09, frisch10, elgabou14].
On the other hand, the availability, efficiency and expressiveness of CP tools makes them of potential interest as backend engines also for FV applications (e.g., [Collavizza2006, Collavizza2010, Grinchtein2015]), in particular with SW verification, where currently SMT is the dominating backend technology, s.t. a large amount of SMTencoded FV problems are available [brst10_smtlib].
In this paper we aim at bridging the gap between CP and OMT, in both directions.
In the CPtoOMT direction, we have extended the stateoftheart OMT solver OptiMathSAT [st_jar18] with a FlatZinc interface (namely “fzn2omt”). In combination with the standard mzn2fzn encoder [minizinc_url], this new interface can be used to either (i) solve CP models with OptiMathSAT directly or (ii) generate OMT formulas encoded in the SMTLIB [cts_cpaior20_extended] format with optimization extensions, to be fed to other OMT solvers, such as bclt [Bofilletal2008CAV] and Z3 [bjorner_tacas15]. This allows stateoftheart OMT technology to be used on MiniZinc problems coming from the CP community.
In the OMTtoCP direction, we have introduced a tool for translating SMT and OMT problems on the theories of linear arithmetic over the integers and rational () and bitvector () into MiniZinc models (hereafter “omt2mzn”). This allows MiniZinc solvers to be used on OMT problems, giving them access to a large amount of OMT problems, mostly coming from formal verification.
With both directions, we first present and discuss the challenges we encountered and the solutions we adopted to address the differences between the two formalisms. Then we present an extensive empirical evaluation comparing three OMT tools with many stateoftheart CP tools on (i) CP problems coming from the MiniZinc challenge, and (ii) OMT problems coming mostly from formal verification. This analysis allowed us to identify some criticalities, in terms of efficiency and correctness, one has to cope with when addressing CP problems with OMT tools, and vice versa.
Overall, our new compilers fzn2omt and omt2mzn in combination with the standard compiler mzn2fzn [minizinc_url] provide a framework for translating problems encoded in the SMTLIB or the MiniZinc format in either direction. This framework enables also for a comparison between OMT solvers and CP tools on problems that do not belong to their original application domain. To the best of our knowledge, this is the first time that such a framework has been proposed, and that the OMT and CP technologies have been extensively compared on problems coming from both fields.
Related Work.
The tight connection between SMT and Constraint Programming (CP) has been known for a relatively long period of time [nieuwenhuisetal2007rta] and it has previously been subject to investigation. Some works considered a direct encoding of CP [frisch09, frisch10] and weighted CP [ansotegui13] into SMT and MaxSMT, or an automatic framework for translating MiniZinc –a standard CP modeling language [stuckey_mzn_07]– into SMTLIB –the standard SMT format– [bof10sys, bof12]. Other works explored the integration of typical SAT and SMT techniques within CP solvers [ohriSC09, feydy09]. Nowadays, several MiniZinc solvers –like, e.g., HaifaCSP [vekslerS16] and Picat [zhou17]– are at least partially based on SAT technology.
To this extent, our first contribution fzn2omt also obviates the loss, due to obsolescence, of the fzn2smt compiler proposed by Bofill et al. in [bof10sys, bof12]. fzn2smt is not compatible with the changes that have been introduced to the MiniZinc and FlatZinc standards starting from version of the MiniZinc distribution. Since some of these changes are not backward compatible, it is also not possible to use fzn2smt in conjunction with an older version of the mzn2fzn compiler when dealing with recent MiniZinc models. Furthermore, fzn2smt translates satisfaction problems into the Version of the SMTLIB standard and produces no SMTLIB output in the case of optimization problems, that are solved directly. However, the optimization interface of modern OMT solvers is based on the Version of the SMTLIB standard. This makes it difficult to use it together with OMT solvers. Unfortunately, the fzn2smt compiler is closed source, with only the binaries being freely distributed, and seemingly no longer maintained. This made it necessary to provide a new alternative solution to fzn2smt. To this extent, our new FlatZinc interface of OptiMathSAT, fzn2omt, translates both satisfaction and optimization problems in the Version of the SMTLIB standard enriched with the optimization extensions for OMT described in [st_jar18].
Content.
The rest of the paper is organized as follows. In §2 we provide some background on OMT, MiniZinc and FlatZinc. In §3 we describe the process from MiniZinc to OMT. In §4 we describe the process from OMT to MiniZinc. In §5 we describe an empirical evaluation comparing a OMTbased tool with many stateoftheart CP tools. Finally, in §6 we conclude and point out some further research directions.
A longer and more detailed version of this paper is publicly available as [cts_cpaior20_extended].
2 Background
Satisfiability Modulo Theories (SMT) is the problem of deciding the satisfiability of a firstorder formula with respect to a combination of decidable firstorder theories. Typical theories of SMT interest are (the theory of) linear arithmetic over the rationals (), the integers () or their combination (), nonlinear arithmetic over the rationals () or the integers (), arrays (), bitvectors (
), floatingpoint arithmetic (
), and their combinations thereof. (See [nieotjacm06, sebastiani07, BSST09HBSAT] for an overview.). The last two decades have witnessed the development of very efficient SMT solvers based on the socalled lazySMT schema [sebastiani07, BSST09HBSAT]. This has brought previouslyintractable problems to the reach of stateoftheart SMT solvers.Optimization Modulo Theories (OMT), [nieuwenhuis_sat06, stijcar12, li_popl14, LarrazORR14, st_tacas15, bjorner_tacas15, NadelR16, ts_cade19], is an extension to SMT that allows for finding a model of a firstorder formula that is optimal with respect to some objective function expressed in some background theory, by means of a combination of SMT and optimization procedures. Stateofthe art OMT tools allow optimization in a variety of theories, including linear arithmetic over the rationals () [stijcar12] and the integers () [bjorner_tacas15, st_tacas15], bitvectors () [NadelR16] and floatingpoint numbers (OMT ()) [ts_cade19].
A relevant strict subcase of is OMT with PseudoBoolean objective functions () in the form s.t. are rational values and are Boolean variables whose values are interpreted as . Notice that is also equivalent to (partial weighted) MaxSMT, the SMT extension of MaxSAT, and that and MaxSMT can be encoded into but not vice versa [st_tocl14]. Encoding /MaxSMT into , however, is not the most efficient way to solve them, so that modern OMT solvers such as bclt [Bofilletal2008CAV], OptiMathSAT [st_jar18] and Z3 [bjorner_tacas15] implement specialized /MaxSMT procedures which are much more efficient than generalpurpose ones [bjorner_tacas15, st_tacas17, st_jar18].
We stress the fact that —unlike with purelycombinatorial problems, which are encoded into SAT or MaxSAT and are thus solved by purelyBoolean search– typically OMT problems involve the interleaving of both Boolean and arithmetical search: search not only for the best truthvalue assignment to the atomic subformulae, but also for the best values to the numerical variables compatible with such truthvalue assignment [st_tocl14].
To this date, few OMT solvers exist, namely bclt [Bofilletal2008CAV], Cegio [araujo16], Hazel [NadelR16], OptiMathSAT [st_jar18], Puli [kovas18], Symba [li_popl14] and Z3 [bjorner_tacas15]. To this aim, we observe that (i) some of these solvers are quite recent, (ii) most of these solvers focus on different, partially overlapping, niche subsets of Optimization Modulo Theories, and (iii) the lack of an official Input/Output interface for OMT makes it hard to compare some of these tools with one another. OMT finds applications in the context of static analysis [candeago16, karpenkov17], formal verification and model checking [liutbt17, ratschan17], scheduling and planning with resources [kovas18, leofante18, roselli18, oliver18], software security and requirements engineering [nguyensgm16], workflow analysis [bertolissi18]
[teso17], and quantum computing [bian17_frocos17].Remark 1
A distinctive trait of SMT (and OMT) solvers is the tradeoff of speed against the ability to certify the correctness of the result of any computation, which is particularly important in the contexts of Formal Verification (FV) and Model Checking (MC). When dealing with linear arithmetic in particular, SMT solvers employ infiniteprecision arithmetic software libraries to avoid numerical errors and overflows.
SMTLIB [cts_cpaior20_extended] is the standard input format by SMT solvers, it provides a standardized definition of the most prominent theories supported by SMT solvers and the corresponding language primitives to use these features. At present, there is no standard input format for modeling optimization problems targeting OMT solvers, although there exist only minor syntactical differences between the major OMT solvers. The tools presented in this paper conform to the extended SMTLIB format for OMT presented in [st_jar18], that includes language primitives for modeling objectives.
OptiMathSAT [stijcar12, st_tocl14, st_cav15, st_tacas15, st_tacas17, st_jar18] is a stateoftheart OMT solver based on the MathSAT5 SMT solver [mathsat5url, mathsat5_tacas13]. OptiMathSAT features both single and multiobjective optimization over arbitrary sets of , , , , , PseudoBoolean () and MaxSMT cost functions. Multiple objective functions can be combined with one another into a Lexicographic or a Pareto optimization problem, or independently solved in a single run (for the best efficiency).
MiniZinc [stuckey_mzn_07, minizinc_url] is a widely adopted highlevel declarative language for modeling Constraint Satisfaction Problems (CSP) and Constraint Optimization Problems (COP). The MiniZinc format defines three scalar types (bool, int and float) and two compound types (sets and fixedsize arrays of some scalar type). The standard provides an extensive list of predefined global constraints, a class of highlevel language primitives that allows one to encode complex constraints in a compact way. Furthermore, MiniZinc supports useful language constructs such as ifthenelse, let expressions, universal and existential comprehensions over finite domains, userdefined predicates and more.
FlatZinc is a lowerlevel language whose purpose is to bridge the gap between the highlevel modeling in MiniZinc and the need for a fixed, and easytoparse, format that can simplify the implementation of the input interface of MiniZinc solvers. A MiniZinc model is typically flattened into a FlatZinc instance using the mzn2fzn compiler [minizinc_url], and then solved with some MiniZinc tool.
3 From MiniZinc to Omt
We consider the problem of translating MiniZinc models into OMT problems first. Similarly to other MiniZinc solvers, we assume that the MiniZinc model is first translated into FlatZinc using the mzn2fzn standard compiler, as depicted in Figure 1. We describe the main aspects of fzn2omt, focusing on the challenges we have encountered and on the solutions we have adopted.
FlatZinc datatypes.
The first challenge is to find a suitable representation of the datatypes supported by FlatZinc in SMTLIB.
One possible choice for modeling the three basic scalar types of FlatZinc –namely bool, int and float– with SMTLIB are the Boolean, bitvector and floatingpoint theories respectively. However, the decision procedures for the bitvector and floatingpoint numbers theories can be significantly more resource demanding than the decision procedure for the linear arithmetic theory (), in particular when dealing with a substantial amount of arithmetic computations. For this reason, we have opted to model FlatZinc int and float datatypes with the SMTLIB integer and rational types respectively, by default. For the case in which no substantial linear arithmetic computation is performed, we also optionally allow for encoding the FlatZinc int datatype as a SMTLIB bitvector.
For what concerns the two compound types of FlatZinc, that is the set and array datatypes, we have chosen to proceed as follows. Given that OptiMathSAT lacks a decision procedure for the theory of finite sets [sets_url], we model a set using the Boolean and integer theories, similarly to what has been done in [bof12]. The basic idea is to introduce a fresh Boolean variable for each element in the domain of a set, and use such variable as a placeholder for the membership of an integer element to the set instance. Differently from [bof12], we make an extensive use of cardinality networks [Asin2011] to encode constraints over the sets because they are handled more efficiently, for their nice arcconsistency properties. No action is required to encode a FlatZinc array into SMTLIB, because it is used only as a container for other variables.
FloatingPoint precision.
A consequence of encoding the FlatZinc int and float datatypes with the linear arithmetic theory is that all of our computation is performed with infiniteprecision arithmetic. This can result in a performance disadvantage wrt. other MiniZinc solvers using finiteprecision arithmetic, due to the increased cost of each operation, but it has the benefit of guaranteeing the correctness of the final result of the computation.
Currently, the MiniZinc language does not allow one to express a certain quantity as an infiniteprecision fraction between two constant numbers. Instead, the mzn2fzn compiler computes onthefly the result of any division operation between two constant integers or floatingpoint numbers applying the rules corresponding to the type of the operands. However, there are some instances in which we really need to be able to both express quantities and perform computation with infiniteprecision arithmetic. One of such situations is to doublecheck the correctness of the MiniZinc models generated by the omt2mzn compiler described in Section §4 (we have done this for the experimental evaluations in Section §5.2). In order to get around this limitation we developed a simple wrapper around the mzn2fzn compiler, called emzn2fzn [emzntofzn_url], that replaces any fraction among two constant floats with a fresh variable, and then sets its value using the FlatZinc constraint float_div().
FlatZinc constraints.
The SMTLIB encoding of the majority of FlatZinc constraints in OptiMathSAT follows their definition in the FlatZinc Standard, with the exception of PseudoBoolean constraints, which we examine in detail later on. Several global constraints are also supported in the same way, because the OMTsolver currently lacks ad hoc and efficient decision procedures for dealing with them. Constant values and alias variables (e.g. those arising from the definition of some arrays) are propagated through the formula, so as to keep the set of problem variables as compact as possible. Those constraints requiring nonlinear arithmetic –like, e.g., trigonometric, logarithmic and exponential functions– are currently not supported; this situation may change soon due to the recent extension of MathSAT5 with a procedure for it [ac_tocl18].
PseudoBoolean constraints.
When dealing with PseudoBoolean sums of the form , where is a Boolean variable and is a numerical weight, the mzn2fzn compiler associates a fresh / variable to each , and encodes the sum as . Notice that the original s may not be eliminated from the FlatZinc model, because they typically occur elsewhere in the problem, i.e. as part of a Boolean formula. From our own experience, this situation arises frequently, because PseudoBoolean sums are typically used to express cardinality constraints that have a variety of uses. As described in [st_tacas17], one limitation of this naive approach is that SMT and OMT solvers do not typically handle this encoding efficiently. The main reason is that the pruning power of the conflict clause resulting from a conflicting assignment is typically limited to one specific Boolean assignment at a time, meaning that a large number of conflict clauses (possibly exponential) has to be generated along the search. As shown in [st_tacas17], SMT and OMT solvers can benefit from encoding PseudoBoolean constraints with cardinality networks.
fzn2omt goes through some effort in order to recognize PseudoBoolean sums over the integers, and replace the naive encoding with one based on cardinality networks. We note that using this technique generally results in a tradeoff between solving time and the overhead of generating cardinality networks prior to starting the search, especially when dealing with a large number of variables.
Multiobjective optimization.
fzn2omt allows for multiple optimization goals, of heterogeneous type, being defined within the same FlatZinc model. This is a nonstandard extension to the FlatZinc format. Multiple objectives can be solved independently from one another, or combined into a Lexicographic or Pareto optimization goal. We refer the reader to [st_jar18] for details on the input encoding and the solver configuration.
Functionality.
Given a satisfiability or optimization problem encoded in the FlatZinc format, OptiMathSAT can be used in the following ways (Figure 1):

to directly solve the problem, optionally enumerating any suboptimal solution found during the search or all possible solutions with the same optimal value;

to produce an OMT problem encoded with the extended SMTLIB format described in [st_jar18]. This problem can be directly solved with OptiMathSAT or, with minor transformations^{1}^{1}1To make this step as easy as possible, we collected our scripts into a public repository [fzntoomt_url]., fed as input to other OMT solvers such as bclt and Z3.
4 From Omt to MiniZinc
In this section, we consider the problem of translating OMT formulas, encoded in the optimizationextended SMTLIB format of [st_jar18], into MiniZinc models. Hereafter, we describe the main challenges we have faced and the solutions we have adopted. Further details about this conversion are available in [omt2mzn].
General Translation Approach.
The main challenge is to design an encoding from OMT to MiniZinc that is correct (i.e., it preserves in full the semantics of the input OMT problems), effective (i.e., it produces as output MiniZinc models which are as compact and easytosolve as possible), and efficient (i.e. it does it with the least consumption of time and memory). For best effectiveness, the internal representation of the input OMT formula adopted by the compiler is that of a Directed Acyclic Graph (DAG), that represents repeating subformula elements with the same node. A simple, treelike, traversal of this datastructure can result in some node –and its entire subtree– being visited multiple times during the translation of the OMT formula, and hence in a potentially exponential blowup of the generated MiniZinc model (leftmost tree in Figure 2). The converse approach, that is, to assign an explicit label to each node of the DAG and visit each node exactly once, can suffer from the creation of a large number of fresh variables that do not exist in the original problem, even for those nodes that would be traversed only once (center DAG in Figure 2). To cope with this problem, we adopted the idea of assigning a label only to those DAG nodes that have at least two father nodes (rightmost DAG in Figure 2). This limits the number of artificial variables introduced in the encoding, and at the same time produces the most compact representation of the input problem. In our experiments, the latter idea showed to be essential to produce much more compact and easytosolve problems.
Theories Restriction.
The SMTLIB standard describes a wide number of SMT theories, most of which have no direct counterpart in MiniZinc due to the few datatypes supported (see §2). Hence, hereafter we deal only with the theory of linear rational and integer arithmetic, their combination, and the theory of bitvectors. We leave the handling of other SMT theories to future work.
Linear Arithmetic Theory.
On the surface, encoding linear arithmetic constraints coming from OMT in MiniZinc, using the int and float datatypes, looks like a trivial task. In reality, this poses several challenges and it is subject to several limitations, due to a couple of facts.
First, in SMTLIB the linear arithmetic theory requires the capability to perform infiniteprecision computations. Unfortunately, to the best of our knowledge, no MiniZinc solver provides infiniteprecision arithmetic reasoning, and the mzn2fzn compiler itself prevents representing arbitrarilylarge and arbitrarilyprecise quantities (e.g. the finegrained decimal weights of the machine learning application in [teso17]).
Second, in OMT linear arithmetic variables are not required to be bounded and have quite often no explicit domain (i.e. they lack a lowerbound, an upperbound or both), because it is not necessary for the problem at hand or it is implied by other constraints. This is in contrast with MiniZinc, whereby linear arithmetic variables are expected to have a finite domain and, when they lack one, their domain appears to be capped with some solverdependent pair of values.
These restrictions are currently part of the MiniZinc language and the target application domain, and we do not see any obvious workaround solving them. We note that although there exist methods for bounding all variables in a given LP problem (e.g. [papadimitriou81]), these have been deemed too impractical at this stage of our investigation. Nonetheless, we have chosen to translate SMTLIB linear arithmetic constraints with a corresponding MiniZinc encoding based on the int and float datatypes. Although the encoding is not always applicable, it does still allow one to correctly translate a number of interesting OMT problems into MiniZinc, as witnessed by our experimental evaluation in Section §5.2. More in detail, the translation is done as follows. We declare each integer variable as unbounded, and then extend the MiniZinc model with the appropriate constraints bounding its domain when the input OMT formula contains any such information. Our empirical observation is that MiniZinc models generated in this way are correctly handed by all MiniZinc solvers which we have tried, with the exception of Gurobi, which returns an “unsupported” message. FloatingPoint variables, instead, are always declared with a userdefined domain. This is because all of the MiniZinc solvers we have tried, among those that can handle floatingpoint constraints, require such information.
BitVector Theory.
Dealing with bitvectors is also challenging. In essence, an OMT bitvector can be seen as a vector of bits of fixed size, that can be represented as a Boolean array in MiniZinc. However, because bitvectors are typically used to encode machine registers and numbers at the binarylevel, the bitvectors theory provides both signed and unsigned arithmetic operators, as well as those based on Boolean logic. Performing arithmetic computation over a Boolean array in MiniZinc requires one to define and instantiate of the Boolean circuit encoding the given operation as a set of constraints. As far as we can see, this has two main disadvantages. First, it can result in a massive blowup of the formula size, due to the nature of some operations (e.g. multiplication of two bitvectors). Second, heavily relying on Boolean reasoning may put MiniZinc solvers at a disadvantage when trying to solve the generated formulas.
An alternative solution is to represent SMTLIB bitvectors using the MiniZinc int datatype treated as a binary sequence of bits. On the one hand, this approach provides the mostefficient handling of arithmetic operations over bitvectors (both signed and unsigned). On the other hand, this encoding can only be applied to bitvectors whose width is less than the int datatype used by the target MiniZinc application (i.e. less than bits). All things considered, we opted for this approach because it resulted in the most compact, and easy to handle, encoding of the problem.
Other Omt Functionalities.
Several problems of OMT interest require the capability of dealing with softconstraints (i.e. Weighted MaxSMT) and also with multiple objectives, that are either considered independent goals or combined in a Lexicographic or Paretolike fashion. To the best of our knowledge, the MiniZinc standard does not allow for an explicit encoding of softconstraints, nor to deal with more than one objective function at a time.
We encode (weighted) MaxSMT problems using a standard PseudoBoolean encoding, such as the one used in [stijcar12]. When dealing with OMT problems that contain goals , for , we use the following approach. If these objectives are independent targets, we generate MiniZinc models, each with a different goal , and separately solve each model. If instead the multiple objectives belong to a Lexicographic OMT problem, then we generate a unique MiniZinc model that leverages the lexicographicoptimization functionality provided by MiniSearch [MiniSearch15]. (In all other cases, MiniSearch is not used). We do not have any encoding for dealing with Paretooptimization, yet.
5 Experimental Evaluations
In this section we present an extensive empirical evaluation comparing OMT tools with many stateoftheart CP tools on CP problems coming from the MiniZinc challenge (§5.1), and on OMT problems coming mostly from formal verification (§5.2).
The OMT solvers under evaluation are bclt, OptiMathSAT (v. 1.6.0) and Z3 (v. 4.8.5). These are compared with some of the topscoring solvers that participated at recent editions of the MiniZinc challenge, including Choco (v. 4.0.4), Chuffed, G12(fd) (v. 1.6.0), Gecode (v. 6.0.1), Gurobi (v. 8.0.1), HaifaCSP (v. 1.3.0), JaCoP (v. 4.5.0), iZplus (v. 3.5.0), ORTools (v. 6.7.4981) and Picat (v. 2.4).
Remark 2
We could not include fzn2smt [bof10sys, bof12] in our experimental evaluation because it is not compatible with the features of MiniZinc that have been added since version 2.0.
We run all these experimental evaluations on two identical 8core 2.20Ghz Xeon machines with of RAM and running Ubuntu Linux. All the benchmarksets, the tools and the scripts used to run these experiments, and some of the plots for the results in Tables 13 which could not fit into this paper, can be downloaded from [expeval_url].
We stress the fact that the goal of these experiments is not to establish a winner among OMT and MiniZinc tools; rather, it is to assess the correctness, effectiveness and efficiency of our OMTtoCP and CPtoOMT encoders and, more generally, to investigate the feasibility of solving MiniZinc problems with OMT tools and vice versa, and to identify the criticalities in terms of efficiency and correctness in these processes.
5.1 Evaluation on MiniZinc Benchmark Sets
We consider the benchmarksets used at the MiniZinc Challenge of (MC16) and (MC19), each comprised by instances. For compatibility reasons, the version of mzn2fzn used to convert the problems to the FlatZinc format differs between the two benchmarksets. We use version and (with patches) for the problems in MC16 and MC19 respectively. Due to recent changes in the FlatZinc format that affect the benchmarks in MC19, the version of some MiniZinc tools differs from what described in Section §5 (see Table 1). In some cases, we had to download and compile the latest source available for the tool, i.e. the “nightly” version.
We run each MiniZinc solver with the corresponding directory of global constraints, and we run each MiniZinc and OMT tool with the default options. We consider two OMT encodings of the original FlatZinc problems, la and bv. The first encodes the FlatZinc int type with the theory of linear integer arithmetic, whereas the second is based on the theory of bitvectors. We evaluate each OMT solver on both SMTLIB encodings, except for bclt that has no support for bitvector optimization. For uniformity reasons with the other OMT solvers, we evaluate OptiMathSAT using its SMTLIB interface only, using thus its fzn2omt interface as an external tool, like with the other OMT solvers. We note that the solving time for all OMT solvers includes the time required for translating the formula from the FlatZinc to the SMTLIB format. Each solver, either OMT or MiniZinc, is given up to to solve each problem, not including the time taken by mzn2fzn to flatten it.
We verify the correctness of the results by automatically checking that all terminating solvers agree on the (possibly optimal) solution and, when this is not the case, we manually investigate the inconsistency.
Experiment Results.
terminated  
inst.  timeout  error  unsup.  incor.  correct  time (s.)  BT1  BT2  
tool, configuration & encoding  s    o  s    o  s    o  s    o  s    o  s    o  s    o  s    o  s    o 
MiniZinc Challenge 2016  
Picat(cp)  15    85  9    70  0    0  0    0  0    0  6    15  2281    6043  0    0  0    0 
G12(fd)  15    85  4    71  1    3  0    0  0    0  10    11  4436    4220  0    0  0    0 
Choco()  15    85  3    50  0    0  0    0  0    0  12    35  4256    11423  1    0  1    0 
iZplus()  15    85  6    44  0    0  0    0  0    0  9    41  999    5492  3    4  3    4 
Chuffed()  15    85  2    40  0    0  5    0  0    0  8    45  635    4187  0    5  0    5 
JaCoP()  15    85  3    39  0    0  0    0  0    0  12    46  3411    12825  0    0  0    0 
Gurobi()  15    85  6    22  0    0  0    0  0    0  9    63  2346    3037  0    15  0    15 
HaifaCSP()  15    85  4    23  0    0  0    0  0    0  11    62  591    4444  0    11  0    11 
Picat(sat)  15    85  1    26  0    0  0    0  0    0  14    59  151    7293  10    1  10    1 
ORTools(sat)  15    85  1    15  0    0  0    0  0    0  14    70  555    1338  1    45  1    45 
virtual best(MiniZinc)  15    85  0    7  0    0  0    0  0    0  15    78  146    3514             
OptiMathSAT(int)  15    85  10    38  0    0  0    0  0    0  5    47  604    4856  1    20  0    0 
OptiMathSAT(bv)  15    85  2    42  0    0  0    0  0    0  13    43  3664    8561  11    2  0    0 
bclt(int)  15    85  10    33  0    0  0    0  0    0  5    52  1117    5998  0    15  0    2 
Z3(int)  15    85  10    32  0    0  0    0  0    0  5    53  676    10424  0    11  0    0 
Z3(bv)  15    85  5    28  0    0  0    0  0    0  10    57  2938    11113  2    19  0    0 
virtual best(OMT)  15    85  1    21  0    0  0    0  0    0  14    64  3842    6432             
virtual best(all)  15    85  0    7  0    0  0    0  0    0  15    78  146    3514             
MiniZinc Challenge 2019  
Picat(cp) [2.7b12]  10    90  8    67  0    11  0    5  0    0  2    7  54    1440  1    0  1    0 
iZplus()  10    90  5    71  0    4  0    0  0    0  5    15  14    3077  1    3  1    3 
G12(fd)  10    90  5    64  0    10  0    0  0    0  5    16  323    4010  0    0  0    0 
Choco(std)  10    90  4    63  0    5  0    0  0    0  6    22  415    4312  0    0  0    0 
Gecode() [6.2.0]  10    90  4    63  0    0  0    0  0    0  6    27  420    5094  0    6  0    6 
JaCoP() [4.8]  10    90  4    55  0    6  0    0  0    0  6    29  260    5467  0    1  0    1 
HaifaCSP()  10    90  0    47  0    10  0    0  2    2  8    31  2    6408  4    7  4    4 
Chuffed() [nightly]  10    90  0    43  0    0  5    10  0    0  5    37  1    4886  3    19  3    19 
Gurobi() [8.1.1]  10    80  0    48  0    0  0    0  0    0  10    32  705    2895  2    6  2    4 
Picat(sat) [2.7b12]  10    90  0    45  0    5  0    0  0    1  10    39  275    9894  0    7  0    5 
ORTools(sat) [nightly]  10    90  5    42  0    3  0    0  0    0  5    45  8    7239  0    13  0    11 
virtual best(MiniZinc)  10    90  0    29  0    0  0    0  0    0  10    61  9    5247             
OptiMathSAT(int) [1.6.4.1]  10    90  5    62  0    0  0    0  0    0  5    28  4    3650  2    10  0    0 
OptiMathSAT(bv) [1.6.4.1]  10    90  4    59  0    5  0    0  0    0  6    26  484    7271  0    1  0    0 
bclt(int)  10    90  5    60  0    0  0    0  0    0  5    30  6    3369  0    6  0    5 
Z3(int)  10    90  5    64  0    0  0    0  0    0  5    26  4    5358  3    6  0    1 
Z3(bv)  10    90  0    55  0    2  0    0  0    0  10    33  1629    7550  5    17  0    3 
virtual best(OMT)  10    90  0    48  0    2  0    0  0    0  10    40  1624    5179             
virtual best(all)  10    90  0    29  0    0  0    0  0    0  10    61  9    4919             
The results of this experiment are shown in Table 1, with separate numbers for satisfiability (s) and optimization (o) instances in each benchmarkset. Using the experimental data, we separately computed the virtual best configuration among all MiniZinc solvers (i.e. virtual best(MiniZinc)), all OMT solvers (i.e. virtual best(OMT)), and also the virtual best among all tools considered in the experiment (i.e. virtual best(all)). The last two columns in the table list the number of problems solved by the given configuration in the same amount of time as the virtual best() of each group (col. BT1) and as the virtual best(all) (col. BT2).
We start by looking at the MiniZinc solvers in Table 1. The performance ladder is dominated by ORTools(sat) and Picat(sat), closely followed by Gurobi, HaifaCSP and Chuffed (in MC19). By looking at column BT1, we observe that the topperforming MiniZinc solvers tend to dominate over all the others. Looking at the results of the MC19 experiment, we notice a significant increase in the number of errors with respect to the benchmarkset of the MC16 edition, as well as a handful of problems solved incorrectly. In the case of Gurobi and Picat(sat), the mzn2fzn compiler encountered an error over a few instances. As a consequence, the total number of problems is smaller than for both tools. After taking a closer look, we ascribe this phenomenon to the recent changes in the MiniZinc/FlatZinc format, that has created some minor issues with some tools that have not been adequately updated.
Looking at the OMT tools only, we observe that Z3 has leading performance over the other solvers. When compared to the MiniZinc solvers, the OMT solvers place themselves in the middle of the rank on both benchmarksets. Given the fact that none of the OMT solvers has specialized procedures or encodings for dealing with global constraints, we consider this an interesting result.
5.2 Evaluation on Omt Benchmark Sets
In this experimental evaluation we use OMT formulas taken from wellknown, publicly available, repositories. We characterize these benchmarksets as follows:

SAL [integers]: SMTbased Bounded Model Checking and KInduction parametric problems created with the SAL model checker [salurl];

SAL [rationals]: as above, with problems on the rationals;

Symba [rationals]: bounded^{2}^{2}2 We discarded any unbounded instance in the original benchmarkset in [li_popl14]. software verification instances derived from a set of C programs used in the Software Verification Competition of 2013 [li_popl14];

Jobshop and Strip Packing [rationals]: problems taken from [DBLP:journals/cce/SawayaG05, stijcar12];

Machine Learning [rationals]: OMT instances generated with the pyLMT tool [pylmt_url] based on Machine Learning Modulo Theories [teso17].
The first benchmarkset is on the integers, whereas the other four are on the rationals. We stress the fact that all formulas contained in all benchmarksets are satisfiable.
Remark 3
Although there exists a repository of multiobjective OMT formulas (e.g. [li_popl14, st_tacas15]), we have chosen to not include these in our experimental evaluation. The reason for this is twofold. First, such comparison would likely be unfair wrt. CP tools because that the workaround for dealing with multiindependent OMT formulas described in Section §4 is not competitive with the integrated optimization schema provided by OMT solvers [li_popl14, st_tacas15]. In fact, the experimental evidence in [li_popl14, st_tacas15] collected on a group of OMT solvers indicates that the latter approach can be an order of magnitude faster than the former one. Second, the workaround for dealing with lexicographicoptimization is limited by the fact that MiniSearch is not fully compatible with recent versions of MiniZinc, and it only works with a restricted set of tools.
We have used the omt2mzn tool described in Section §4 to translate each OMT formula to the MiniZinc format. omt2mzn is written in Python and it is built on top of pySMT [PySMT], a generalpurpose Python library for solving SMT problems, and it is available at [omt2mzn]. During this step, it has been necessary to impose a finite domain to any unconstrained SMTLIB rational variable, because otherwise none of the MiniZinc solvers would have been able to deal with them. We have experimented with two different domains: the largest feasible domain for floatingpoint variables of bits (i.e. ) for the first two benchmarksets, and the largest feasible domain for integer variables (i.e. ) for the last two.
We consider two OptiMathSAT configurations: OptiMathSAT(smt), solving the original OMT formulas, and OptiMathSAT(fzn), executed on the generated MiniZinc instances. The benefits of this choice is twofold. First, we can doublecheck the correctness of such encoding, by comparing the optimum models generated in the two cases. Second, we can verify whether there is any performance loss caused by the encoding of the formula.
Only four of the MiniZinc solvers listed in Section §5 support floatingpoint reasoning. This limited the number of tools that could be used with some OMT benchmarksets. The runningtime of each MiniZinc solver reported in these experiments (including OptiMathSAT(fzn)) is comprehensive of the time taken by the mzn2fzn compiler, because the latter can sometime solve the input formulas on its own. The overall timeout is set to .
Notice that the optimal solutions found by OptiMathSAT(smt) have been previously independently verified with a thirdparty SMT tool as reported in previous publications [st_cav15, st_tacas15, st_tacas17].^{3}^{3}3 For every OMT problem s.t. OptiMathSAT(smt) returns a minimum value for on the formula , we say is correct iff is satisfiable and is unsatisfiable. (Dual for maximization.) Therefore, we verify the correctness of the results found by any other configuration by comparing them with those found by OptiMathSAT(smt), and otherwise mark the result as “unverified”.
Terminated  
Tools & Configuration  inst.  timeout  tooler.  unsupp.  incor.  correct  time(s) 
Gurobi()  66  0  0  66  0  0  0 
G12(fd)  66  0  66  0  0  0  0 
iZplus()  66  0  66  0  0  0  0 
JaCoP()  66  0  66  0  0  0  0 
Chuffed()  66  19  47  0  0  0  0 
ORTools(sat)  66  57  9  0  0  0  0 
Choco()  66  66  0  0  0  0  0 
HaifaCSP()  66  66  0  0  0  0  0 
Picat(cp)  66  66  0  0  0  0  0 
Gecode()  66  66  0  0  0  0  0 
Gurobi(l)  66  63  0  0  0  3  166 
Picat(sat)  66  62  0  0  0  4  1667 
Virtual Best(MiniZinc)  66  62  0  0  0  4  718 
OptiMathSAT(fzn)  66  18  0  0  0  48  7113 
Virtual Best(fzn)  66  18  0  0  0  48  7113 
OptiMathSAT(smt)  66  22  0  0  0  44  2657 
Virtual Best(all)  66  16  0  0  0  50  5037 
Experimental Results over the Integers.
In this experiment, we evaluate the SAL (over integers) benchmarkset. The results are collected in Table 2.
We notice first that OptiMathSAT(fzn) always produces correct results and it shows comparable performances in terms on number of problems solved wrt. the baseline OptiMathSAT(smt), solving even 4 problems more. (We conjecture that the latter fact should be attributed to the limited, but effective, deduction capabilities of the mzn2fzn compiler, that may have helped OptiMathSAT in solving the input formulas.) This suggests that, at least on problems on the integers, omt2mzn is efficient and effective and does not affect correctness.
In general, MiniZinc solvers do not seem to deal efficiently with this benchmarkset. Some tools have experienced some internal error (e.g. dumpedcore, segmentation fault), some others have been killed to to a high memory consumption (over GB), whereas the majority of the remaining tools had a timeout.
We explain this behavior with the fact that the given benchmark set is characterized by the presence of a heavy Boolean structure combined with arithmetical constraints, which requires the efficient combination of strong Booleanreasoning capabilities (e.g., efficiently handling chains of unit propagations) with strong arithmeticalsolving&optimization capabilities, which is a typical feature of OMT solvers.
None of the input formulas was initially supported by Gurobi. After restricting the bound of every integer variable to , Gurobi(l) was able to solve instances within the timeout. Among the MiniZinc solvers, the best result is obtained by Picat(sat), that solved problems out of .
Terminated  Incorrect Results  
Tools & Configuration 
Instances 
Timeout 
Toolerrors 
Incorrect 
Verified 
Unverified 
Time(s) 
unsat 





SAL over rationals  
Gurobi()  66  40  0  26  0  0  0  15  11  11  11  0  0 
Gecode()  66  66  0  0  0  0  0  0  0  0  0  0  0 
G12(mip)  66  0  0  54  0  12  305  54  0  0  0  0  0 
Virtual Best(MiniZinc)  66  0  0  54  0  12  305             
OptiMathSAT(fzn)  66  16  0  0  47  3  6952  0  0  0  0  0  0 
Virtual Best(fzn)  66  0  0  7  47  12  6096             
OptiMathSAT(smt)  66  12  0  0  54  0  4907  0           
Virtual Best(all)  66  0  0  0  54  12  5212             
Software Verification (Symba)  
G12(mip)  2632  0  0  2632  0  0  0  2632  0  0  0  0  0 
Gurobi()  2632  0  0  2596  36  0  103  2595  1  1  1  1  0 
Gecode()  2632  2477  0  0  155  0  10800  0  0  0  0  0  0 
Virtual Best(MiniZinc)  2632  0  0  2463  169  0  10402             
OptiMathSAT(fzn)  2632  0  0  0  2632  0  1366  0  0  0  0  0  0 
Virtual Best(fzn)  2632  0  0  0  2632  0  1366             
OptiMathSAT(smt)  2632  0  0  0  2632  0  284  0           
Virtual Best(all)  2632  0  0  0  2632  0  283             
JobShop + Strippacking  
Gecode()  190  190  0  0  0  0  0  0  0  0  0  0  0 
G12(mip)  190  10  0  169  0  11  12  79  90  90  86  39  0 
Gurobi()  190  8  0  106  68  8  3858  74  32  20  20  0  0 
Virtual Best(MiniZinc)  190  0  0  111  68  11  3093             
OptiMathSAT(fzn)  190  15  0  0  175  0  14002  0  0  0  0  0  0 
OptiMathSAT(smt)  190  11  0  0  179  0  10484  0           
Virtual Best(fzn)  190  0  0  4  175  11  13211             
Virtual Best(all)  190  0  0  0  179  11  9687             
Machine Learning  
G12(mip)  510  108  0  400  2  0  224  400  0  0  0  0  0 
Gurobi()  510  9  0  476  24  1  135  468  8  8  6  4  0 
Gecode()  510  322  0  164  24  0  10  147  17  2  0  0  0 
Virtual Best(MiniZinc)  510  9  0  468  32  1  317             
OptiMathSAT(fzn)  510  298  0  177  35  0  32  177  0  0  0  0  0 
OptiMathSAT(smt)  510  10  0  0  500  0  7707  0           
Virtual Best(fzn)  510  9  0  428  72  1  359             
OptiMathSAT(fzn+e)  510  442  0  0  68  0  80  0  0  0  0  0  0 
Virtual Best(all)  510  9  0  0  500  1  7737             
Experimental Results Over the Rationals.
We consider first the first three benchmarksets over the rationals: SAL over rationals, Symba, JobShop&StripPacking. Of all MiniZinc solvers we have tried, only three are able to deal with floatingpoint constraints. The results are shown in Table 3. Since each of the input formulas is satisfiable, we consider a result incorrect either when it is equal to unsat, or when the relative error exceeds , s.t.: and being the optimal value found by OptiMathSAT(smt) and the optimal value found by the MiniZinc solver under test respectively. (Recall that the former was previously checked to be correct.)
Similarly to the previous experiment on the integers, OptiMathSAT(fzn) always produces correct results, and display comparable performance wrt. OptiMathSAT(omt) in terms of number of instances being solved, solving somewhat fewer problems. This is not the case of the other three MiniZinc solvers. Among these, Gecode experienced a timeout on the majority of the formulas being considered, G12(mip) returned mostly incorrect answers, whereas Gurobi seems to have the best performance, in particular on the third benchmarkset.
We attribute the large number of incorrect results returned by all three MiniZinc solvers to the fact that these tools use finiteprecision floatingpoint arithmetic internally. The incorrect behavior of some of these solvers (e.g. Gurobi) can also be partially explained with the large domain of floatingpoint variables in these problems. However, given the nature of these input instances, it was not possible for us to assign a smaller domain to each variable in the problem a priori.
We analyze separately the results for the last benchmarkset reported in Table 3. The peculiar aspect of the Machine Learning benchmarkset [teso17] is that it is characterized by PseudoBoolean sums over rational weights, and by very finegrained rational values^{4}^{4}4For example, , which expands to , is a sample weight value from problems in [teso17].. Unfortunately, these finegrained rational values are rounded by the standard mzn2fzn compiler, which causes the incorrect results even of OptiMathSAT(fzn) in Table 3, despite the fact that OptiMathSAT uses infiniteprecision arithmetic.
In order to overcome this issue, we leverage the emzn2fzn compiler described in Section §3 to preserve the original fractional values, and show that with this approach OptiMathSAT does not produce incorrect results any longer (configuration OptiMathSAT(fzn+e) in Table 3), solving correctly 33 problems more than OptiMathSAT(fzn).
Despite this improvement, however, OptiMathSAT(fzn+e) still solves much less problems than OptiMathSAT(smt). We have analyzed this fact, and we noticed that the performance gap is due to the structural changes introduced by the mzn2fzn compiler, which prevents OptiMathSAT(fzn+E) to use some efficient solving techniques when dealing with the same problem given to OptiMathSAT(smt). ^{5}^{5}5For example, an equality like where are float values, in an unbounded float variable and are float variables, is rewritten by mzn2fzn into the equality , where the PseudoBoolean sum is added to an arithmetic term, preventing OptiMathSAT(fzn+E) from recognizing it as the definition of a PseudoBoolean sum, and hence from activating the specialized solving technique for PseudoBoolean equalities which are instead used by OptiMathSAT(smt).
Overall, since there are at least formulas affected by the above issue with the mzn2fzn compiler, we avoid an indepth discussion of the results obtained by the other MiniZinc solvers. However, at a first glance the situation does not seem to differ from the other benchmarksets over the rationals.
5.3 Discussion
On the whole, from our experiments, OMT tools appear to be still at some disadvantage when dealing with MiniZinc problems wrt. specific tools, and vice versa.
On the one hand, OMT solvers seem to be penalized by their lack of efficient ad hoc decision procedures for dealing with global constraints. Moreover, the approach taken by the mzn2fzn compiler, that creates lots of alias Boolean, integer and floatingpoint variables for dealing with PseudoBoolean constraints, is particularly challenging to deal with efficiently by an OMT solver.
On the other hand, MiniZinc solvers seem to suffer with problems needing an arithmeticreasoning component combined with heavy Booleanreasoning component. Even more importantly, the lack of infiniteprecision linear arithmetic procedures causes a number of incorrect results when dealing with OMT problems over the rationals. Both of these points need to be addressed in order to deal with the vast number of Formal Verification and Model Checking applications in the SMT/OMT domain.
6 Conclusions & Future Work
In this paper we have taken a first step forward towards bridging the MiniZinc and the OMT communities. The ultimate goal is to obtain a correct, effective and efficient fullyautomated system for translating problems from one community to the other, so as to extend the application domain of both communities. With our experimental evaluation, we have identified some criticalities that need to be addressed by each community in order to solidify this union.
We plan to push this investigation forward as follows. In the short term, we plan to address the inefficient handling of PseudoBoolean constraints over the rationals revealed by the experimental evaluation in Section §5.2. In order to deal with those FlatZinc constraints that require nonlinear arithmetic, we envisage an opportunity to either extend OptiMathSAT with proper handling of the nonlinear arithmetic theory [ac_tocl18] or to experiment with an encoding based on the floatingpoint theory [ts_cade19]. This objective goes hand in hand with the extension of omt2mzn to deal with other SMT theories. In the long term, OMT solving may also benefit from adopting efficient ad hoc decision procedures for frequently used global constraints. Finally, we plan to broaden the scope of our investigation and include other OMT solvers in our study.
Comments
There are no comments yet.