The filtering algorithms inside constraint programming solvers ([1, 2, 3, 4] etc.) are mainly tested using test suites implemented manually. Creating such unit tests is a significant workload for the developers and is also error prone.
The most elementary yet important test to achieve for a constraint is that no feasible solution is removed. One can always implement a checker verifying the feasibility of the constraint when all the variables are bound. By comparing the number of solutions generated with both the checker and the tested filtering algorithm, one can be confident that no solution is removed. This procedure can be repeated for many (small) instances (possibly randomly generated). Alternatively, one can compare with a decomposition of the constraint into (more) elementary ones. This latter approach can improve the coverage of the test suite.
Those unit tests verifying the non removal of feasible solutions do not verify other properties of constraints generally more difficult to test. For instance, the domain-consistency property is rarely tested outside some hard-coded small test examples.
We introduce CPChecker as a tool to ease the solver developer’s life by automating the testing of properties of filtering algorithms. For instance, algorithm A should filter more than algorithm B or Algorithm A should achieve arc or bound-consistency, etc. The tool does not ensure that the tested filtering does not contain any bug - as it is impossible to test all possible input domains - but it can reveal the presence of one, if a test fails. The large variety of input domains pseudo-randomly generated should make the user confident that the tool would allow to detect most of the bugs.
Many constraint implementations are stateful and maintain some reversible data structures. Indeed, global constraints’ filtering algorithms often maintain an internal state in order to be more efficient than their decomposition.
This reversible state is also a frequent source of bugs.
CPChecker includes the trail-based operations when testing constraints such that any bug due to the state management of the constraint also has a high chance to be detected.
CPChecker is generic and can be interfaced with any JVM trailed based solvers.
CPChecker is able to generate detailed explanations by generating minimal domain examples on which the user’s filtering has failed, if any.
In [5, 6, 7], the authors introduce tools to debug models. Some researches have also been done to help programmers while debugging codes for constraint programming . To the best of our knowledge, these tools, unlike CPChecker, do not focus on the filtering properties of individual constraints.
In the next sections, we first detail how to test static filtering algorithms before explaining the testing of stateful filtering algorithms for trailed based solvers. Finally we introduce how CPChecker can be integrated into a test suite.
2 Testing Static Filtering Algorithms
CPChecker is able to test any static filtering algorithm acting over integer domains. Therefore, the user needs to implement a function taking some domains (array of set of ints) as input and returning the filtered domains111Most of the code fragments presented are in Scala for the sake of conciseness but the library is compatible with any JVM-based language.:
CPChecker also needs a trusted filtering algorithm serving as reference with the same signature. The least effort for a user is to implement a checker for the constraint under the form of a predicate that specifies the semantic of the constraint. For instance a checker for the constraint can be defined as
One can create with CPChecker an Arc/Bound-Z/Bound-D/Range Consistent filtering algorithm by providing in argument to the corresponding constructor the implementation of the checker. For instance
This class implements the filter function as a trusted filtering algorithm reaching the arc consistency by 1) computing the Cartesian product of the domains, 2) filtering with the checker the non solutions and 3) creating the filtered domains as the the union of the values. Similar filtering algorithms’ (Bound-Z, Bound-D and Range) have been implemented from a checker.
Finally the check and stronger functions permit to respectively check that two compared filtering algorithms are the same or that the tested filtering is stronger than the trusted one.
The testing involves the following steps:
Random Domains generation 222A seed can be set to reproduce the same tests..
Execution of the tested and trusted filtering algorithms (from CPChecker’s filterings or another trusted one) to these random domains.
Comparison of the domains returned by the two filtering algorithms.
This process is repeated by default 100 times although all the parameters can be overridden for the creation of random domains, number of tests, etc.
2.1 Generation of Random Test Instances
In order to test a filtering implementation, CPChecker relies on a property based testing library called ScalaCheck333Similar libraries exist for most programming languages, all inspired by QuickCheck for Haskell.. This library includes support for the creation of random generators and for launching multiple test cases given those. CPChecker also relies on the ability of ScalaCheck of reducing the instance to discover a smaller test instance over which the error occurs.
Here is an example for testing with CPChecker the arc-consistent AllDifferent constraint’s in OscaR  solver :
The trusted filtering algorithm is created thanks to the ArcFiltering class at line 3. The checker for AllDifferent simply verifies that the union of the values in the array has a cardinality equal to the size of the array, as defined at line 2. The tested filtering implements the filter function using OscaR’s filtering. It first transforms the variables into OscaR’s variables (line 7) then creates the constraint over them (line 8). It is then posted to the solver which filters the domains until fix-point before returning them.
3 Testing stateful constraints
Incremental Filtering Algorithms usually maintain some form of state in the constraints. It can for instance be reversible data-structures for trailed-based solvers. CPChecker allows to test a stateful filtering algorithm by testing it during a search while checking the state restoration. In terms of implementation, the incremental check and stronger functions compare FilterWithState objects that must implement two functions. The setup function reaches the fix-point while setting up the solver used for the search. The branchAndFilter function applies a branching operation on the current state of the solver and reaches a new fix-point for the constraint. The branching operations represent standard branching constraints such as and the push/pop operations on the trail allowing to implement the backtracking mechanism (see  for further details on this mechanism).
The process of testing an incremental/stateful filtering algorithm is divided into four consecutive steps :
Application of the setup function of the tested and trusted filtering algorithms.
Comparing the filtered domains returned at step 2.
Execution of a number of fixed number dives as explained next based on the application of branchAndFilter function.
A dive is performed by successively interleaving a push of the state and a domain restriction operation. When a leaf is reached (no or one solution remaining) the dive is finished and a random number of states are popped to start a new dive as detailed in the algorithm 1.
3.2 Illustration over an Example
The next example illustrates CPChecker to test the OscaR’s filtering for the constraint . It should reach Bound-Z consistency.
In this example, two FilterWithState are compared with the check function.
In CPChecker, the IncrementalFiltering class implements the
FilterWithState abstract class for any Filter object. Therefore, the
IncrementalFiltering created with a BoundZFiltering object is used as the trusted filtering (line 4) which it-self relies on the very simple sumChecker function provided by the user and assumed to be bug-free.
4 Custom Assertions
To ease the integration into a JUnit like test suite, CPChecker has custom assertions extending the AssertJ library. The classes FilterAssert and
FilterWithStateAssert follow the conventions of the library with the filterAs and weakerThan functions to respectively test a filtering algorithm, as in the check and stronger functions. An example of assertion is:
5 Code Source
CPChecker’s code source is publicly available in the Github repository444https://github.com/vrombouts/Generic-checker-for-CP-Solver-s-constraints. This repository also contains several examples of usage of CPChecker with both Scala solver and Java solvers, namely OscaR, Choco and Jacop. From those examples, CPChecker detected that the arc consistent filtering of the Global Cardinality constraint of OscaR was not arc consistent for all the variables (the cardinality variables). This shows the genericity of CPChecker and that it can be useful to test and debug filtering algorithms with only a small workload for the user. Further details on the architecture and implementation of CPChecker can be found in the Master Thesis document available at the github repository††footnotemark: .
6 Conclusion and Future Work
This article presented CPChecker, a tool to test filtering algorithms implemented in any JVM-based programming language based on the JVM. Filtering algorithms are tested over domains randomly generated which is efficient to find unexpected bugs. Principally written in Scala, CPChecker can be used to test simple and stateful filtering algorithms. It also contains its own assertions system to be directly integrated into test suites. As future work, we would like to integrate into CPChecker properties of scheduling filtering algorithms  such as edge-finder, not-first not-last, time-table consistency, energy filtering, etc. for testing the most recent implementation of scheduling algorithms [13, 14, 15, 16, 17, 18].
-  Charles Prud’homme, Jean-Guillaume Fages, and Xavier Lorca. Choco Documentation. TASC - LS2N CNRS UMR 6241, COSLING S.A.S., 2017.
-  OscaR Team. OscaR: Scala in OR, 2012. Available from https://bitbucket.org/oscarlib/oscar.
-  Krzysztof Kuchcinski, Radoslaw Szymanek and contributors. Jacop solver. Available from https://osolpro.atlassian.net/wiki/spaces/JACOP/.
-  Gecode Team. Gecode: Generic constraint development environment, 2006. Available from http://www.gecode.org.
-  Micha Meier. Debugging constraint programs. In International Conference on Principles and Practice of Constraint Programming, pages 204–221. Springer, 1995.
-  Peter J. Stuckey Carleton Coffrin, Siqi Liu and Guido Tack. Solution checking with minizinc. In ModeRef2017, The Sixteenth International Workshop on Constraint Modelling and Reformulation, 2017.
-  Nadjib Lazaar, Arnaud Gotlieb, and Yahia Lebbah. A cp framework for testing cp. Constraints, 17(2):123–147, 2012.
-  Frédéric Goualard and Frédéric Benhamou. Debugging Constraint Programs by Store Inspection, pages 273–297. Springer Berlin Heidelberg, Berlin, Heidelberg, 2000.
-  ScalaCheck. http://scalacheck.org.
-  Laurent Michel, Pierre Schaus, and Pascal Van Hentenryck. Minicp: A minimalist open-source solver to teach constraint programming. Technical Report, 2018.
-  AssertJ Library. http://joel-costigliola.github.io/assertj/index.html.
-  Philippe Baptiste, Claude Le Pape, and Wim Nuijten. Constraint-based scheduling: applying constraint programming to scheduling problems, volume 39. Springer Science, 201.
Steven Gay, Renaud Hartert, and Pierre Schaus.
Time-table disjunctive reasoning for the cumulative constraint.
International Conference on AI and OR Techniques in Constriant Programming for Combinatorial Optimization Problems, pages 157–172. Springer, 2015.
-  Cyrille Dejemeppe, Sascha Van Cauwelaert, and Pierre Schaus. The unary resource with transition times. In International conference on principles and practice of constraint programming, pages 89–104. Springer, 2015.
Hamed Fahimi and Claude-Guy Quimper.
Linear-time filtering algorithms for the disjunctive constraint.
Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 2637–2643. AAAI Press, 2014.
-  Petr Vilím. Global constraints in scheduling. 2007.
-  Petr Vilím. Timetable edge finding filtering algorithm for discrete cumulative resources. In International Conference on AI and OR Techniques in Constriant Programming for Combinatorial Optimization Problems, pages 230–245. Springer, 2011.
-  Alexander Tesch. A nearly exact propagation algorithm for energetic reasoning in o (n2logn). In International Conference on Principles and Practice of Constraint Programming, pages 493–519. Springer, 2016.