Because the hardest instances for a given solver are not necessarily the hardest ones for another solver, it is difficult to evaluate and to compare the performances of sat solvers. In the competitions like , the solvers are evaluated with several kinds of instances (random, hand crafted, industrial…), and the results show that no solver outperforms the other ones in all the categories.
The underlying question is about the robustness of sat solvers. How to identify the hardest instances for a given solver ? We propose to use local search algorithms to find hard instances.
In this section, we report the results of a first prospective experimentation using the sat4j framework . The sat solver provided by the java library Sat4j can either be used as a stand alone solver or integrated into a java application. We used this second way to develop a prototype allowing the evolution of a sat instance by local search.
A sat instance is a cnf propositional formula, i.e., a conjunction of clauses, where each clause is a disjunction of literals. Each literal is either a propositional variable or a negated propositional variable. Our initial formula contains randomly generated clauses of literals. The variables are uniformly drawn in a set of variables under the constraint that no variable can occur more than one times in any clause.
Each variable is negated with probability
variables under the constraint that no variable can occur more than one times in any clause. Each variable is negated with probability.
We propose two ways to evaluate the hardness of an instance, which are the number of propagations and the number of decisions required to solve it. A propagation consists to fix a variable thanks to the unit resolution rule, which is the basic filtering method used in sat solvers. A decision is a binary node of the search tree.
The used local search process aims to increase the number of propagations the solver needs to solve the current formula. At each generation, the current instance is modified by replacing an existing randomly drawn clause by a new randomly generated clause. Then the solver is ran and the new number of propagations is compared to . If then the old current formula is restored, else the the new one is kept.
Figure 1 show the evolution of the number of propagations and the number of decisions during generations. The required number of propagations increases two orders of magnitude, while the required number of decisions increases one order of magnitude. The instance remains satisfiable throughout the evolution.
3 Evolutions strategies
The local search algorithm used in section 2 is based on a rudimentary greedy search strategy with a transition operator that replaces a randomly drawn clause by a new randomly generated one. Two ways can be explored toward more sophiticated evolution schemes.
3.1 Search strategies
The aim of a local search strategy is essentially to avoid the stance of evolution, which may be due to local extremums. We propose three research directions.
The algorithm presented in section 2 could be improved by modifying several clauses simultaneously when the score (i.e., the number of propagations, the number of decisions, or any other relevant indicator) does not increase for some time. Such a "break" is expected to allow the search process to escape from the current basin of attraction (if applicable), with the hope of finding a more promising evolution path.
Another idea is to try guiding the search by focusing it on some variables : A weight is assigned to each variable. When changing a clause improve the current score, the weights of the related variables increases. The clauses containing variables with highest weight are preferentially modified. On the contrary, when changing a clause does not improve the current score, the weights of the corresponding variables decreases.
Population based search techniques, like genetic algorithms, could be used. This suppose to find relevant crossover operators, in the sense that the offspring of two (or more) formulae should be likely to be as hard as its parents. An idea worth exploring is first to aggregate two formulae, then reduce the size of the resulting offspring tanks to a local search process designed to remove clauses. A more standard crossover operator is used in with convincing results.
Moreover, it is also possible to start the search with a formula already known to be hard with respect to its size.
3.2 Transition operators
It is known that the choice of the transition (or neighborhood) operator can dramatically impact the efficiency of a local search process. In section 1, this operator consists to replace a clause by a new one. The number of clauses remains constant during the search. Other ways can be considered, like the following ones.
Instead of changing a whole clause, we can try to change one literal at each generation.
Sometimes, a new clause can be added, or an existing one can be removed.
The current instance can be maintained satisfiable or unsatisfiable by rejecting the candidates that do not verify the criterion.
4 A few experiments
In this section, we present the first results obtained with a local search algorithm designed to evolve an unsatisfiable formula in two stages. The initial formula is randomly generated with a ratio number of clauses / number of variables which is substantially above the satisfiability threshold , in such a way that the obtained sat instance is most likely unsatisfiable. In the first stage, the transition operator consists in removing a clause and the selection criteria accepts the new formula if its remains unsatisfiable. The duration of this stage is 10 times the number of clauses. In the second stage, the transition operator consists in replacing a randomly chosen clause by a new randomly generated one. The selection criterion accepts the new formula if the number of decisions required to solve it does not decrease.
Figure 2 shows the evolution of both the size of the -cnf formula of 50 variables and the number of decision required to solve it on generations. The second stage begins after about 1000 generations.
Figure 3 shows the evolution of a -cnf formula of 100 variables in the same conditions. The number of decisions increases from 122 to 1572 during the first stage, where the number of clauses decreases from 600 to 280. In the second stage, the number of clauses remains constant while the numbers of decisions increase up to 2180. We examined whether the hardness of this instance also increases with respect to another sat solver, i.e., cryptominisat  : the initial formula requires 91 decisions, and the final one 2051 decisions.
For a first glimpse of how the initial instance impacts the result of the evolution, Figure 4 shows the initial and final number of decisions for twenty formula of 50 variables : there is no apparent correlation between the initial and final hardness, which is consistently greatly increased.
5 Related works
We presented the idea of using local search for finding hard sat instances in , but this work, although promising, has not been resumed since.
To the best of our knowledge, the only other work really close to our approach is , which use genetic algorithms to evolve combinatorial problem instances in order that there become difficult to solve. This already fairly advanced work proposes an analysis of the generated instances for identifying what make them hard to solve.
6 Concluding remarks
We proposed to use local search algorithms to produce sat instances which are harder to solve than randomly generated k-cnf formulae.
The specificity of the proposed approach is that the computation of the objective function is very expensive, because the aim is to produce hard instances, with the result to increase the cost of evaluate the hardness of these instances.
Nevertheless, the first results obtained with rudimentary search algorithms show that the approach deserves further study. It could be used as a test of robustness for sat solvers, and could help to investigate how branching heuristics, learning strategies, and other criteria, impact this robustness.
-  Olivier Bailleux and Jacquline Chabrier. Local search against local search. ECAI’96 Workshop on Advances in Propositional Deduction, August 1996.
-  Daniel Le Berre and Anne Parrain. The sat4j library, release 2.2. JSAT, 7:59–64, 2010.
-  Olivier Dubois. Upper bounds on the satisfiability threshold. Theoretical Computer Science, 265(1-2):187 – 197, 2001.
-  M. Soos, K. Nohl, and C. Castelluccia. Extending sat solvers to cryptographic problems. In proc. of SAT 09, LNCS 5584, pages 244–257. Springer, 2009.
-  Ewald Speckenmeyer, Chu Min Li, Vasco Manquinho, and Armando Tacchella. JSAT Special Issue on the 2007 Competitions. IOS Press, 2007.
-  Jano I. van Hemert. Evolving combinatorial problem instances that are difficult to solve. Evolutionary Computation, 14(4), 2006.