1 Introduction
Multiple Criteria Decision Analysis (MCDA) aims at supporting a decision maker (DM) in making decisions among options described according various points of view, formally represented by monotone functions called criteria. In this paper, decisions are modeled as an ordinal sorting problem, where alternatives are to be assigned to a class in the set of predefined ordered classes.
The literature contains several multiple criteria sorting methods which can be distinguished into (i) value based sorting methods (see e.g. [10, 6, 7, 16])), (ii) outranking based sorting methods (see e.g. [2, 3, 9, 13, 17, 11]) and, (iii) rule based sorting methods (see e.g. [18, 19]).
We address the problem of ordinal sorting with an outranking based sorting model: the non compensatory model (NCS, cf.[2, 3]), in which an object need not be at least as good as the profile’s value on all criteria. NCS assigns an alternative to a category above a profile if it is at least as good as the profile on a sufficient coalition of criteria; the family of sufficient coalitions can be any upset of the set of all subsets of criteria. A particular case of NCS occurs when the family of sufficient coalitions of criteria can be defined using additive criteria weights and threshold. The literature refers to this additive case as the MRSort model (see e.g. [9, 13]). Both MRSort and NCS models are particular cases of the Electre Tri model, a method for sorting alternatives in ordered categories based on an outranking relation (see [12], pp. 389401 or [4], pp. 381385).
Our aim is to learn a NCS model from preference information given in the form of a reference assignment. Such approach makes it possible to integrate the decision maker preferences in the model without asking her for the preference parameter values. Such indirect elicitation has been developped for Electre Tri [20], MRsort [9], UTADIS [21].
Algorithm 1 describes a general framework that has been widely used (see e.g. [23, 9]) in order to leverage the power of generic mathematical programming solvers to learn the parameters of a multicriteria sorting procedure from examples. The work flow is divided into three phases: the problem is encoded into a formulation, this formulation is passed to an external solver, and a solution, if found, is then decoded into a model. The faithfulness of this approach is guaranteed if, and only if:

the encoded formulation must have a solution as soon as the assignment can be represented in the model;

the solver is complete, in the sense that it yields a solution if and only if there is at least one;

the decoded model actually represents the assignment.
To the authors’ knowledge, until now, general NCS models have been deemed too computationally difficult to address with this approach. Restrictions to MRSort have been considered, either in [9] with a mixed integer programming (MIP) formulation, but this approach turned out to be inadequate to handle large datasets, or by [13, 14] using a metaheuristic solving procedure that handles large datasets but offers no guarantee of its completeness (cf. point 2 above). The aim of this paper is to investigate an alternative venue: considering UNCS, a broad subset of NCS models, that encompasses MRSort (for precise definitions of these models, see Section 2.2), and formulating the problem of representing an assignment by a model in UNCS as a boolean satisfiability problem (SAT). We prove that both the encoding and the decoding satisfy the faithfulness requirements 1 and 3 above. We are thus able to leverage the advances made in the field of generic SAT solvers, to reach unprecedented computational performance in the learning of non compensatory sorting models.
The paper is organized as follows. In Section 2, we present the notions and concepts related to the formulation of the problem of learning parameters of a non compensatory sorting model. In Section 3, we develop our binary satisfaction (SAT) problem formulation for inferring a UNCS model from a learning set, and show it has the desired properties of necessity and sufficiency regarding the representation of an assignment in the UNCS model. In Section 4, we recall the bases of using a mixed integer formulation to learn the parameters of a MRSort model. After that, we propose experiments to assess the pertinence and interest of this formulation in Section 5. In Section 6, we discuss the obtained results. Finally, in Section 7, we conclude by pointing some future interesting perspectives .
2 Position of the problem
In this section, we detail the notions permitting to formulate the problem of learning the parameters of a non compensatory sorting models. We define the vocabulary of ordinal sorting and we formalize the notion of ordinal sorting procedure. We are then able to precisely describe the problem of representing a given assignment in a given ordinal sorting model. We proceed by describing the broad class of non compensatory sorting models, and two narrower subclasses of particular interest, namely UNCS and MRSort. In Section 2.1, we formulate the NCS model and define its parameters. In Section 2.3, we specify the expected inputs and outputs of the learning problem.
2.1 Vocabulary of multicriteria ordinal sorting
An ordinal sorting problem consists in assigning a category, taken among a given, finite set of categories ordered by desirability, to alternatives described by several attributes.
We assume is a finite set of criteria, where each criterion maps alternatives to values among an ordered set , the order relation meaning “weakly worse that”^{1}^{1}1 This setting differs from the one described by [2, 3], in the sense that we suppose the attributes describing the alternatives are already sorted by the criteria according to their desirability: here, the order relation on each set needs not be constructed from holistic preference statements, but is assumed to be established beforehand, e.g. in a previous phase of a decision aiding process structured according to [4] (this is often the case in applications).. Alternatives are thus described by a tuple of multiple criteria values called profiles. We denote the set of all possible profiles– either describing actual alternatives or virtual ones.
As an analogy with a voting system where criteria would act as voters, subsets of are called coalitions of criteria. The following function maps a pair of profiles to the coalition of criteria weakly favorable to the former.
When , the alternative is at least as good as the alternative with respect to all criteria, and we say weakly dominates in the sense of Pareto. Weak dominance defines a partial order on the set of profiles .
In the remainder of this article, we assume the sets of criteria , of profiles and of categories are given, and we endeavor to define a sorting procedure, a nondecreasing mapping , ordered by dominance to .
2.2 Noncompensatory sorting models
In [2, 3], Bouyssou and Marchant define a set of sorting procedures deemed as noncompensatory. We briefly recall the definition of the non compenastory sorting (NCS) model, as well as two specific subsets of this model, UNCS and MRSort.
All these classes of noncompensatory sorting models, rely on the notions of satisfactory values of the criteria and sufficient coalitions of criteria that combines into defining the fitness of an alternative: an alternative is deemed fit if it has satisfactory values on a sufficient coalition of criteria.
This notion is straightforward to implement when there are only two categories: the sufficient coalitions form an upset^{2}^{2}2An upset is an upward closed subset of an ordered set, i.e. if is greater than and belongs to an upset, then so does . of the power set of and, for each criterion , the satisfactory values form an upset that can be described by its lower bound – meaning a value is satisfactory if, and only if, it is greater or equal to the threshold , thus defining a limiting profile . With more than two categories, the notions of sufficient coalitions and satisfactory values are declined per category – denoted respectively and . The ordering of the categories translates into a nesting of the sufficient coalitions:
(1a)  
and also a nesting of the satisfactory values:  
(1b)  
Condition (1b) translates into an ordering of the values for a given criterion , or an ordering of the limiting profiles:  
(1c) 
For convenience, these sequences are augmented with trivial elements on both ends: , , , .
Definition 1 (Noncompensatory sorting NCS, [3])
The set of preference parameters – all the pairs satisfying (1a) and (1c) – can be considered too wide and too unwieldy for practical use in the context of a decision aiding process. Therefore, following [3], one may consider to restrict either the sequence of limiting profiles, or the sequence of sufficient coalitions. In order to remain compatible with Electre Tri, we elect the latter.
Definition 2 (Noncompensatory sorting with a unique set of sufficient coalitions UNCS)
Given a set of criteria and an ordered set of categories , for all pairs where the tuple satisfies (1c) and is an upset of coalitions, the sorting function UNCS maps a profile to the category such that and .
A further restriction of UNCS is of particular interest: in the MRSort model, introduced in [9], the sufficient coalitions are represented in a compact form which is more amenable to linear programming. As an analogy to a voting setting, each criterion may be assigned a voting power so that a given coalition of criteria is deemed sufficient if, and only if, its combined voting power is greater than a given qualification threshold .
Definition 3 (majority rule sorting MRSort)
Given a set of criteria , the majority rule MR maps a pair , where is a tuple of nonnegative real numbers and a nonnegative real number, to an upset MR of the power set of defined by the relation:
(MR) 
Given, in addition, a set of categories , for all triples where the tuple satisfies (1c), is a tuple of nonnegative real numbers and a nonnegative real number, MRSort is the sorting function UNCS.
Example 1
Terry is a journalist and prepares a car review for a special issue. He considers a number of popular car models, and wants to sort them in order to present a sample of cars “selected for you by the redaction” to the readers.
This selection is based on 4 criteria : cost (€), acceleration (measured by the time, in seconds, to reach 100 km.h from full stop – lower is better), braking power and road holding, both measured on a qualitative scale ranging from 1 (lowest performance) to 4 (best performance). The performances of six models are described in Table 1:
model  cost  acceleration  braking  road holding 

16 973  29  2.66  2.5  
18 342  30.7  2.33  3  
15 335  30.2  2  2.5  
18 971  28  2.33  2  
17 537  28.3  2.33  2.75  
15 131  29.7  1.66  1.75 
In order to assign these models to a class among (average) (good) (excellent), Terry considers a NCS model:

where the values on each criterion are sorted between (average) and (good) by the following profiles: , , , . The boundary between and (excellent) is fixed by the profiles: , , , .
model cost acceleration braking road holding
Table 2: Categorization of performances

These appreciations are then aggregated by the following rule: an alternative is categorized good or excellent if it is good or excellent on cost or acceleration, and good or excellent on braking or road holding. It is categorized excellent if it is excellent on cost or acceleration, and excellent on braking or road holding. Being excellent on some criterion does not really help to be considered good overall, as expected from a non compensatory model. Sufficient coalitions are represented on Figure 2.
Finally, the model yields the following assignments (Table 3):
model  assignment 

2.3 The disaggregation paradigm: learning preference parameters from assignment examples
For a given decision situation, assuming the NCS model is relevant to structure the DM’s preferences, what parameters should be selected to fully specify the NCS model that corresponds to the DM viewpoint? An option would be to simply ask the decision maker to describe, to her best knowledge, the limit profiles between class and to enumerate the minimal sufficient coalitions. In order to get this information as quickly and reliably as possible, an analyst could make good use of the modelbased elicitation strategy described in [1], as it permits to obtain these parameters by asking the decision maker to only provide holistic preference judgment – should some (fictitious) alternative be assigned to some category – and builds the shortest questionnaire.
We opt for a more indirect setup, close to a machine learning paradigm, where a set of reference assignment is given and assumed to describe the decision maker’s point of view, and the aim is to
extend these assignments with a NCS model. In this context, we usually refer to an assignment as a function mapping a subset of reference alternatives to the ordered set of classes . These reference alternatives highlight values of interest on each criterion . We are looking for suitable preference parameters specifying a non compensatory sorting model, i.e. a tuple of profiles satisfying (1c) and an upset of coalitions (respectively, nonnegative voting parameters of a majority rule) so that UNCS (respectively, MRSort) maps all reference alternatives to its assigned class .Throughout this paper, we assume this expression of preference is free of noise. We are only interested in determining if the given assignment can be represented in the non compensatory sorting model.
3 SAT formulation for learning Ncs
In this section, we begin by giving a brief reminder of some key concepts regarding boolean satisfiability problems (SAT). Then, we proceed by describing the pivotal contribution of this work: the encoding of the problem of representing a given assignment in the UNCS model as a SAT problem. We conclude this section by providing the decoding procedure that prove this SAT formulation is equivalent to the original problem, and can be used in the context of Algorithm 1 together with a SAT solver.
3.1 Boolean satisfiability (SAT)
A boolean satisfaction problem consists in a set of boolean variables and a logical proposition about these variables . A solution is an assignment of the variables mapped to 1 by the proposition: . A binary satisfaction problem for which there exists at least one solution is satisfiable, else it is unsatisfiable. Without loss of generality, the proposition can be assumed to be written in conjunctive normal form: , where each clause is itself a disjunction in the variables or their negation , so that a solution satisfies at least one condition (either positive or negative) of every clause.
The models presented hereafter make extensive use of clauses where there is only one nonnegated variable (a subset of Horn clauses): , which represent the logical implication .
It is known since Cook’s theorem [34] that the Boolean satisfiability problem is NPcomplete. Consequently, unless , we should not expect to solve generic SAT instances quicker than exponential time in the worst case. Nevertheless, efficient and scalable algorithms for SAT have been – and still are – developed, and are sometimes able to handle problem instances involving tens of thousands of variables and millions of clauses in a few seconds (see e.g. [24, 22]).
3.2 A SAT encoding of a given assignment in UNCS
Definition 4 (SAT encoding for UNCS)
Let an assignment. We define the boolean function with variables:

, indexed by a criterion , a frontier between classes , and a value taken on criterion by a reference alternative,

indexed by a coalition of criteria
as the conjunction of clauses:

[label=(3):]

For all criteria , for all frontiers between adjacent classes
, for all ordered pairs of values
:(2a) 
For all criteria , for all ordered pairs of frontiers , for all values :
(2b) 
For all ordered pairs of coalitions :
(2c) 
For all coalitions , for all frontiers , for all (i.e. reference alternatives just below the frontier) :
(2d) 
For all coalitions , for all frontiers , for all (i.e. reference alternatives just above the frontier) :
(2e)
Clauses of types (2a), (2b) and (2c) are easily interpreted as enforcers of some monotonicity conditions inherent to ordinal sorting and to the parameters of the UNCS model:

[label=(3):]

Ascending scales – if and is above some threshold , then so is . It is necessary and sufficient to consider the clauses where and are consecutive values of .

Hierarchy of profiles – if and is above the threshold , then it is also above . It is necessary and sufficient to consider the clauses where .

Coalitions strength – if a coalition is sufficient, then any coalition containing is also sufficient. It is necessary and sufficient to consider the clauses where the coalition contains exactly one more criterion than , corresponding to the edges represented on Fig. 1.
Clauses of types (2d) and (2e) ensure the correct representation of all reference alternatives contained by the assignment in the UNCS model. They rely on the following lemmas.
Lemma 1
Let an assignment extended by a UNCS model with profiles and sufficient coalitions . If is a coalition of criteria such that, there is an alternative stronger than the upper frontier of its class on every criterion in , then this coalition is not sufficient.
Proof
Let an assignment, correct UNCS parameters , a coalition of criteria and an alternative that satisfy the premises, and suppose is sufficient. The alternative would be better than the boundary and so would be assigned to a class strictly better than , and the NCS model with parameters and would not extend the assignment.
Clauses of type (2d) leverage Lemma 1 to ensure alternatives are outranked by the boundary above them, relating insufficient coalitions to the strong points of an alternative.
Lemma 2
Let an assignment extended by a UNCS model with profiles and sufficient coalitions . If is a coalition of criteria such that, there is an alternative weaker than the lower frontier of its class on every criterion in , then the complementary coalition is sufficient.
Proof
Let an assignment, correct UNCS parameters, a coalition of criteria and an alternative that satisfy the premises, and suppose is insufficient. The set of criteria on which the alternative would be better than the boundary is a subset of , and would thus be considered insufficient. Hence, would be assigned to a class strictly worse than , and the NCS model with parameters and would not extend the assignment.
Clauses of type (2e) leverage Lemma 2 to ensure alternatives outrank the boundary below them, relating the weak points of an alternative to a complementary insufficient coalition.
We are now able to describe the decoding function required by Algorithm 1 and prove the faithfulness of both the encoding and the decoding.
3.3 Faithfulness of the SAT representation
Theorem 3.1 (from a UNCS model representing an assignment to a solution of the SAT formulation)
Given an assignment , if the tuple of profiles satisfies (1c), the set is an upset of coalitions of criteria, and the sorting function UNCS extends , then the binary tuple:

, indexed by a criterion , a frontier between classes , and a value taken on criterion by some reference alternative, and defined by

indexed by a coalition of criteria and defined by
is mapped to 1 by the Boolean function .
Proof
The clauses (2a) are satisfied because if and is above some threshold , then so is . The clauses (2b) are satisfied because the frontier profiles are assumed to satisfy (1c) (hence, if a given value is above some threshold , then it is also above inferior thresholds for ). The clauses (2c) are satisfied because is assumed to be an upset (hence, if a coalition is deemed sufficient, then so are wider coalitions). If the NCS model with profiles and sufficient coalitions extends the given assignments, then clauses (2d) are satisfied – else, by Lemma 1, one of the alternative assigned to the class would outrank the profile on a sufficient coalition of criteria  and so are clauses (2e)  else, by Lemma 2, one alternative assigned to class would not outrank the profile , as the set of criteria on which is better than would be smaller than some insufficient coalition.
Corollary 1 (Faithful encoding)
Let be an assignment . If is unsatisfiable, then cannot be represented in the model UNCS.
Theorem 3.2 (Decoding a solution of the SAT formulation into a UNCS model)
Given an assignment , if the binary tuple:

, indexed by a criterion , a frontier between classes , and a value taken on criterion by a reference alternative,

indexed by a coalition of criteria
satisfies , then the profiles defined by satisfy (1c), the set of coalitions is an upset, and the sorting function UNCS, extends the assignment .
Proof
Clauses (2a) ensure that , so that . Clauses (2b) ensure the tuple of profiles satisfies (1c). Clauses (2c) ensure the set is an upset of coalitions. The sorting function UNCS extends the given assignment because, for each reference alternative , there is a clause (2e) that ensures outranks the lower frontier of its class (if ), and a clause (2d) that ensures does not outrank the upper frontier of its class (if ).
Corollary 2 (Faithfullness of the SAT representation)
The assignment can be represented in the model UNCS if, and only if, is satisfiable.
4 Learning MRSort using Mixed Integer Programming
Learning the parameters of an MRSort model using mixed integer programming has been studied in [9]. We recall here the method used in [9] in order to obtain the mixed inter program (MIP) formulation that infers an MRSort model on the basis of examples of assignments.
With MRSort, the condition for an alternative to be assigned to a category (Equation (MR)) reads:
The linearization of these constraints induces the use of binary variables. For each variable
, with , we introduce a binary variable that is equal to when the performance of is at least as good as or better than the performance of on the criterion and 0 otherwise. For an alternative assigned to a category with , it introduces binary variables. For alternatives assigned to one of the extreme categories, the number of binary variables is divided by two. The value of each variable is obtained thanks to the following constraints:(3a)  
in which is an arbitrary large positive constant. The value of are finally obtained thanks to the following constraints:  
(3b) 
The dominance structure on the set of profiles is ensured by the following constraints:
(3c) 
As the equation (MR) defining the majority rule is homogenous, the coefficients and can be multiplied by any positive constant without modifying the upset of coalitions they represent. Thus, the following normalization constraint can be added without loss of generality:
(3d) 
To obtain a MIP formulation, the next step consists to define an objective function. In [9], two objective functions are considered, one of which consists in maximizing the robustness of the assignments. It is done by adding continuous variables for each alternative such that:
(3e) 
The objective function consists in optimizing a slack variable that is constrained by the values of the variables and as follows:
(3f) 
The combination of the objective function and all the constraints listed above leads to MIPs that can be found in [9].
Definition 5 (MIPO formulation for MR Sort)
Theorem 4.1 (Faithfulness of the MIPO formulation [9])
An assignment can be represented in the model MRSort if, and only if, is feasible. If the tuple , , , , , , , is a feasible solution of , then the tuple of profiles , the tuple of voting powers and the majority threshold are suitable parameters of a MRSort model that extends the assignment .
We are looking to compare this stateoftheart formulation to the boolean satisfiability formulation we propose in the next section in terms of computational efficiency, and in terms of quality of the result. Yet, we suspect the two approaches differ in too many aspects to be meaningfully compared. The MIPO formulation is based on a numerical representation of the problem, considers the set of every MRSort model extending the assignment, and selects the best according to the objective function – here, returning the model that gives the sharpest difference in voting weights between sufficient and insufficient coalitions of criteria. Meanwhile, the SAT formulation is based on a logical representation of the problem, considers the wider set of every UNCS model extending the assignment, and randomly yields a suitable model. In order to be able to credit the effects we would observe to the correct causes, we introduce a third formulation, called MIPD, that helps bridging the gap between MIPO and SAT. MIPD is formally a mixed integer program with a null objective function. This trick enables us to use the optimization shell of the MIP formulations to express a decision problem assessing the satisfiability of the constraints, and yielding a random solution (which, in our context, represents a particular MRSort model), rather than looking for the best one in the sense of the objective function. Another instance of this configuration, where an optimization problem is compared to its feasibility version, can be found in [35]. Here, it should be noted that the MIPD formulation is not exactly the feasibility version of MIPO, as insufficient coalitions of criteria are characterized by a strict comparison. The optimization version circumvents this obstacle by maximizing the contrast in normalized voting power between sufficient and insufficient coalitions, while the feasibility version addresses it by leaving the total weight unconstrained, but requiring the minimal difference between sufficient and insufficient coalitions is at least one vote. This slight difference might account for some divergence of behavior we observe during our experiment (see Section 5, and particularly 5.3).
Definition 6 (MIPD formulation for MR Sort)
Theorem 4.2 (Faithfulness of the MIPD formulation)
An assignment can be represented in the model MRSort if, and only if, is feasible. If the tuple , , , , , , is a feasible solution of , then the tuple of profiles , the tuple of voting powers and the majority threshold are suitable parameters of a MRSort model that extends the assignment .
Proof
This theorem results from Theorem 4.1, with only minor changes to the constraints. As noted previously, the normalization constraint (3d) has no effect on the feasibility of the problem. Instead, constraints (3g) ensure we are looking for voting parameters large enough to have at least a difference of one unit between the votes gathered by any sufficient coalition on the one hand and any insufficient coalition on the other hand.
5 Implementation
In this section, we study the performance of the formulation proposed in section 3, both intrinsic and comparative with respect to stateoftheart techniques. We implement Algorithm 1, using a stateoftheart SAT solver, in order to solve instances of the problem of learning a UNCS model, given the assignment of a set of reference alternatives. We also implement two formulations relying on Mixed Integer Programming, presented in Section 4, using an adequate solver. We begin by describing our experimental protocol, with some implementation details. Then, we provide the results of the experimental study concerning computation time of our program, and particularly the influence of learning set size, the number of criteria, and the number of classes, as well as elements of comparison between the three approaches.
5.1 Experimental protocol and implementation details
The algorithm we test takes as an input the assignment of a set of alternatives , each described by a performance tuple on a set of criteria , to a set of classes .
The performance of the solvers needs to be measured in practice, by solving actual instances of the problem and reporting the computation time required. This experimental study is run on an ordinary laptop with Windows 7 (64 bit) equipped with an Intel Core i74600 CPU at 2.1 GHz and 8 GB of RAM.
Dataset generation.
In the scope of this paper, we only consider to use a carefully crafted, random dataset as an input. On the one hand, the algorithm we describe is not yet equipped with the capability to deal with noisy inputs, so we do not consider feeding it with actual preference data, such as the one found in preference learning benchmarks [33]. On the other hand, using totally random, unstructured, inputs makes no sense in the context of algorithmic decision. In order to ensure the preference data we are using makes sense, we use a decision model to generate it, and, in particular, a model compatible with the non compensatory stance we are postulating. Precisely, we use a MRSort model for generating the learning set, a model that particularize NCS and UNCS by postulating the set of sufficient coalitions possess an additive structure (see Section 2.2). This choice ensures the three formulations we are using should succeed in finding the parameters of a model extending the reference assignment.
When generating a dataset, we consider the number of criteria , the number of classes , and the number of reference alternatives as parameters. We consider all criteria take continuous value in the interval , which is computationally more demanding for our algorithm than the case where one criterion has a finite set of values. We generate a set of ascending profiles by uniformly sampling numbers in the interval and sorting them in ascending order, for all criteria. We generate voting weights by sampling numbers in the interval , sorting them, and using them as the cumulative sum of weights.
is then randomly chosen with uniform probability in the interval
. Finally, we sample uniformly tuples in , defining the performance table of the reference alternative, and assign them to classes in according to the model MRSort with the generated profiles, voting weights, and qualified majority threshold.Solving the SAT problem.
We then proceed accordingly to Algorithm 1, translating the assignment into a binary satisfaction problem, described by sets of variables and clauses, as described by Definition 4. This binary satisfaction problem is written in a file, in DIMACS format^{3}^{3}3http://www.satcompetition.org/2009/formatbenchmarks2009.html, and passed to a command line SAT solver  CryptoMiniSat 5.0.1 [25], winner of the incremental track at SAT Competition 2016 ^{4}^{4}4 http://baldur.iti.kit.edu/satcompetition2016/, released under the MIT license. If the solver finds a solution, then it is converted into parameters for a UNCS model, as described by Theorem 3.2. The model UNCS yielded by the program is then validated against the input. As the ground truth used to seed the assignment is, by construction, a MRSort model and therefore a UNCS model, Theorem 3.1 applies and we expect the solver to always find a solution. Moreover, as Theorem 3.2 applies to the solution yielded, we expect the UNCS model returned by the program should always succeed at extending the assignment provided.
Solving the MIP problems.
We transcribe the problem consisting of finding a MRSort model extending the assignment with parameters providing a good contrast into a mixed integer linear optimization problem described extensively in Section 4 that we refer to as MIPO, where O stands for optimization. In order to bridge the gap between this optimization stance and the boolean satisfiability approach that is only preoccupied with returning any model that extends the given assignment, we also transcribe the problem consisting of finding some MRSort model extending the assignment into a MIP feasibility problem (optimizing the null function over an adequate set of constraints), also described in Section 4 that we refer to as MIPD, where D stands for decision. These MIP problems are solved with Gurobi 7.02, with factory parameters except for the cap placed on the number of CPU cores devoted to the computation (two), in order to match a similar limitation with the chosen version of the SAT solver. When the solver succeeds in finding a solution before the time limit – set to one hour – the sorting functions returned are called and , respectively.
Evaluating the ability of the inferred models to restore the original one.
In order to appreciate how “close” a computed model ,, is to the ground truth from which the assignment example were generated , we proceed as follows: we sample a large set of profiles in and compute the assignment of these profiles according to the original and computed MR Sort models ( and ). On this basis, we compute the proportion of “errors”, i.e. tuples which are not assigned to the same category by both models. To obtain a reasonable sample for , we vary size of the sample of according to the number of criteria :
5.2 Intrinsic performance of the SAT formulation
We run the experimental protocol described above by varying the various values of the parameters governing the input. In order to assess the intrinsic performance of Algorithm 1 we consider all the combinations where

the number of criteria is chosen among ;

the number of reference alternatives is chosen among 16, 32, 64, 128, 256, 512.

the number of categories is chosen among
For each value of the triplet of parameters, we sample 100 MRSort models , and record the computation time () needed to provide a model
Figure 3 displays the time needed by Algorithm 1 to compute , versus the number of reference alternatives , both represented in logarithmic scale, in various configurations of the number of criteria. The fact that each configuration is seemingly represented by a straight line hints at a linear dependency between and . The fact that the various straight lines, corresponding to various number of criteria, seem parallel, with a slope close to 1, is compatible to a law where is proportional to . The same observations in the plane (number of criteria computation time) (not represented) leads to infer a law
where the computing time is proportional to the number of reference alternatives and to the number of coalitions (corresponding to the number of ary clauses of the SAT formulation). Finally, as a rule of thumb: the average computation time is about 10 s for 11 criteria, 3 categories and 100 reference alternatives; it doubles for each additional criterion, or when the number of reference alternatives doubles.
5.3 Comparison between the formulations
In order to compare between models, we focus on a situation with three categories, nine criteria, and 64 reference alternatives, serving as a baseline. We then consider situations deviating from the baseline on a single parameter – either the number of categories , from 2 to 5, or the number of criteria, among , or the number of reference alternative among . For each considered value of the triple of parameters, we sample 50 MR Sort models representing the ground truth , and we record the computation time needed to provide each of the three models , and , as well as the generalization indexes for the three models. The MIP are solved with a timeout of one hour.
Results on the computation time.
For the three formulations under scrutiny and the set of considered parameters governing the input, the computation time ranges from below the tenth of a second to an hour (when the timeout is reached), thus covering about five orders of magnitude. The left side of Figure 4 depicts the distribution of the computation time for the baseline situation (9 criteria, 3 categories, 64 reference assignments). While the computing time for the SAT and the MIPD formulations seem to be centered around similar values (with and for the baseline), the distribution of the computing time for the SAT algorithm around this center is very tight, while the spread of this distribution for the MIPD formulation is comparatively huge: the slowest tenth of instances run about a thousand time slower than the quickest tenth. The computation time of the MIPO formulation appears about 50 times slower than the SAT, with a central value of , and covers about two orders of magnitude.
In order to assess the influence of the parameters governing the size and complexity of the input, we explore situations differing from the baseline on a single parameter.

The number of reference assignments . Figure 5 indicates that the distribution of the computing time for SATbased algorithm remains tightly grouped around its central value, and that this value steadily increases with the number of reference assignments. Meanwhile, the two MIP formulations display a similar behavior, with an increase of the central tendency steeper than the one displayed by the SAT, and a spread that widens when taking into account additional reference assignments.

the number of criteria . Figure 6 indicates that the distribution of the computing time for SATbased algorithm remains tightly grouped around its central value, and that this value steadily increases with the number of criteria. This increase is steeper in the case of the SAT and MIPO formulations than for the MIPD formulation.

the number of categories . Figure 7 displays an interesting phenomenon. The number of categories seems to have a mild influence on the computation time, without any restriction for the SATbased algorithm, and as soon as there are three categories or more for the MIPbased algorithm, with a clear exception in the case of two categories, which yields instances of the problem solved ten times faster than with three or more categories.
Results on the ability of the inferred model to restore the original one.
The right side of Figure 4 depicts the distribution of the proportion of correct assignments (as compared to the ground truth) for the baseline situation (9 criteria, 3 categories, 64 reference assignments). The situation depicted is conveniently described by using the distribution of outcomes yielded by the MIPD formulation as a pivotal point to which we compare those yielded by the SAT and MIPO formulations: the central 80% of the distribution (between the whiskers) of outcomes for the MIPO corresponds to the central half (the box) for the MIPD, while the best half of the distribution of outcomes for the SAT corresponds to the central 80% for the MIPD. In other terms, compared to the MIPD, the MIPO offers consistently good results, while the SAT has a 50% chance to yield a model that does not align very well with the ground truth.
Figures 8, 9 and 10 depict the variations of the alignment of the models yielded by the three algorithms with the ground truth with respect to the number of reference assignments, of criteria, or of categories, respectively. The experimental results display a tendency towards a degradation of this alignment as the number of criteria or the number of categories increase. Conversely, as expected, increasing the number of reference assignments noticeably enhances the restoration rate. The three algorithms seem to behave in a similar manner with respect to the modification of these parameters.
Reliability.
The three formulations expressing the problem we solve – finding a non compensatory sorting model extending a given assignment of reference alternatives – into technical terms are theoretically faithful. Moreover, as we generate the input assignment with a hidden ground truth which itself obeys a noncompensatory sorting model, the search we set out to perform should provably succeed. Unfortunately, a computer program is but a pale reflection of an algorithm, as it is restricted in using finite resource. While we take great care in designing the experimental protocol in order to avoid memory problems, we have purposefully used offtheshelf software with default setting to solve the formulations. While this attitude has given excellent result for the implementation of the SATbased algorithm, which has never failed to retrieve a model that succeeds in extending the given assignment, the two MIPbased implementations have suffered from a variety of failures, either not terminating before the timeout set at one hour or wrongly concluding on the infeasibility of the MIP. We report these abnormal behaviors in Table 4.
number of criteria  5  7  9  11  13  7  9  9 
number of categories  3  3  3  3  3  3  5  7 
number of reference assignments  64  64  64  64  64  128  64  64 
MIPD  4%  8%  4%  0  0  42%  10%  12% 
MIPO  0  0  0  10%  48%  4%  0  0 
SAT  0  0  0  0  0  0  0  0 
6 Discussion and perspectives
In this section, we strive at interpreting the results presented in Section 5. In Section 6.1, we address the influence of the parameters governing the size and structure of the input  the reference assignment we set out to extend with a noncompensatory sorting model  on the computing time of the programs implementing the three formulations modeling the problem. In Section 6.2, we discuss the differing approaches to knowledge representation underlying these different formulations, and their practical consequences.
6.1 Influence of the parameters
The influence of the various parameters (, the number of reference assignments; , the number of criteria; , the number of categories) governing the input on the ability of the output model to predict the ground truth seeding the input is best understood from a machine learning perspective. The input assignments form the learning set of the algorithm, while the number of criteria and the number of categories govern the number of parameters describing the noncompensatory sorting model. Hence, an increase in adds constraints upon the system, while increases in or relieve some constraints, but demand more resources for their management.

The comparison between MIPO and MIPD informs the influence of the loss function. This influence is threefold: optimizing this function demands a lot more time than simply returning the first admissible solution found; formalizing the problem of extending the input assignment with a model as an optimization problem incorporates a kind of robustness into the algorithm, which translates to a decrease in the number of failures; paradoxically, the strategy consisting in finding the most representative model (in the sense of the chosen loss function) does not yield models with a better alignment to the ground truth than the one consisting to return a random suitable model.

The MIPD and SAT formulations implement the same binary attitude concerning the suitability of a noncompensatory model to extend a given assignment, and both arbitrarily yield the firstencountered suitable model. Nevertheless, algorithms based on these formulations display marked differences in behavior: while the running time of the SATbased algorithm is very homogeneous between instances and follows very regular patterns when the input parameters change, the MIPD algorithm behaves a lot more erratically, with some failures (displayed in table 4) and a tremendous spread. We credit this difference in behavior to a difference of approach to knowledge representation, as discussed in section 6.2. Also, with the same input parameters, the model returned by the MIPD algorithm seems on average to be more faithful to the ground truth than the model returned by the SAT algorithm. As both models return random suitable models in different categories (MR Sort for the MIP, and the superset NCS for the SAT, while the ground truth is chosen in the MR Sort category), we interpret the difference in the proportion of correct assignment to the respective volumes of the two categories of model, and discuss the pros and cons of assuming one or the other in section 6.2.

Reference assignments are a necessary evil. On the one hand, they provide the information needed to entrench the model, and refine the precision up to which its parameters can be known. On the other hand, they erect a computational barrier which adds up more quickly for the MIP formulations we are considering than for the SAT one, as shown in Figure 5 . Overcoming this barrier demands time and threatens the integrity of the somewhat brittle numerical representation underlying the MIPD formulation.

From the perspective of the modelfitting algorithm, the number of criteria and the number of categories are usually exogenous parameters, fixed according to the needs of the decision situation. The specific numbers of criteria we considered during the experiment, from 5 to 13, cover most of the typical decision situations considered in MCDA. Introducing more criteria demands to assess more parameters, which has a compound effect on complexity, as it requires at the same time to build a higher dimension representation of the models, and to provide more reference examples in order to be determined with a precision suitable to decision making. Apart from a noticeable exception (see below), the number of categories does not seem to have much influence (as shown on figures 7 and 10).

Underconstrained models are not very good at providing recommendations. When fed with scarce information, the task of finding a suitable extension is easy, but there are very little guarantees this extension matches the unexpressed knowledge and preferences of the decision maker concerning alternatives outside the learning set. We interpret the decrease in the ability to align with the ground truth as the number of criteria increases displayed on Figure 9 as an expression of an overfitting phenomenon, where too many parameters are chosen to faithfully represent a too little slice of the set of alternatives, but poorly represent cases never seen before.

Mixed integer programs can represent decision problems, in theory. Practically though, some complex inputs have proven overwhelming for the MIPD formulation, whereas the MIPO has shown more robustness, as evidenced by Table 4. It seems fair to assume this lack of stability is related to the absence of a normalization constraint such as (3d) in the MIPD formulation .

MR Sort with two categories is structurally different than models with more than two categories. While we have defined it as a procedure where alternatives are compared holistically to a profile, it can also be described as an additive value sorting model with stepwise, nondecreasing, 2valued marginals. The experimental results, both for the computing time and the alignment with the ground truth (see figures 7 and 10
, where the points corresponding to two categories are outliers with respect to the rest of the series) highlight this peculiarity, and tend to show that the valuebased representation of the MRSort model with two categories is computationally efficient. Determining a good lower bound on the difference of normalized voting power between sufficient and insufficient coalitions would therefore likely help alleviating this issue.
6.2 Numeric or symbolic representation of coalitions
Our proposal to infer non compensatory sorting models from assignment examples using a SAT formulation relies on a symbolic representation of sufficient coalitions of criteria. It departs fundamentally from the stateoftheart approach of representing the upset of sufficient coalitions with a numeric majority rule (MR).
Obviously, UNCS is more general than MRSort as additive weights/majority level induce a set of minimal criteria coalitions, while a set of minimal coalitions might not be additive. [5]
studies the proportion of additive representations: all NCS models are additive up to 3 criteria, but the proportion of additive NCS models tends to be quickly marginal when the number of criteria increases. It is also possible possible to extend the MRSort model up to UNCS by considering a capacity instead of a weight vector (see e.g.
[14]). This leads to MIP formulations more and more awkward as the arity of the capacity increases, increasing the number of decision variables and the computation time. Also, [14] shows that a MRSort model learned from NCS generated examples provides a good approximation of this NCS model.A distinctive feature of MRSort is its parsimony with respect to interaction between criteria, a notion that the SAT formulation of UNCS fails to capture. However, there are many ways to additively represent a set of minimal coalitions, and the intuitive interpretation of the weights can therefore be misleading: there is no one to one correspondence between the tuples of voting powers and majority level, and the sets of additive coalitions of criteria. For instance, consider a three criteria problem in which minimal coalitions are all combination of two criteria out of three. This set of minimal coalitions can be represented by and , or and . It is obvious that these two numerical representations yield erroneously to two very distinct interpretations about the relative importance of criteria. In this sense, the symbolic representation avoiding weights used in the SAT formulation is more faithful than a numerical representation. As a consequence, this nonuniqueness of the additive representation penalizes the effectiveness of loss functions involved in MIPs.
Also, as mentioned in Section 5.3, the feasibility version of the MIP suffers from numerical instability, perhaps because of the lack of a normalization constraint. The symbolic representation of sufficient coalitions circumvents the difficult mathematical question of providing a good lower bound on the worst case difference of normalized voting power between sufficient and insufficient coalitions.
7 Conclusion
In this paper, we consider the multiple criteria noncompensatory sorting model [2, 3] and propose a new SAT formulation for inferring this sorting model from a learning set provided by a DM. Learning this model has already been addressed by the literature, and solved by the resolution of a MIP [9] or via a specific heuristic [13, 14]. Due to high computation time, the MIP formulation can only apply to learning sets of limited size. Heuristic methods can handle large datasets, but can not ensure to find a compatible model with the learning set whenever it exists. Our new algorithm provides such guaranty. We implemented and tested our SAT formulation, and it outperforms MIP approaches in terms of computation time (reduction of the computation time by a factor of about 50).
Moreover, it could have been the case that this good performance in terms of computing time would be counterbalanced by a limited ability of the inferred model in terms of generalization. Indeed, a MIP approach focuses the effort in finding a relevant representative model among the compatible models (through the use of an objective function), while our SAT approach does return the first compatible model found.
Our experiments show that MIP and SAT approaches have similar performances in terms of generalization. Therefore, we believe this algorithm to be a strong advance in terms of learning NCS models based on learning sets, in particular when learning sets become relatively large.
Thanks to its efficiency – finding a model compatible to some preference information takes seconds instead of minutes – this algorithm is well suited to be embedded in an interactive process, where the decision maker is invited to interactively elicit a non compensatory sorting model by incrementally building a learning set (and possibly additional preference information). In order to address realworld decision aiding situations, the algorithm we propose needs to be equipped with techniques permitting to account for noisy or inconsistent data. While the numeric formulations may rely on Lagrangian techniques to handle the requirement of correctly representing the data as a set of soft constraints rather than hard ones, the logic formulation we propose could usefully investigate the notions of maximally consistent or minimally inconsistent set of clauses (see e.g. [27] for solving techniques, or e.g. [28] for an application in a MCDA context). The increased speed, as compared to the previous MIPbased approach, opens the door to the exploration of the set of all UNCS models extending a given assignment, in the vein of the version space theory [31] and robust decision aiding [32]. The knowledge representation underlying our approach may also permit to support a recommendation with an explanation [26, 29, 30].
References
 [1] K. Belahcène, V. Mousseau, M. Pirlot and O. Sobrie. Preference Elicitation and Learning in a Multiple Criteria Decision Aid perspective. LGI Research report 201702, CentraleSupélec, http://www.lgi.ecp.fr/Biblio/PDF/CRLGI201702.pdf, 2017.
 [2] D. Bouyssou, and T. Marchant. An axiomatic approach to noncompensatory sorting methods in MCDM, I: The case of two categories. European Journal of Operational Research, 178(1):217–245, 2007.
 [3] D. Bouyssou, and T. Marchant. An axiomatic approach to noncompensatory sorting methods in MCDM, II: More than two categories. European Journal of Operational Research, 178(1):246–276, 2007.
 [4] D. Bouyssou, T. Marchant, M. Pirlot, A. Tsoukiàs, and Ph. Vincke. Evaluation and decision models with multiple criteria: Stepping stones for the analyst. International Series in Operations Research and Management Science, Volume 86. Springer, Boston, 1st edition, 2006.
 [5] E. Ersek Uyanik, O. Sobrie, V. Mousseau, and M. Pirlot. Enumerating and categorizing positive Boolean functions separable by a kadditive capacity. Discrete Applied Mathematics, 229, 1730, 2017.
 [6] S. Greco, V. Mousseau, and R. Slowinski. Multiple criteria sorting with a set of additive value functions. European Journal of Operational Research, 207(3), 14551470, 2010.
 [7] S. Greco, M. Kadzinski, and R. Slowinski. Selection of a representative value function in robust multiple criteria sorting. Computers & Operations Research, 38(11), 16201637, 2011.
 [8] Ő. Karsu, Approaches for inequityaverse sorting, Computers & Operations Research, 66, 6780, 2016.

[9]
A. Leroy, V. Mousseau, and M. Pirlot. Learning the parameters of a multiple criteria sorting method. In R. Brafman, F. Roberts, and A. Tsoukiás, editors, Algorithmic Decision Theory, volume 6992 of Lecture Notes in Artificial Intelligence, pages 219–233. Springer, 2011.
 [10] J.L. Marichal, P. Meyer, and M. Roubens. Sorting multiattribute alternatives: The TOMASO method. Computers & Operations Research, 32(4), 861877, 2005.
 [11] P. Meyer, A.L. Olteanu. Integrating large positive and negative performance differences into multicriteria majorityrule sorting models. Computers & Operations Research, 81, 216–230, 2017.
 [12] B. Roy, and D. Bouyssou. Aide multicritère à la décision: méthodes et cas. Economica, Paris, 1993.
 [13] O. Sobrie, V. Mousseau, and M. Pirlot. Learning a majority rule model from large sets of assignment examples. In P. Perny, M. Pirlot, and A. Tsoukiàs, editors, Algorithmic Decision Theory, pages 336–350, Brussels, Belgium, 2013. Springer.
 [14] O. Sobrie, V. Mousseau, and M. Pirlot. Learning the Parameters of a Non Compensatory Sorting Model. In Algorithmic Decision Theory, ADT 2015, pages 153–170, 2015.
 [15] O. Sobrie. Learning preferences with multiplecriteria models. PhD thesis, CentraleSupélec and UMONS, 2017.
 [16] B. Soylu. A multicriteria sorting procedure with Tchebycheff utility function. Computers & Operations Research, 38(8), 10911102, 2011.
 [17] J. Zheng, S.A. Metchebon TakougangV. Mousseau, and M. Pirlot. Learning criteria weights of an optimistic Electre Tri sorting rule. Computers & Operations Research, 49, 2840, 2014.
 [18] S. Greco, B. Matarazzo and R. Slowinski. Rough sets methodology for sorting problems in presence of multiple attributes and criteria. European Journal of Operational Research; 138(2), 247  259, 2002
 [19] S. Greco, B. Matarazzo and R. Slowinski. Decision rule approach. in Multiple Criteria Decision Analysis  State of the art surveys, S. Greco, M. Ehrgott and J. Figueira (eds.) second edition, 497552, 2016
 [20] V. Mousseau and R. Slowinski. Inferring an ELECTRE TRI model from assignment examples, Journal of Global Optimization, 12(2), 157174, 1998
 [21] C. Zopounidis and M. Doumpos. Multicriteria classification and sorting methods: A literature review, European Journal of Operational Research; 138(2), 229246, 2002.
 [22] A. Biere, M. Heule, H. van Maaren, and T. Walsh. Handbook of Satisfiability. Frontiers in Artificial Intelligence and Applications 185, IOS Press, 2009.
 [23] E. JacquetLagreze and Y. Siskos. Assessing a set of additive utility functions for multicriteria decisionmaking, the UTA method. European Journal of Operational Research, 10(2), 151164, 1982.
 [24] M. Moskewicz, C. Madigan, Y. Zhao, L. Zhang, and S. Malik. Chaff: engineering an efficient SAT solver. In Proceedings of the 38th annual Design Automation Conference (DAC ’01). ACM, New York, NY, USA, 530535. 2001.
 [25] M. Soos, The CryptoMiniSat 5 set of solvers at SAT Competition 2016. In Proceedings of SAT Competition 2016: Solver and Benchmark Descriptions, volume B20161 of Department of Computer Science Series of Publications B, University of Helsinki, 2016.
 [26] L. Amgoud and M. Serrurier. Agents that argue and explain classifications. In Autonomous Agents and MultiAgent Systems, AAMAS 2008.
 [27] Ph. Besnard, E. Grégoire and J.M. Lagniez. On Computing Maximal Subsets of Clauses that Must Be Satisfiable with Possibly MutuallyContradictory Assumptive Contexts. Proceedings of the TwentyNinth AAAI Conference on Artificial Intelligence, January 2530, 2015, Austin, Texas, USA.
 [28] V. Mousseau, J. Rui Figueira,L. C. Dias, C. Gomes da Silva and J. C. N. Clímaco. Resolving inconsistencies among constraints on the parameters of an MCDA model. European Journal of Operational Research 147(1), 72–93, 2003.
 [29] Ch. Labreuche. A general framework for explaining the results of a multiattribute preference model. Artificial Intelligence 175(78), 1410–1448, 2011.
 [30] K. Belahcene, Ch. Labreuche, N. Maudet, V. Mousseau and W. Ouerdane. A Model for Accountable Ordinal Sorting. Proceedings of the TwentySixth International Joint Conference on Artificial Intelligence, IJCAI 2017, 814–820.
 [31] T. Mitchell. Generalization as Search. Artificial Intelligence 18(2) 203–226, 1982.
 [32] S. Greco, V. Mousseau and Roman Slowinski. Ordinal regression revisited: Multiple criteria ranking using a set of additive value functions. European Journal of Operational Research 191(2), 416–436, 2008.
 [33] J. Furnkranz, E. Hullermeier. Preference Learning. SpringerVerlag New York, Inc, 2011.

[34]
S. Cook. The complexity of theorem proving procedures. Proceedings of the third annual ACM symposium on Theory of computing, 151158, 1971.
 [35] J. Dickerson, J. Goldman, J. Karp, A. Procaccia, T. Sandholm. The Computational Rise and Fall of Fairness. AAAI 2014, 14051411
Comments
There are no comments yet.