A Comparative Study of Ranking-based Semantics for Abstract Argumentation

02/02/2016 ∙ by Elise Bonzon, et al. ∙ Laboratoire d'Informatique de Paris 6 0

Argumentation is a process of evaluating and comparing a set of arguments. A way to compare them consists in using a ranking-based semantics which rank-order arguments from the most to the least acceptable ones. Recently, a number of such semantics have been proposed independently, often associated with some desirable properties. However, there is no comparative study which takes a broader perspective. This is what we propose in this work. We provide a general comparison of all these semantics with respect to the proposed properties. That allows to underline the differences of behavior between the existing semantics.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

Introduction

Argumentation consists in reasoning with conflicting information based on the exchange and evaluation of interacting arguments. The most popular way to represent argumentation process was proposed by Dung dung95 with argumentation frameworks modelized by binary graphs, where the nodes represent the arguments, and the edges represent the attacks between them. From these argumentation frameworks, several semantics indicating which sets of arguments, called extensions, are mutually compatible were proposed (see [Baroni, Caminada, and Giacomin2011] for an overview). However, for applications with a big number of arguments, it can be problematic to have only two levels of evaluations (arguments are either accepted or rejected). For instance, such a limitation can be questionable when using argumentation for debate platforms on the web (see [Leite and Martins2011] for such a discussion).

In order to fix these problems, a solution consists in using semantics that distinguish arguments not with the classical accepted/rejected evaluations, but with a large number of levels of acceptability. A lot of these semantics, called ranking-based semantics, were proposed in recent years [Amgoud and Ben-Naim2013, Cayrol and Lagasquie-Schiex2005, Leite and Martins2011, Matt and Toni2008, Gabbay2012] with, for each semantics, different behaviour and logical properties. However, all these semantics have never been compared between them.

This is what we propose in this work. We study the existing ranking-based semantics in the literature (focusing on the semantics that return a unique ranking between arguments) in the light of the proposed properties. That allows us to underline the differences of behavior between those semantics, and to propose a better reading of the different choices one has on this matter.

The paper is organized as follows. The following section gives the relevant background regarding abstract argumentation and ranking-based semantics. The next section presents the different properties that have been introduced in the literature, whereas the next one formally introduces the existing ranking-based semantics. Note that due to space constraints, we can not recall all the details and justifications of semantics and properties of the literature, but the reader can find them in the corresponding papers. In the next section we discuss the different properties and compare the semantics, and the final section concludes.

Preliminaries

In this section, we start by briefly recalling what is a Dung’s abstract argumentation framework [Dung1995].

Definition 1.

An argumentation framework (AF) is a pair with a set of arguments and a binary relation on , i.e. , called the attack relation. A set of arguments attacks an argument , if there exists , such that . We note .

Let be the set of all argumentation frameworks. For two AFs and , we define the union .

We can now introduce some useful notions in order to formalize properties of argumentation frameworks.

Definition 2.

Let be an AF and . A path from to , noted , is a sequence of arguments such that , and . We denote by the length of P. A defender (resp. attacker) of

is an argument situated at the beginning of an even-length (resp. odd-length) path. We denote the multiset of defenders and attackers of

by and respectively. The direct attackers of are arguments in . An argument is defended if .
A
defense root (resp. attack root) is a non-attacked defender (resp. attacker). We denote the multiset of defense roots and attack roots of by and respectively. A path from to is a defense branch (resp. attack branch) if is a defense (resp. attack) root of . Let us note and .

The connected components of an AF are the set of largest subgraphs of AF, denoted by , where two arguments are in the same component of AF if and only if there is some path (ignoring the direction of the edges) between them.

In Dung’s framework [Dung1995], the acceptability of an argument depends on its membership to some sets, called extensions. Another way to select a set of acceptable arguments is to rank arguments from the most to the least acceptable ones. Ranking-based semantics aim at determining such a ranking between arguments.

Definition 3.

A ranking-based semantics associates to any argumentation framework AF = a ranking on A, where is a preorder (a reflexive and transitive relation) on . means that is at least as acceptable as ( is a shortcut for and , and is a shortcut for and ).

When there is no ambiguity about the argumentation framework in question, we will use instead of .

Finally, we need to introduce the notion of lexicographical order in order to define some ranking-based semantics.

Definition 4.

A lexicographical order

between two vectors of real number

and , is defined as iff s.t. and .

Properties

Let us recall the logical properties proposed in the literature for ranking-based semantics. Please note that all the properties are not mandatory (we will see later that some of them are incompatible), but we want to give all of them for completeness and since we will check them for the existing ranking-based semantics. Unless stated explicitly, all the properties are defined for a ranking-based semantics , and .

Definition 5.

An isomorphism between two argumentation frameworks AF = and AF’ = is a bijective function such that , iff . With a slight abuse of notation, we will note .

Abstraction. The ranking on should be defined only on the basis of the attacks between arguments.
(Abs) Let . For any isomorphism s.t. , we have

Independence. The ranking between two arguments and should be independent of any argument that is neither connected to nor to .
(In) , ,
                          

We may have expectations regarding the best and worst arguments that we may find in an AF:

Void Precedence. A non-attacked argument is ranked strictly higher than any attacked argument.
(VP) and

Self-Contradiction. A self-attacking argument is ranked lower than any non self-attacking argument.
(SC) and

The following local properties are concerned with the direct attackers, or defenders, of arguments:

Cardinality Precedence. The greater the number of direct attackers for an argument, the weaker the level of acceptability of this argument.
(CP)

Quality Precedence. The greater the acceptability of one direct attacker for an argument, the weaker the level of acceptability of this argument.
(QP) s.t.

Before defining the next properties, we need to introduce a relation that compares sets of arguments on the basis of their rankings [Amgoud and Ben-Naim2013]:

Definition 6.

Let be a ranking on a set of arguments A. For any , is a group comparison iff there exists an injective mapping f from to such that . And is a strict group comparison iff and or .

Counter-Transitivity. If the direct attackers of are at least as numerous and acceptable as those of , then is at least as acceptable as .
(CT)

Strict Counter-Transitivity. If CT is satisfied and either the direct attackers of are strictly more numerous or acceptable than those of , then is strictly more acceptable than .
(SCT)

Defense Precedence. For two arguments with the same number of direct attackers, a defended argument is ranked higher than a non-defended argument.
(DP) and

Definition 7.

Let and . The defense of is simple iff every defender of attacks exactly one direct attacker of . The defense of is distributed iff every direct attacker of is attacked by at most one argument.

Distributed-Defense Precedence. The best defense is when each defender attacks a distinct attacker.
(DDP) and , if the defense of is simple and distributed and the defense of is simple but not distributed, then

The following properties check if some change in an AF can improve or degrade the ranking of one argument. These properties have been proposed informally by Cayrol and Lagasquie-Schiex CLS05, in the context of their semantics. We propose a formalization that generalize them for any argumentation frameworks. We first define the addition of a defense/attack branch to an argument.

Definition 8.

Let , . The defense branch added to is , with , , , , and . The attack branch added to , denoted is defined similarly except that the sequence is of odd length (i.e. ).

The following properties are defined such that exists an isomorphism with , and . We use as a clone of .

Strict addition of Defense Branch. Adding a defense branch to any argument improves its ranking.
(DB) If , then

Addition of Defense Branch. It could make sense to treat differently non-attacked arguments. So in [Cayrol and Lagasquie-Schiex2005], this property is defined in a more specific way: adding a defense branch to any attacked argument improves its ranking.
(+DB) If and , then

Increase of Attack branch. Increasing the length of an attack branch of an argument improves its ranking.
(AB) If , and , then

Addition of Attack Branch. Adding an attack branch to any argument degrades its ranking.
(+AB) If , then

Increase of Defense branch. Increasing the length of a defense branch of an argument degrades its ranking.
(DB) If , and , then

One can find the properties Abs, In, VP, DP, CT, SCT, CP, QP and DDP in [Amgoud and Ben-Naim2013], the properties In, VP and SC in [Matt and Toni2008] and the property VP in [Cayrol and Lagasquie-Schiex2005].

To this set of properties from the literature we want to add some other important properties.
Total. All pairs of arguments can be compared.
(Tot)

The next property states that all the non-attacked arguments should have the same ranking.

Non-attacked Equivalence. All the non-attacked argument have the same rank.
(NaE)

The last property describes the behavior adopted by a semantics concerning the notion of defense, and can be viewed as some kind of compatibility with usual Dung’s semantics. The idea is that a defended argument is always better than an attacked argument.
Attack vs Full Defense. An argument without any attack branch is ranked higher than an argument only attacked by one non-attacked argument.
(AvsFD) and

Let us now check the incompatibilities/dependencies between properties.

Proposition 1.

For every ranking-based semantics, the following pairs of properties are not compatible :

The following properties are not independent :

Existing Ranking-based Semantics

Categoriser

Besnard and Hunter BH01 propose a categoriser function which assigns a value to each argument, given the value of its direct attackers.

Definition 9 (Besnard and Hunter 2001).

Let be an AF. The categoriser function is defined as:

Definition 10.

The ranking-based semantics Categoriser associates to any AF = a ranking on such that iff .

Example 1.

Let with and .

d

a

b

e

c
Figure 1: An argumentation framework

The categoriser of each argument are , , , and . So we obtain the ranking : .

This semantics takes into account only the value of the direct attackers to compute the strength of an argument. This is why the argument of the example, which is attacked twice but by arguments that are attacked by a non-attacked argument, is ranked higher than the argument , which is attacked just once, but by a stronger argument.

Proposition 2.

The ranking-based semantics Categoriser satisfies111The properties Abs, In, VP, DP, CT, SCT have already been checked by Pu et al. PLZL14. Abs, In, VP, DP, CT, SCT, AB, DB, +AB, Tot and NaE. The other properties are not satisfied.

Social Abstract Argumentation Framework

Leite and Martins LM11 introduce an extension of Dung’s abstract argumentation frameworks that include social voting on the arguments: the Social Abstract Argumentation Frameworks (SAF). They also propose a family of semantics where a model is a solution to the equation system222An equational approach was also proposed by Gabbay G12. This method returns multiple solutions, and thus several rankings for one AF. This is why we do not consider this method in this paper. with one equation for each argument, based on its social support and its direct attackers. In order to compare SAFs with the existing ranking-based semantics, we chose to ignore the social support of arguments by giving them the same value.

Definition 11.

Let be an AF and be a (well-behaved) SAF semantic. The total mapping is a social model of under semantics such that :

  • is a totally ordered set with top and bottom elements, containing all possible valuations of an argument;

  • is an attenuation factor. is monotonic w.r.t. the first argument and antimonotonic w.r.t the second argument;

  • combines the initial score with the score of direct attackers. is continuous, commutative, associative, monotonic w.r.t. both arguments and is its identity element;

  • aggregates the score of direct attackers. is continuous, commutative, associative, monotonic w.r.t. both arguments and is its identity element;

  • restricts the value of the attacked argument. is antimonotonic, continuous, , and .

One possible (well-behaved) SAF semantic proposed in [Leite and Martins2011] is the simple product semantic where (with , to ensure the uniqueness of the semantics), (Product T-Norm), (Probabilistic Sum T-CoNorm) and .

Definition 12.

The ranking-based semantics SAF associates to any a ranking on such that iff .

Example 1 (cont.).

With , we obtain , , , and . We obtain the ranking: .

As for the Categoriser semantics, the strength of attackers is more important than their numbers, and thus is preferred to . However the impact of a defense branch on an argument is weaker with SAF than with Categoriser.

Proposition 3.

SAF satisfies Abs, In, VP, DP, CT, SCT, AB, DB, +AB, Tot and NaE. Other properties are not satisfied.

Discussion-based semantics

The Discussion-based semantics [Amgoud and Ben-Naim2013] compares arguments by counting the number of paths ending to them. If some arguments are equivalent (they have the same number of direct attackers), the size of paths is recursively increased until a difference is found.

Definition 13.

Let be an AF, , and .

The discussion count of is denoted .

Definition 14.

The ranking-based semantics Dbs associates to any AF = a ranking on such that , iff .

Example 1 (cont.).
step a b c d e
1 2 0 1 1 2
2 -1 0 0 -2 -3

Using the lexicographical order, one obtains the following ranking:

The number of attackers is more important than their strength, thus is here stronger than .

Proposition 4.

Dbs satisfies Abs, In, VP, DP, CT, SCT, CP, AB, DB, +AB, Tot and NaE. The other properties are not satisfied.

Burden-based semantics

The Burden-based semantics [Amgoud and Ben-Naim2013] assigns, at each step , a Burden number to every argument, that depends on the Burden numbers of its direct attackers.

Definition 15.

Let be an AF, and .

The Burden number of is denoted .

Two arguments are lexicographically compared on the basis of their Burden numbers.

Definition 16.

The ranking-based semantics Bbs associates to any a ranking on such that , iff .

Example 1 (cont.).
step a b c d e
1 3 1 2 2 3
2 2.5 1 2 1.33 1.83

Using the lexicographical order, one obtains the following ranking:

As on this example, Dbs and Bbs often return the same result. The main difference between these semantics is that Bbs satisfies DDP, so examples related to that kind of structures lead to distinct results.

Proposition 5.

Bbs satisfies Abs, In, VP, DP, CT, SCT, CP, DDP, AB, DB, +AB, Tot and NaE. The other properties are not satisfied.

Valuation with tuples

The semantics proposed by Cayrol and Lagasquie-Schiex CLS05 takes into account all the ancestors branches of an argument (defender and attacker) stored in tupled values :

Definition 17.

Let be an AF and . Let be the (ordered) tuple of even integers representing the lengths of all the defense branches of , i.e. is the smallest ordered tuple such that , where means ”appears at least times”. Similarly let be the (ordered) tuple of odd integers representing the lengths of all the attack branches of , i.e. is the smallest ordered tuple such that . A tupled value for is the pair .

When cycles exist in the AF, some tuples can be infinite. To calculate them, this method requires a highly involved process, that turn cyclical graphs into infinite acyclic graphs. We thus consider this approach for acyclic graphs only, and denote it by .

Once the tupled value of each argument has been computed, one can compare them. To do so one has to compare the length of attack/defense branches and, in case of a tie, to compare the values inside each tuples (see Algorithm 1).

Input: two tupled values of arguments and
Output: A ranking between and
1 begin
2        if v = w then  and ;
3        else
4               if  and  then
5                      if  and  then ;
6                      else
7                             if  and  then ;
8                             else  and ;
9                            
10                     
11              else
12                      if  and  then ;
13                      else
14                             if  and  then ;
15                             else  and ;
16                            
17                     
18              
19       
20
Algorithm 1

Let us remark that two arguments can be incomparable. It is the case, for example, if an argument has strictly more attack branches and more defense branches than another one. Consequently, this semantics returns a partial ranking between arguments.

As example 1 contains a cycle, we can not compute Tuples on this running example.

Proposition 6.

The ranking-based semantics Tuples satisfies Abs, In, VP, +DB, AB, DB, +AB, NaE and AvsFD. The other properties are not satisfied.

Matt & Toni

Matt and Toni MT08 compute the strength of an argument using a two-person zero-sum strategic game. This game confronts two players, a proponent and an opponent of a given argument, where the strategies of the players are sets of arguments. For an and , the sets of strategies for the proponent and opponent are and respectively.

Definition 18.

Let be an AF and . The set of attacks from to is defined by . The degree of acceptability of w.r.t is given by where .

Definition 19.

Let be an AF. The rewards of , denoted by , are defined by :

Proponent and opponent choose mixed strategies, according to some probability distributions, respectively

and , with and . For each argument , the proponent’€™s expected payoff is then given by . Finally the value of the zero-sum game for an argument , denoted by , is .

Definition 20.

The ranking-based semantics M&T associates to any AF = a ranking on such that iff .

Example 1 (cont.).

One obtains , , , and and the following preorder: .

On this example, we can see that once again the strength of attackers is more important than their numbers ( is ranked higher than ).

Proposition 7.

The ranking-based semantics M&T satisfies Abs, In, VP, +AB, SC, Tot, NaE and AvsFD. Other properties are not satisfied.

Discussion

As it can be easily checked on the running example, all these proposed ranking semantics have distinct behaviors (the ranking obtained is different for each semantics - see the summary in Table 1): this justifies the need of some axiomatic work.

Semantics Order between arguments
Cat
SAF
M&T
Dbs
Bds
Table 1: Orders obtained on the Example 1

Our work initiates this study, by checking properties that have been proposed in the papers that introduce the different semantics. Our analysis is applied to existing semantics, but any new semantics could be inspected through the same lens.333For instance, the semantics very recently proposed in [Grossi and Modgil2015]. Table 2 summarizes the properties satisfied by the ranking semantics we consider in this paper. We also checked what are the properties satisfied by the usual Dung’s Grounded semantics, that gives some hints on the compatibility of these properties with classical semantics. Note that, in this case, this is a degenerate ranking semantics with only two levels (accepted/rejected):

Proposition 8.

The grounded semantics satisfies Abs, In, CT, QP, Tot, NaE, AvsFD. Other properties are not satisfied.

A cross means that the property is not satisfied, symbol means that the property is satisfied, symbol means that the property can not be applied to the semantics (because the semantics is not compatible with the constraint given by the rule), and the shaded cells highlight the results already proved in the literature.

Properties SAF Cat Dbs Bbs M&T Grounded
Abs
In
VP
DP
CT
SCT
CP
QP
DDP
SC -
DB
+DB

AB
DB
+AB
Tot
NaE
AvsFD
Table 2: Properties satisfy by the studied ranking semantics.

There are a number of observations that we can make regarding these axioms and the results reported in Table 2:

Some axioms seem to be widely accepted and shared by all semantics. We see that the properties Abs, In and VP are satisfied by all the ranking semantics. This is expected, since these properties really seem necessary for a good ranking semantics. Indeed, we recall that the input is a Dung’s abstract argumentation framework where there is no information about the nature of arguments, so only the attacks have to be taken into account, hence the importance of Abs. Concerning the property Independence (In), no justification could explain the fact that an argument can influence others arguments without an existing link between them. Finally, the non-attacked arguments are obviously the best arguments in an AF, it is why VP is necessary.
NaE is also satisfied by all semantics. This is also a very basic requirement for a ranking semantics, it mainly says that the non-attacked arguments are all equivalent. This is a kind of compatibility principle with usual Dung’s semantics, and it says that only your attackers should impact your ranking, not the arguments you attack.444Note that it could make sense to make a distinction between arguments that attack a lot of arguments and the ones that do not — so to violate NaE, in particular. This could be considered as some kind of power index. But this is not the aim of ranking semantics. Another property that we consider as a requirement is the Tot property, which is in line with the idea of “ranking” semantics. It would be necessary if one wants to use these semantics in real applications. This is a drawback of Tuples. An interesting question is to know if it is possible to refine Tuples, i.e. to define a semantics close to Tuples, but that is easily computable for argumentation frameworks with cycles, and that satisfies Tot. A last property satisfied by all semantics is +AB, which states that adding an attack branch towards an argument degrades its ranking. This also seems to be a perfectly natural requirement for ranking semantics: the more you are attacked, the worse you are.

Overall, this gives us a set of 6 properties that should be satisfied by any ranking semantics: Abs, In, VP, NaE, Tot and +AB. One can note that Abs, In, NaE and Tot are satisfied by the grounded semantics, so they are compatible with usual Dung’s semantics. VP and +AB are not satisfied by the grounded semantics, because it only has two levels of evaluation (accepted/rejected), and these two properties really introduce graduality in the evaluation.

Some axioms are very discriminatory and provide a rough classification of semantics. As a general comment, one can check in the table that SAF, Cat, Dbs and Bds share a lot of properties. Tuples and M&T seem to belong to another class of semantics: they are the only ones that satisfy AvsFD. The property AvsFD, illustrated in Figure 2, which states that an argument that is (only) attacked once by a non-attacked argument (it is the case of only attacked by ) is worse than an argument that have any number of attacks that all belong to defense branches (it is the case of which have four defense branches and no attack branch), is a very discriminating property.

Figure 2: AF that illustrates the property AvsFD

So this property can be seen as a kind of boundary between two sub-classes of ranking properties. The ones that satisfy it take care of the whole branches of attack/defense. Whereas for the properties that do not satisfy it, a defense branch (that still ends by an attack towards the argument) always penalizes it.

More specific properties. As mentioned already, axioms operate at different levels. We observe that ‘local’ axioms (CP, QP, DP, (S)CT), just looking at direct attackers (or defenders), make choice which can be justified in some situations, but which seem hardly general (and sometimes impossible to reconcile with some more global properties, as our Prop. 1 shows). Properties related to ‘change’ (DB, +DB, AB, DB, +AB) are very appealing. One of our contribution is to have systematically generalized them.

Defining axiomatically the worst arguments is not obvious. Interestingly, while all semantics agree axiomatically on which arguments should be the best in a system (VP), there is no consensus regarding the worst arguments. SC is very interesting in that respect. It makes the observation that a self-contradicting argument is intrinsically flawed, without even requiring other arguments to defeat it. But as can be observed none of the semantics comply with it, except the one of Matt and Toni. It is because all semantics consider that an argument that attacks itself is a path like the other ones. So an argument which attacks itself (and by no other argument) is better than an argument which is attacked several times.

The interplay of axioms is often instructive. We have identified a number of incompatibilities between axioms. There is an additional remark that we can make in that respect, that is related to the incompatibility between VP and DB. One can easily remark that DB is more general than +DB, and in a sense more natural: the property is stated for any cases, it does not treat some arguments (the non-attacked arguments here) differently. But it contradicts VP in this case. +DB is a less “systematic” property (it was the original one proposed in [Cayrol and Lagasquie-Schiex2005]) but is compatible with VP : if one accepts that non-attacked arguments should be the best (VP), it cannot be the case that adding a defense branch always improve the situation of a given argument.

This set of axioms is yet to be augmented. This can be observed by the fact that SAF and Cat satisfy the same set of properties, whereas they have quite different definitions and behavior. This mean that at least one property is lacking in order to discriminate these two operators.

Conclusion

In this work we proposed a comparative study of existing ranking-based semantics. It turns out that the existing ranking-based semantics exhibit quite different behaviors and satisfy different properties. We propose to take as basic properties for ranking-based semantics Abs, In, VP, NaE, Tot and +AB. We also put forward AvsFD that discriminates two subclasses of semantics.

There is still work needed on the topic. First to propose other ranking-based semantics. But it is also important to find other logical properties, and to try to characterize classes of semantics with respect to these properties. An ambitious research agenda would be to identify situations where controversial axioms are justified or not.

Acknowledgements

This work benefited from the support of the project AMANDE ANR-13-BS02-0004 of the French National Research Agency (ANR).

References

  • [Amgoud and Ben-Naim2013] Amgoud, L., and Ben-Naim, J. 2013. Ranking-based semantics for argumentation frameworks. In Proceedings of the 7th International Conference on Scalable Uncertainty Management, (SUM’13), 134–147.
  • [Baroni, Caminada, and Giacomin2011] Baroni, P.; Caminada, M.; and Giacomin, M. 2011. An introduction to argumentation semantics.

    The Knowledge Engineering Review

    26(4):365–410.
  • [Besnard and Hunter2001] Besnard, P., and Hunter, A. 2001. A logic-based theory of deductive arguments. Artificial Intelligence 128(1-2):203–235.
  • [Cayrol and Lagasquie-Schiex2005] Cayrol, C., and Lagasquie-Schiex, M. 2005. Graduality in argumentation. Journal of Artificial Intelligence Research 23:245–297.
  • [Dung1995] Dung, P. M. 1995.

    On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games.

    Artificial Intelligence 77(2):321–358.
  • [Gabbay2012] Gabbay, D. M. 2012. Equational approach to argumentation networks. Argument & Computation 3(2-3):87–142.
  • [Grossi and Modgil2015] Grossi, D., and Modgil, S. 2015. On the graded acceptability of arguments. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, (IJCAI’15), 868–874.
  • [Leite and Martins2011] Leite, J., and Martins, J. 2011. Social abstract argumentation. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, (IJCAI’11),, 2287–2292.
  • [Matt and Toni2008] Matt, P., and Toni, F. 2008. A game-theoretic measure of argument strength for abstract argumentation. In Proceedings of the 11th European Conference on Logics in Artificial Intelligence, (JELIA’08), 285–297.
  • [Pu et al.2014] Pu, F.; Luo, J.; Zhang, Y.; and Luo, G. 2014. Argument ranking with categoriser function. In Proceedings of the 7th International Conference on Knowledge Science, Engineering and Management, (KSEM’14), 290–301.