A new simple and effective measure for bag-of-word inter-document similarity measurement

02/09/2019 ∙ by Sunil Aryal, et al. ∙ Deakin University Monash University Federation University Australia Osaka University 0

To measure the similarity of two documents in the bag-of-words (BoW) vector representation, different term weighting schemes are used to improve the performance of cosine similarity---the most widely used inter-document similarity measure in text mining. In this paper, we identify the shortcomings of the underlying assumptions of term weighting in the inter-document similarity measurement task; and provide a more fit-to-the-purpose alternative. Based on this new assumption, we introduce a new simple but effective similarity measure which does not require explicit term weighting. The proposed measure employs a more nuanced probabilistic approach than those used in term weighting to measure the similarity of two documents w.r.t each term occurring in the two documents. Our empirical comparison with the existing similarity measures using different term weighting schemes shows that the new measure produces (i) better results in the binary BoW representation; and (ii) competitive and more consistent results in the term-frequency-based BoW representation.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Pairwise similarity measurements of documents is a fundamental task in many text mining problems such as query-by-example, document classification and clustering.

In the bag-of-words (BoW) (Salton and McGill, 1986; Manning et al., 2008) vector space model, a document is represented by an -dimensional vector where is the number of terms in a given dictionary, i.e., ; and it has the following two representations:

  1. Term-frequency-based representation: each ( is a set of non-negative integers) is the occurrence frequency of term in document .

  2. Binary representation: each where 0 represents the absence of term in document and 1 represents the presence of in .

Because the number of terms in a document is significantly less than that in the dictionary, every document is represented as a sparse BoW vector, where many entries are zero. Because of sparsity, Euclidean distance is not a good similarity measure and the angular distance, aka cosine distance, is a preferred choice of inter-document similarity measure (Salton and McGill, 1986; Salton and Buckley, 1988).

Because all terms in a document are not equally important to represent its subject, different ‘term weighting’ schemes (Manning et al., 2008; Salton and Buckley, 1988) are used to adjust vector components based on the importance of their terms.

The idea of term weighting was first introduced in the field of Information Retrieval (IR) where the task is to measure the relevance of documents in a given collection for a given query phrase consisting of a few terms. It is based on the following two assumptions (Manning et al., 2008; Salton and Buckley, 1988; Zobel and Moffat, 1998):

  1. A term is important in a document if it occurs multiple times in the document.

  2. A rare term (that occurs in a few documents in the collection) is more important than frequent terms (that occur in many documents in the collection).

The importance of terms in a document are estimated independent of the query. Because a query in the IR task is short and each term generally occurs only once, it is not an issue that the weights are determined independent of the query.

However, it can be counter-productive in the query-by-example task where the query itself is a document, and terms often occur more than once in the query document. For example, to a query document , a document having more occurrences of the terms in may not be more similar than which has exactly the same occurrences of terms in .

Prior research in the BoW inter-document similarity measurement task were focused on developing effective term weighting schemes to improve the task specific performances of existing measures such as cosine and Best Match 25 (BM25) (Salton and Buckley, 1988; Robertson et al., 1994; Joachims, 1997; Singhal, 1997; Roberston and Zaragoza, 2009; Paltoglou and Thelwall, 2010; Han et al., 2012; Wang and Zhang, 2013). In contrast, we investigate an alternative similarity measure where an adjustment of vector components using term weighting is not required.

This paper makes the following contributions:

  1. Identify the shortcomings of the underlying assumptions of term weighting schemes employed in existing measures; and provide an alternative which is more congruous with the requirements of inter-document similarity measurements.

  2. Introduce a new simple but effective inter-document similarity measure which is based on the new assumption and does not require explicit term weighting. It uses a more nuanced probabilistic approach than those used in term weighting to measure the similarity of two documents w.r.t each term occurring in the two documents under measurement.

  3. Compare the performance of the new measure with existing measures (which use different term weighting schemes) in the query-by-example task. Our result reveals that the new measure produces (i) better results than existing measures in the binary BoW representation; and (ii) competitive and more consistent results to existing measures in the term-frequency-based BoW representation.

The rest of the paper is organized as follows. Related work in the areas of term weighting and inter-document similarity measures are discussed in Section 2. Issues of term weighting in inter-document similarity measurements are discussed Section 3. The proposed new inter-document similarity measure is presented in Section 4, followed by empirical results in Section 5, related discussion in Section 6, and the last section presents the conclusions.

The key notations used in this paper are defined in Table 1.

A collection of documents (i.e., )
BoW vector of a document
The term in the dictionary
The number of documents in having
The set of terms in
The importance or weight of in
Term frequency factor of in
Inverse document frequency factor of
The similarity of two documents and
The length of document (i.e., )
The average length of documents in
is more similar to than w.r.t
is equally similar to as w.r.t
Table 1: Key notations

2 Related work

In this section, we present the pertinent details of term weighting and some widely used existing BoW inter-document similarity measures.

2.1 Term weighting

In the field of IR, there has been considerable research investigating the effective term weighting scheme. The importance of a term in document , , is estimated using different variants and combinations of two factors (Manning et al., 2008; Salton and Buckley, 1988; Joachims, 1997; Robertson et al., 1994; Singhal, 1997; Roberston and Zaragoza, 2009; Paltoglou and Thelwall, 2010; Han et al., 2012; Wang and Zhang, 2013): (i) document-based factor based on the frequency of in , ; and (ii) collection-based factor based on the number of documents where occurs, .

The most widely used term weighting scheme is tf-idf (term frequency - inverse document frequency) where (Manning et al., 2008; Salton and Buckley, 1988); and it consists of:

  1. Document-based factor: if , and 0 otherwise;

  2. Collection-based factor: .

In the IR task, the idea of tf-idf term weighting is based on the following assumptions (Zobel and Moffat, 1998):

  1. Documents with multiple occurrences of query terms are more relevant than documents with single occurrence of query terms [the tf assumption].

  2. Documents having rare query terms (occurring in a few documents in the collection) are more relevant to the query than documents having frequent query terms (occurring in many documents in the collection) [the idf assumption].

The tf factor considers the importance of in a document. Even though a document with multiple occurrences of a query term is more likely to be relevant to the given query, a document having higher occurrences of one query term is not necessarily more relevant than a document having lower occurrences of two query terms. Therefore, the logarithmic scaling of raw term frequencies is used to reduce the over influence of high frequencies of query terms (Manning et al., 2008; Salton and Buckley, 1988).

The idf factor considers the importance of in the given collection. Basically, it ranks the importance of terms in the given dictionary based on the number of documents where they occur. Terms occurring only in a few documents (i.e., rare terms) are considered to be more important in documents; and they are given more weights than the terms occurring in many documents (i.e., frequent terms) (Manning et al., 2008; Salton and Buckley, 1988).

2.2 Inter-document similarity measures

Here, we discuss three commonly used measures to estimate the similarity of two document vectors and , where is a real domain.

2.2.1 Cosine similarity

The cosine similarity measure with the tf-idf term weighting is the most commonly used inter-document similarity measure. Using term weighted vectors, the cosine similarity of two documents and is estimated as:

(1)

Note that the two terms in the denominator of Eqn 1 are the Euclidean lengths (-norms) of the term weighted vectors.

It is important to normalize the similarity of documents by their lengths, otherwise it favors longer documents which have higher probability of having more terms in common with the query document over shorter documents

(Salton and McGill, 1986; Manning et al., 2008; Salton and Buckley, 1988; Singhal et al., 1996).

2.2.2 Best Match 25 (BM25)

BM25 (Roberston and Zaragoza, 2009; Jones et al., 2000) is the state-of-the-art document ranking measure in IR. It is based on the probabilistic framework of term weighting by Robertson et al. (1994). Han et al. (2012) used BM25 to measure the similarity of two documents and as follows:

(2)

where is the normal length of document (i.e., -norm of the unweighted vector); is the average normal document length; and are free parameters that control the influence of the term frequencies and document lengths; and is the idf factor of term defined as follows:

(3)

It uses different variants of tf and idf factors in the similarity measure. The pivoted normal document length (Singhal et al., 1996) is used in the tf factor so that longer documents which have higher probability of having more terms in common with the query document are not favored over shorter documents.

2.2.3 Jaccard similarity

The Jaccard similarity (Jaccard, 1901) of two documents and is estimated as follows:

(4)

where is the set of terms in document and is the cardinality of a set.

It only considers the number of terms occurring in both and and does not take into account the importance of terms in documents. The similarity is normalized by the number of distinct terms occurring in either or to take into account that and have higher chance of having terms in common if they have more terms.

The weighted or generalized version of Jaccard similarity (Chierichetti et al., 2010) of two documents using term weighted vectors is defined as follows:

(5)

The similarity of and w.r.t depends on the importance of in the two documents. The similarity is normalized by the sum of maximum weights of all .

Note that the weighted Jaccard similarity of and (Eqn 5) in the binary BoW vector representation without any term weighting is equivalent to the traditional Jaccard similarity (Eqn 4).

3 Issues of the tf-idf assumptions in inter-document similarity measurements

Even though the tf and idf assumptions discussed in Section 2.1 are intuitive in the IR task to rank documents for a given query phrase of a few terms, they can be counter-intuitive in the query-by-example task which requires inter-document similarity measurements to rank documents in w.r.t a given query document.

In the literature, the query-by-example task is treated as the IR task where query is a document; and the same idea of the tf-idf term weighting is used. However, there is a fundamental difference between the two tasks — unlike in the typical IR task where the query comprises of a few distinct terms (i.e., each term generally occurs only once in the query phrase), the query in the query-by-example task is a long document which often has multiple occurrences of terms.

3.1 Issue of the tf assumption

For a query document with terms , a document having more occurrences of terms in than in , may not be more similar to than another document , which has similar occurrences of terms in as in . For example, let’s assume and have frequencies of as and , respectively. If has , it is difficult to say that is more similar to than w.r.t , simply because of (and ). It might be the case that is exactly the same document as .

Because of the tf-based term weighting factor, can be more similar to than itself using some existing measure such as BM25It depends on the lengths of documents and parameters and .. Thus, the tf assumption can be counter-intuitive in inter-document similarity measurements.

3.2 Issue of the idf assumption

Similarly, having rare terms of may not be more similar to than having frequent terms of . For example, let’s assume the scenario presented in Table 2:

1 1 1
0 1 10 10
Table 2: A scenario to demonstrate the issue of the idf assumption. Note that all documents having have its frequency of 1; and all documents having have its frequency of 1 except where
tf-idf term weighting Sp (the proposed approach)
 (i) Underlying Assumptions
tf: if if
idf: if (for ;    
           ; ) where
 (ii) Example discussed in Section 3.1 ()
      because (even though because
                            )            
 (iii) Example discussed in Section 3.2 (Table 2)
   because (i) ; because (i)
                  and                
(ii) (even though            (ii)
    and )                
Table 3: The tf-idf weighting (in existing measures) versus Sp: (i) Underlying assumptions for documents to be relevant/similar to a query document ; and the relation of similarities of and to (ii) in the same example discussed in Section 3.1 and (iii) in the same example used in Section 3.2.

Because of , the term will be completely ignored. However, is more useful than because is the only document in which has as many occurrences of as . Even though there is no discrimination between documents w.r.t (all documents with have its frequency of 1), is assigned more weight with than with . As a result, and become equally similar to w.r.t and even though has exactly the same occurrences of and as . This example shows that the idf assumption can be counter-intuitive in document similarity measurements.

4 Our proposal to overcome the issues of tf-idf based term weighting in document similarity measurements

The main problem of the tf-idf term weighting in inter-document similarity measurements is that the importance of in , , is estimated without considering the frequency of in , . It is not an issue in the IR task because is almost always 1 if occurs in the given query phrase . In a query document, can be larger than 1. Therefore, judging the importance of in , without considering , can be counter-productive in inter-document similarity measurements.

A more fit-to-the-purpose approach would be to evaluate the importance of in by examining the similarity of w.r.t. . However, as discussed in Section 3.2, simply having similar occurrences of (i.e., ) is not sufficient to consider them to be similar. The similarity shall also consider how rare the frequency of is in the collection.

Putting the above requirements together, the similarity of and w.r.t shall be based on the number of documents in which have similar occurrence frequencies of as in both and . More formally, and are more likely to be similar w.r.t if is small. The first part in Table 3 compares the underlying assumptions of the tf-idf term weighting (used in existing measures) and the proposed approach called Sp, to be introduced in the next subsection.

This approach addresses the limitations of both the tf and idf assumptions discussed in Section 3. The results of the new approach using the same examples discussed in Sections 3.1 and 3.2 are provided in the second part of Table 3. The comparisons demonstrate that the new approach provides more intuitive outcomes than the tf-idf term weighting.

4.1 Sp: A new document similarity measure

Recently, Aryal et al. (2014, 2017) introduced a data dependent measure where the similarity of two data objects and depends on the distribution of data between and (Aryal et al., 2014, 2017). The intuition is that and are more likely to be similar if there are less data between them. For example, two individuals earning 800k and 900k are judged to be more similar by human than two individuals earning 50k and 150k because a lot more people earn in [50k, 150k] than [800k, 900k].

Using the similar idea, the similarity of two documents and can be estimated as:

(6)

where is a normalization term to account for the probability of a term occurring in both and . It reduces the bias towards documents having more terms because they have a higher probability of having terms, which also exist in a query document, than documents having fewer terms.

The number of distinct terms is used as a normalization factor (as in the traditional Jaccard similarity) because it is not sensitive to multiple occurrences of the terms in a document which do not occur in the query document. In the IR task, Singhal et al. (1996) have shown that it is more effective than the cosine or normal length normalizations which penalize more to documents having multiple occurrences of the terms which are not in the query phrase.

Sp can be interpreted as a simple probabilistic measure where the similarity of two documents w.r.t is assigned based on the probability of the frequency of to be in [, ], (where

be a random variable representing the occurrence frequency of term

in a document). In practice, , which is the inverse of the term used in Eqn 6.

4.2 Characteristics of Sp

The proposed measure has the following characteristics:

  1. Term weighting is not required:

    Unlike in existing measures such as cosine and BM25, and are not used directly in the similarity measure. They are used just to define and ; and the similarity is based on which is invariant to the monotonic scaling of frequency values. Hence, Sp does not require additional term weighting to adjust frequency values.

  2. Self-similarity is data dependent and the upper bound of similarity:

    Unlike cosine and both variants of Jaccard similarity where the self-similarity of documents is fixed with the maximum of 1, Sp has data dependent self similarity because depends on the for all . Thus, and can be different.

    The similarity in Sp is bounded by its self-similarity i.e., . Although BM25 also has data dependent self-similarity, it is possible to have similarity of different documents to be larger than the self-similarity, i.e., there may be with It depends on the lengths of and and parameters and ..

  3. Relationship with the traditional Jaccard similarity and idf term weighting:

    The formulation of Sp (Eqn 6) looks similar to the formulation of the traditional Jaccard similarity (Eqn 4) except that the similarity of and w.r.t is based on in Sp, whereas it is the fixed constant of 1 in the traditional Jaccard similarity.

    In the binary BoW vector representation, when and , Sp assigns the similarity of and w.r.t based on , whereas in the traditional Jaccard similarity, it is 1, irrespective of whether is rare or frequent in .

    In the term-frequency-based BoW representation, Sp is different from the idf weighting because is based on , whereas Sp is based on , where and when .

4.3 Computational complexity

In the term-frequency-based BoW vector representation, it appears that computing naively can be expensive as it requires a range search to find the number of documents having the frequencies of between and . Since, all are integers (term occurrence frequency counts), it can be computed in constant time by the following simple preprocessing.

Let be the maximum occurrence frequency of term in any document in the given collection . We can maintain a cumulative frequency count array of size where contains the number of documents having occurrences of fewer than or equal to .

Using , it can be estimated in constant time as . Note that can not be 0 because is computed only if (i.e., and ) and thus .

The above preprocessing requires time complexity and space complexity, where is the average maximum frequency of terms.

With the above preprocessing, the runtime of computing the similarity of and using Sp is the same as that of the existing similarity measures which is .

5 Empirical evaluation

In this section, we present the results of experiments conducted to evaluate the task specific performances of Sp, BM25, weighted Jaccard and cosine similarity in the query-by-example task to retrieve documents similar to a given query document. We did experiments with both the term-frequency-based and binary BoW vector representations. We used different combinations of tf and idf based term weighting factors with the weighted Jaccard and cosine similarity measures.

5.1 Datasets and experimental setup

We used 10 datasets from 6 benchmark document collections. The characteristics of data sets are provided in Table 4. NG20, R8, R52 and Webkb are from Cardoso-Cachopo (2007)BoW vectors available at: http://web.ist.utl.pt/acardoso/datasets/; and others are from Han and Karypis (2000)BoW vectors available at: http://www.cs.waikato.ac.nz/ml/weka/datasets.html.

Name Collection
Fbis 2,463 2,000 17 TREC collection
La1s 3,204 13,195 6 TREC collection
La2s 3,075 12,432 6 TREC collection
New3s 9,558 26,832 44 TREC collection
Ng20 18,821 5,489 20 20 Newsgroup collection
Ohscal 11,162 11,465 10 Ohsumed patients records
R8 7,674 3,497 8 Reuters collection
R52 9,100 7,379 52 Reuters collection
Wap 1,560 8,460 20 Yahoo web pages
Webkb 4,199 1,817 4 University web pages
Table 4: Characteristics of datasets (: Number of documents, : Number of terms, : Number of classes).

Each dataset was divided into two subsets and using a 10-fold cross validation which have 90% and 10% of the documents, respectively. was used as a given collection from which similar documents were extracted for each query document in . For each , documents in were ranked in descending order of their similarities to using different contending similarity measures. The top documents were presented as the similar documents to .

For performance evaluation, a document was considered to be similar to if they have the same class label. In order to demonstrate the consistency of a measure at different top retrieved results, we evaluated the precision at the top retrieved results ( in terms of percentage) with and used the mean average precision up to . The performance evaluation criterion is: .

We repeated the experiment 10 times using each of the 10 folds as and the remaining 9 folds as . The average

and standard error over the 10 runs were reported. The average

of two measures were considered to be significantly different if their confidence intervals based on two standard errors were not overlapping.

The free parameters and in BM25 were set to 1.2 and 0.95, respectively, as recommended by Paltoglou and Thelwall (2010) and Jones et al. (2000).

All the experimental setups and similarity measures were implemented in Java using the WEKA platform (Hall et al., 2009). All the experiments were conducted in a Linux machine with 2.27 GHz processor and 16 GB memory. We discuss the experimental results with the term-frequency-based and binary BoW vector representations separately in the following two subsections.

5.2 Results in the term-frequency-based BoW vector representation

Here we used two term weighting schemes - tf factor only and tf-idf factors, with weighted Jaccard and cosine. The six contending measures are: Sp, BM25, Cos.tf-idf (cosine with tf-idf), Cos.tf (cosine with tf only), WJac.tf-idf (weighted Jaccard with tf-idf) and WJac.tf (weighted Jaccard with tf only).

The average and standard error over 10 runs of six contending measures are provided in Table 5 and the summarized results in terms of pairwise win-loss-draw counts of contending measures based on the two standard errors significance test over the 10 datasets used in the experiment are provided in Table 6.

BM25 Cos.tf-idf Cos.tf WJac.tf-idf WJac.tf Sp
Fbis 65.120.62 68.420.61 68.280.58 68.480.49 66.750.54 67.770.51
La1s 74.410.32 75.970.42 73.080.49 79.180.33 77.540.47 79.360.32
La2s 76.420.49 78.110.42 75.240.44 81.060.42 79.450.37 80.890.40
New3s 67.010.18 68.310.19 70.190.19 69.360.16 68.450.15 68.980.16
Ng20 76.470.19 74.810.24 67.800.28 73.670.23 64.280.24 72.300.20
Ohscal 59.720.22 53.590.21 61.060.26 59.680.21 60.810.20 60.140.19
R52 85.500.20 80.800.27 86.570.15 84.550.21 84.720.19 84.390.22
R8 91.050.14 86.140.22 92.930.19 91.030.18 91.940.21 91.400.17
Wap 19.670.42 65.330.34 61.970.41 70.540.46 65.100.48 70.920.50
Webkb 70.280.23 68.550.24 73.040.27 73.900.31 75.250.25 74.910.33
Table 5: Term-frequency-based BoW representation: Average and standard error over 10 runs. The best result is underlined and the results equivalent (insignificant difference based on two standard errors) to the best result are bold faced.
Sp WJac.tf WJac.tf-idf Cos.idf Cos.tf-idf
BM25 8-2-0 8-2-0 6-2-2 7-0-3 5-5-0
Cos.tf-idf 8-1-1 6-2-2 8-1-1 5-4-1
Cos.tf 5-4-1 4-5-1 5-4-1
WJac.tf-idf 3-2-5 3-6-1
WJac.tf 5-2-3
Table 6: Term-frequency-based BoW representation: Win-loss-draw counts of measures in columns against those in rows based on the two standard error significance test over 10 runs.

Table 5 shows that Sp and Cos.tf produced the best or competitive to the best result in five datasets each; followed by WJac.tf-idf in four; whereas Cos.tf-idf, BM25 and WJac.tf were best or competitive to the best measure in only one dataset each.

The first column in Table 6 shows that Sp had more wins than losses over all contending measures. It had one more wins than losses against the closest contenders Cos.tf and WJac.tf-idf.

Of the two cosine measures, Cos.tf had more wins than losses to Cos.tf-idf. This shows that the idf term weighting can be counter-productive with cosine in inter-document similarity measurements. It is mainly due to the cosine normalization which penalizes more to documents having rare terms (with high idf weights) which are not in . In comparison to BM25, Cos.tf produced better results with seven wins and no loss; and Cos.tf-idf was competitive with five wins versus five losses.

It is interesting to note that, in the Wap dataset, BM25 produced significantly worse result than other contenders. It is due to the idf factor used in BM25. If a term occurs in more than half of the documents in (i.e., ), is negative and has negative contribution in the similarity of two documents. When was replaced by the traditional in the formulation of BM25 (Eqn 2), it produced = 67.04% which was still worse than those of Sp and WJac.tf-idf.

In weighted Jaccard similarity, WJac.tf-idf produced better retrieval results than WJac.tf. It is interesting to note that WJac.tf-idf produced better retrieval results than Cos.tf-idf, Cos.tf and BM25. This could be mainly due to the vector length normalization used in BM25 and cosine that penalizes more to documents having higher frequencies of terms which are not in .

It is interesting to note that Sp and WJac.tf-idf produced more consistent results than the other contending measures. They did not produce the worst result in any dataset whereas WJac.tf produced the worst result in one dataset (NG20) followed by Cos.tf in two datasets (La1s and La2s); BM25 in three datasets (Fbis, New3s and Wap); and Cos.tf-idf in four datasets (Ohscal, R8, R52 and Webkb).

In terms of runtime, all measures had runtime in the same order of magnitude. For example, in the NG20 dataset, the average total runtime of one run (including preprocessing) using Sp took 15935 seconds; whereas BM25, Cos.tf-idf and WJac.tf-idf took 27432, 16089 and 14875 seconds, respectively.

5.3 Results in the binary BoW vector representation

Here, six contending measures are: Sp, BM25, Cos.idf (cosine with idf), Cos (cosine without idf), WJac.idf (weighted Jaccard with idf) and WJac (weighted Jaccard without idf). Note that WJac which is not using any term weighting is equivalent to the traditional Jaccard similarity defined in Eqn 4.

The average and standard error over 10 runs of the six contending measures are provided in Table 7; and the summarized results in terms of pairwise win-loss-draw counts of contending measures based on the two standard errors significance test over the 10 datasets used in the experiment are provided in Table 8.

BM25 Cos.idf Cos WJac.idf WJac Sp
Fbis 67.900.50 66.460.50 63.240.56 67.170.46 64.580.52 66.940.47
La1s 74.780.25 76.780.34 75.960.38 78.540.34 77.550.39 79.040.30
La2s 76.710.48 78.480.42 77.550.38 80.020.39 79.120.35 80.540.40
New3s 69.610.20 66.730.16 64.880.15 67.760.15 65.660.16 68.130.16
Ng20 74.370.16 73.800.17 64.120.20 72.260.19 63.070.21 72.610.20
Ohscal 58.950.19 55.060.17 58.560.18 58.660.21 58.450.17 59.230.19
R52 83.870.24 79.010.28 84.190.20 83.230.22 83.360.21 83.800.22
R8 90.540.16 86.030.19 91.600.17 90.240.17 91.100.20 90.920.18
Wap 16.470.34 66.970.47 59.160.44 70.180.54 65.090.48 70.020.53
Webkb 73.290.39 70.860.23 75.610.27 74.190.37 75.590.29 74.970.35
Table 7: Binary BoW representation: Average and standard error over 10 runs. The best result is underlined and the results equivalent (insignificant difference based on two standard errors) to the best result are bold faced.
Sp WJac WJac.idf Cos Cos.idf
BM25 5-2-3 5-5-0 4-3-2 4-4-2 3-7-0
Cos.idf 8-1-1 5-4-1 8-1-1 4-6-0
Cos 7-2-1 5-3-2 6-3-1
WJac.idf 5-0-5 2-6-2
WJac 8-0-2
Table 8: Binary BoW representation: Win-loss-draw counts of measures in columns against those in rows based on the two standard error significance test over 10 runs.

Table 7 shows that Sp produced the best or competitive to the best result in six datasets; followed by BM25 in five; WJac.idf in four; Cos in two; and WJac in one dataset only. Cos.idf did not produce competitive result to the best performing measure in any dataset.

In terms of pairwise win-loss-draw counts as shown in the first column in Table 8, Sp had many more wins than losses against all other contending measures.

It is interesting to note that BM25, Cos.idf and Cos using the binary BoW representation produced better retrieval results than their respective counterparts using the term-frequency-based BoW representation in some datasets. For example: (i) BM25 in Fbis, New3s and Webkb; (ii) Cos.idf in La1s, Ohscal, Wap and Webkb; and (iii) Cos in La1s, La2s and Webkb. In contrast, WJac.idf, WJac and Sp using binary BoW vectors did not produce better retrieval results than their respective counterparts using term-frequency-based BoW vectors.

Like in the term-frequency-based BoW representation, all measures had runtimes in the same order of magnitude.

6 Discussion

Even though some studies have used different variants of tf and idf term weighting factors with the most widely used cosine similarity, the tf and idf factors discussed in Section 2.1 have been shown to be the most consistent in the IR task (Singhal, 1997).

For the tf factor, instead of using the logarithmic scaling of , some researchers have used other scaling approaches such as augmented (Salton and Buckley, 1988) and Okapi (Robertson et al., 1994). Similarly, for the idf factor, instead of using , some researchers have used the probabilistic idf factor based on (Robertson et al., 1994; Singhal, 1997). Note that BM25 (Eqn 2) uses tf factor similar to Okapi and idf factor similar to the probabilistic idf factor (Roberston and Zaragoza, 2009).

In the supervised text mining task of document classification, different approaches utilizing class information are proposed to estimate the collection-based term weighting factors (Wang and Zhang, 2013; Debole and Sebastiani, 2003; Lan et al., 2009). Inverse category frequency (icf) (Wang and Zhang, 2013) has been shown to produce better classification result than the traditional idf factor with the cosine similarity measure. It considers the distribution of a term among classes rather than among documents in the given collection. The intuition behind icf is that the fewer classes a term occurs in, the more discriminating power the term contributes to classification (Wang and Zhang, 2013). If and are the total number of classes and the number of classes in which occurs at least once in at least one document, then the icf factor is estimated as: .

We have evaluated the performance of Sp in the kNN document classification task with existing measures using the supervised term weighting scheme of icf

(Wang and Zhang, 2013). Sp produced either better or competitive classification results with existing measures using supervised or unsupervised term weighting in the 5NN classification task. The classification results are provided in the Appendix.

Even though the weighted Jaccard similarity has been used in other application domains (Chierichetti et al., 2010), it is not widely used in the literature to measure similarities of BoW documents. Our experimental results in Section 5 show that the weighted Jaccard similarity with tf-idf term weighting scheme can be an effective alternative of cosine and BM25 in inter-document similarity measurements.

Sp has superior performance over all contenders in the binary BoW vector representation. It can be very useful in application domains such as legal and medical where the exact term frequency information may not available due to privacy issue because it is possible to infer information in a document from its term frequencies (Zhu et al., 2008).

7 Concluding remarks

For the purpose of inter-document similarity measurements task, we identify the limitations of the underlying assumptions of the most widely used tf-idf term weighting scheme employed in existing measures such as cosine and BM25; and provide an alternative which is more intuitive in this task.

Based on the new assumption, we introduce a new simple but effective inter-document similarity measure called Sp.

Our empirical evaluation in the query-by-example task shows that:

  1. Sp produces better or at least competitive results to the existing similarity measures with the state-of-the-art term weighting schemes in the term-frequency-based BoW representations. Sp produces more consistent results than the existing measures across different datasets.

  2. Sp produces better results than the existing similarity measures with or without idf term weighting in the the case of binary BoW representation.

When cosine and BM25 are employed, our result shows that it is important to use an appropriate BoW vector representation (binary or term-frequency-based) and also an appropriate term weighting scheme. Using inappropriate representation and term weighting scheme can result in poor performance.

In contrast, using Sp, users do not have to worry about applying any additional term weighting to measure the similarity of two documents and still get better or competitive results in comparison to the best results obtained by cosine or BM25.

Acknowledgement

The preliminary version of this paper was published in the Proceedings of the 11th Asia Information Retrieval Societies Conference 2015 (Aryal et al., 2015).

References

  • Aryal et al. [2014] S. Aryal, K. M. Ting, G. Haffari, and T. Washio. Mp-dissimilarity: A data dependent dissimilarity measure. In Proceedings of the IEEE International conference on data mining (ICDM), pages 707–712, 2014.
  • Aryal et al. [2015] S. Aryal, K. M. Ting, G. Haffari, and T. Washio. Beyond tf-idf and cosine distance in documents dissimilarity measure. In Proceedings of the 11th Asia Information Retrieval Societies Conference, pages 400–406, 2015.
  • Aryal et al. [2017] S. Aryal, K. M. Ting, T. Washio, and G. Haffari. Data-dependent dissimilarity measure: an effective alternative to geometric distance measures. Knowledge and Information Systems, 53(2):479––506, 2017.
  • Cardoso-Cachopo [2007] A. Cardoso-Cachopo. Improving Methods for Single-label Text Categorization. PhD thesis, Instituto Superior Tecnico, Technical University of Lisbon, Lisbon, Portugal, 2007.
  • Chierichetti et al. [2010] F. Chierichetti, R. Kumar, S. Pandey, and S. Vassilvitskii. Finding the Jaccard Median. In Proceedings of the Twenty-first Annual ACM-SIAM Symposium on Discrete Algorithms, pages 293–311. Society for Industrial and Applied Mathematics, 2010.
  • Debole and Sebastiani [2003] F. Debole and F. Sebastiani. Supervised term weighting for automated text categorization. In Proceedings of the 2003 ACM Symposium on Applied Computing, pages 784–788, New York, NY, USA, 2003. ACM.
  • Hall et al. [2009] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. The weka data mining software: An update. SIGKDD Exploration Newsletter, 11(1):10–18, Nov. 2009.
  • Han and Karypis [2000] E.-H. Han and G. Karypis. Centroid-based document classification: Analysis and experimental results. In Proceedings of the 4th European Conference on Principles of Data Mining and Knowledge Discovery, pages 424–431, London, UK, 2000. Springer-Verlag.
  • Han et al. [2012] X. Han, S. Li, and Z. Shen. A k-NN method for large scale hierarchical text classification at LSHTC3. In Proceedings of the Workshop on Large Scale Hierarchical Classification, pages 1–12, 2012.
  • Jaccard [1901] P. Jaccard. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin de la Société Vaudoise des Sciences Naturelles, 37:547–579, 1901.
  • Joachims [1997] T. Joachims. A probabilistic analysis of the rocchio algorithm with tfidf for text categorization. In

    Proceedings of the Fourteenth International Conference on Machine Learning

    , pages 143–151, San Francisco, CA, USA, 1997. Morgan Kaufmann Publishers Inc.
  • Jones et al. [2000] K. S. Jones, S. Walker, and S. E. Robertson. A probabilistic model of information retrieval: Development and comparative experiments. Information Processing and Management, 36(6):779–808, 2000.
  • Lan et al. [2009] M. Lan, C. L. Tan, J. Su, and Y. Lu. Supervised and traditional term weighting methods for automatic text categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4):721–735, 2009.
  • Manning et al. [2008] C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA, 2008.
  • Paltoglou and Thelwall [2010] G. Paltoglou and M. Thelwall.

    A study of information retrieval weighting schemes for sentiment analysis.

    In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1386–1395. Association for Computational Linguistics, 2010.
  • Roberston and Zaragoza [2009] S. Roberston and H. Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009.
  • Robertson et al. [1994] S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford. Okapi at trec-3. In Proceedings of the Third Text Retrieval Conference (TREC 1994), pages 109–126, 1994.
  • Salton and Buckley [1988] G. Salton and C. Buckley. Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24(5):513–523, 1988.
  • Salton and McGill [1986] G. Salton and M. J. McGill. Introduction to Modern Information Retrieval. McGraw-Hill, Inc., New York, NY, USA, 1986.
  • Singhal et al. [1996] A. Singhal, C. Buckley, and M. Mitra. Pivoted document length normalization. In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 21–29, New York, NY, USA, 1996. ACM.
  • Singhal [1997] A. K. Singhal. Term Weighting Revisited. PhD thesis, The Faculty of the Graduate School, Cornell University, 1997.
  • Wang and Zhang [2013] D. Wang and H. Zhang. Inverse-category-frequency based supervised term weighting schemes for text categorization. Journal of Information Science and Engineering, 29(2):209–225, 2013.
  • Zhu et al. [2008] X. Zhu, A. B. Goldberg, M. Rabbat, and R. Nowak. Learning Bigrams from Unigrams. In Proceedings of ACL-08: HLT, pages 656–664. Association for Computational Linguistics, 2008.
  • Zobel and Moffat [1998] J. Zobel and A. Moffat. Exploring the similarity space. SIGIR Forum, 32(1):18–34, 1998.

Appendix A: kNN classification results

In order to predict a class label for a test document , its nearest neighbour (or most similar) documents were searched in the given labelled training set of documents using a contending similarity measure and the majority class among the NNs was predicted as the class label for .

All classification experiments were conducted using a 10-fold cross validation (10 runs with each one out of the 10 folds as the test set and the remaining 9 folds as the training set). The average classification accuracy and standard error over a 10-fold cross validation were reported. All collection-based term weighting factors (idf and icf) were computed from the training set and used in both the training and test documents. The parameter was set to a commonly used value of 5 (i.e., 5NN classification was used).

We discuss the 5NN classification results with the term-frequency-based and binary BoW vector representations separately in the following two subsections.

Term-frequency-based BoW vector representation

We used term weighting based on tf only, tf-idf and tf-icf with weighted Jaccard and cosine resulting in eight contending measures: Sp, BM25, Cos.tf-icf, Cos.tf-idf, Cos.tf, WJac.tf-icf, WJac.tf-idf and WJac.tf.

The average classification accuracies and standard errors over a 10-fold cross validation of the eight contending measures are provided in Table 10 and the summarized results in terms of pairwise win-loss-draw counts of contending measures based on the two standard errors significance test in the 10 datasets used in the experiment are provided in Table 11.

BM25.tf Cos.tf.icf Cos.tf.idf Cos.tf WJac.tf.icf WJac.tf.idf WJac.tf Sp
Fbis 76.981.04 80.830.84 79.331.05 80.230.78 79.290.95 79.540.91 79.210.84 79.210.80
La1s 83.680.58 83.430.84 86.700.66 82.300.80 87.300.80 88.890.61 87.050.70 88.480.46
La2s 86.280.50 84.520.47 87.930.77 84.230.61 88.460.63 90.110.41 88.030.56 89.590.48
New3s 79.310.30 79.540.34 78.990.34 81.470.29 80.900.30 80.860.33 80.600.36 80.550.36
Ng20 88.550.15 87.570.22 86.920.22 84.740.34 86.280.27 87.410.25 83.050.28 86.620.17
Ohscal 72.630.29 72.040.43 66.950.45 74.250.46 74.500.29 72.360.21 74.220.33 73.190.34
R52 92.300.20 91.200.28 87.720.54 92.180.18 91.690.22 91.170.22 90.630.21 90.940.25
R8 95.190.21 95.340.17 90.800.25 95.810.23 95.390.20 94.980.23 95.270.31 95.280.27
Wap 17.760.79 75.900.46 76.920.76 72.440.58 80.700.80 82.310.92 76.220.58 82.500.79
Webkb 81.160.38 81.860.48 77.920.43 81.580.57 84.140.42 83.330.61 84.400.38 84.330.53
Table 10: Binary BoW representation: Average 5NN classification accuracy and standard error over a 10-fold cross validation. The best result is underlined and the results equivalent (insignificant difference based on two standard errors) to the best result are bold faced.
BM25 Cos.icf Cos.idf Cos WJac.icf WJac.idf WJac Sp
Fbis 79.290.65 79.620.99 77.750.95 78.200.88 79.821.08 79.860.98 78.280.81 79.050.82
La1s 84.270.63 87.450.79 87.700.63 85.890.72 88.050.73 88.700.54 87.550.62 88.670.49
La2s 86.080.58 87.480.49 89.430.42 86.670.50 88.520.68 89.990.43 88.000.60 90.150.48
New3s 80.720.42 79.100.36 78.310.41 78.190.35 79.790.36 80.030.41 78.740.29 80.150.40
Ng20 87.590.13 87.190.20 87.250.19 82.800.20 85.640.12 86.610.18 82.160.20 86.840.24
Ohscal 72.020.32 72.360.31 68.540.32 72.890.24 73.650.28 72.360.32 72.890.29 72.790.33
R52 91.200.24 89.740.38 86.140.43 90.210.27 90.510.25 89.810.24 89.750.17 90.800.19
R8 94.800.21 95.050.13 90.980.44 94.990.23 95.100.22 94.540.30 94.860.17 95.050.31
Wap 15.510.69 76.860.65 78.271.01 69.680.82 80.000.56 81.920.96 76.280.62 81.600.81
Webkb 83.970.49 84.680.54 81.450.60 84.260.49 84.880.44 84.160.47 84.730.42 84.710.53
Table 9: Term-frequency-based BoW representation: Average 5NN classification accuracy and standard error over a 10-fold cross validation. The best result is underlined and the results equivalent (insignificant difference based on two standard errors) to the best result are bold faced.
Sp WJac.tf WJac.tf-idf WJac.tf-icf Cos .tf Cos.tf-idf Cos.tf-icf
BM25 6-2-2 7-2-1 6-2-2 7-2-1 5-2-3 4-5-1 2-3-5
Cos.tf-icf 6-1-3 5-2-3 5-0-5 6-1-2 4-2-4 2-5-3
Cos.tf-idf 8-0-2 5-1-4 9-0-1 6-1-3 5-4-1
Cos.tf 5-4-1 4-3-3 5-3-2 5-1-4
WJac.tf-icf 2-2-6 0-3-7 3-2-5
WJac.tf-idf 1-1-8 2-5-3
WJac.tf 4-1-5
Table 11: Term-frequency-based BoW representation: Win-loss-draw counts of measures in columns against those in rows based on the two standard errors significance test over a 10-fold cross validation of 5NN classification.
Sp WJac WJac.idf WJac.icf Cos Cos.idf Cos.icf
BM25 4-1-5 4-3-3 3-2-5 4-3-3 3-3-4 3-6-1 3-3-4
Cos.icf 4-0-6 0-1-8 3-2-5 3-1-6 0-4-6 1-5-4
Cos.idf 6-0-4 4-3-3 7-1-2 7-1-2 4-4-2
Cos 6-0-4 3-2-5 5-0-5 6-0-4
WJac.icf 3-1-6 0-5-5 3-3-4
WJac.idf 1-0-9 0-4-6
WJac 6-0-4
Table 12: Binary BoW representation: Win-loss-draw counts of measures in columns against those in rows based on the two standard errors significance test over a 10-fold cross validation of 5NN classification.

The 5NN classification accuracies in Table 5 show that Sp, WJac.tf-idf, WJac.tf-icf and Cos.tf produced the best or competitive to the best result in five datasets each followed by WJac.tf in four; Cos.tf-icf and BM25 in two datasets each; and Cos.tf-idf in one dataset only.

The pairwise win-loss-draw counts of Sp in the first column of Table 11 shows that it had more wins than losses over all contending measures except Wjac.tf-idf and Wjac.tf-icf where it had competitive results with the same number of wins and losses.

It is interesting to note that Sp and all three variants of weighted Jaccard similarity produced better classification results than all three variants of cosine and BM25. Like in the similar document retrieval task discussed in Section 5, BM25 produced the worst classification accuracy in the Wap dataset because of . The classification accuracy was increased to 79.42% when was replaced by the traditional idf .

The supervised term weighting using icf (tf-icf) did not always produce better classification results than the traditional tf-idf based term weighting with both cosine and weighted Jaccard. It had five wins and two losses with cosine whereas it had two wins and three losses with weighted Jaccard.

Binary BoW vector representation

We used weighted Jaccard and cosine similarities with and without idf and icf weighting resulting in eight contending measures: Sp, BM25, Cos.idf, Cos.icf, Cos, WJac.idf, WJac.icf and WJac.

The average classification accuracies and standard errors over a 10-fold cross validation of the eight contending measures are provided in Table 10 and the summarized results in terms of pairwise win-loss-draw counts of contending measures based on the two standard errors significance test in the 10 datasets used in the experiment are provided in Table 12.

The 5NN classification accuracies in Table 10 show that Sp produced the best or competitive to the best result in eight datasets. The closest contenders BM25 and WJac.idf produced the best or competitive to the best result in six datasets each followed by WJac.icf in five; Cos.icf in four; WJac and Cos in three datasets each; and Cos.idf in two datasets only.

In terms of pairwise win:loss:draw counts as shown in the first column in Table 12, Sp had more wins than losses against all other contending measures. It had one win and no loss against WJac.idf and three wins and one loss against WJac.icf.

Like in the term-frequency-based BoW representation, the supervised weighting scheme based on icf did not always produce better classification results than the traditional idf based term weighting scheme with both cosine and weighted Jaccard in the binary BoW vector presentation as well. It had five wins and one loss with cosine whereas it had three wins and three losses with the weighted Jaccard.

It is interesting to note that BM25, Cos.icf, Cos.idf and Cos which are using the binary BoW vector representation produced better classification accuracies than their respective counterparts using the term-frequency-based BoW representation in some datasets; e.g., BM25 was better in three datasets (Fbis, New3s, WebKb); Cos.icf and Cos in three datasets (La1s, La2s, Webkb); and Cos.idf in two datasets (La2s, Webkb). However, all three variants of weighted Jaccard and Sp with the term-frequency-based BoW representation produced either better or competitive results with the binary BoW representation.