Terminology-based Text Embedding for Computing Document Similarities on Technical Content

06/05/2019 ∙ by Hamid Mirisaee, et al. ∙ 0

We propose in this paper a new, hybrid document embedding approach in order to address the problem of document similarities with respect to the technical content. To do so, we employ a state-of-the-art graph techniques to first extract the keyphrases (composite keywords) of documents and, then, use them to score the sentences. Using the ranked sentences, we propose two approaches to embed documents and show their performances with respect to two baselines. With domain expert annotations, we illustrate that the proposed methods can find more relevant documents and outperform the baselines up to 27 NDCG.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

This article has been published in the Proceedings of the TALN-RECITAL 2019 conference. The original manuscript is available on the ACL Anthology website: http://www.aclweb.org/

Many Machine Learning (ML) applications require the calculation of similarities between instances of the task under consideration. Those instances could be of any nature such as numerical and/or categorical time series

[1], gene expression [2] or textual documents [3, 4].

Perhaps the application of similarity assessment between objects that is vastly used on an everyday basis is the one used in search engines. As the most evident example, when a user is looking, via a given query and a search engine such as Google, for relevant documents, there is basically a matching process happening behind. In such application, the user provides a query, as a set of keywords, and looks for the documents which best correspond to those keywords. This mainly leads to a ranking problem where the goal is to rank all the available documents (webpages in case of a web search engine) and provide the user with the most relevant documents. Session search [5] is an interesting challenge of such applications where the user reformulates the query based on the results returned by the system until the desired documents are found.

As one can notice, the principal notion of the above-mentioned task is the similarity between instances. For example, a very common similarity measure between textual documents is the tf-idf which is based on two notions: the frequency of the terms appearing in the document, tf, and the importance of those terms through the entire set of documents, i.e. inverse document frequency, idf

. Many studies still use tf-idf to perform text-related task as it can, for many tasks, properly project the textual data into the numerical space such that the content is reflected accordingly. One can then use the tf-idf vectors of documents to measure the similarity between them, via cosine for instance. The tf-idf representation is widely used in different applications such as document clustering

[6] and topic modeling [7].

As one can see, tf-idf embeds the document by operating at the word-level, i.e. it takes the tf-idf score of each word and represent the document as a (sparse) vector of size of the entire vocabulary. There is, however, another recent word-level operated technique, namely word2vec [8], which also aims at representing words. Nevertheless, there are two main differences between these two approaches. Firstly, tf-idf assigns a scalar to each word while word2vec represents each word by a vector. Secondly, and more importantly, word2vec embeds each word using its context, i.e. its surrounding words. That being said, it is able to capture the semantical context of words. The latter has shown very good results in many applications such as text clustering and text classification [9, 10].

As mentioned previously, calculating document similarity and, consequently, finding similar documents is at the core of many ML tasks. Sometimes, however, focusing on the entire document may not lead to capturing desired similar documents as not all parts of a document have the same importance level. For instance, if a document describes a novel device for people suffering from diabetes, then taking the entire document may not necessarily result in the similar documents talking about the same particular issue, but rather about the medical domain in general. Note that this issue is different from data cleaning, and should be rather considered as data selection/weighting for the task of document embedding.

In this paper, we investigate the above-mentioned problem and propose a technique for document embedding w.r.t. technical content of the documents. We show that the proposed method is able to find better similar documents once the technical content is concerned which, to the best of our knowledge, is the first research targeting this objective. To do so, we propose to first capture the keyphrases, i.e. composite keywords, of the document and, then, rank the sentences based on the keyphrases they contain. To detect keyphrases, we use the state-of-the-art technique explained in [11] which is based on the k-core concept of graph-of-words. We then use the two previously-mentioned embedding techniques to represent the text and show that the proposed, hybrid method can efficiently capture relevant documents based on the technical content.

The reminder of this paper is organized as follows: Section II overviews the most related studies. Section III first details the graph representation of documents and the k-core concept, before providing the framework of embedding documents using the sentence ranking obtained via the k-core approach. Then, in Section IV, we describe the experimental settings and the baselines as well as the collected dataset. We then report our results in the same section. Finally, Section V concludes the paper.

Ii Related work

Over the past few decades, a large body of studies has been considering the relatedness of documents for a wide range of tasks such as text categorization or document classification. Assessing the similarities between documents is at the core of many machine learning applications such as information retrieval, recommendation systems and text generation. In this section, we consider the most relevant studies with respect to the topic under investigation and position our work in regard to them.

Classically, the similarity between two documents has been measured by the cosine similarity between their tf-idf vectors, see

[12] and [13] for example. Alternatively, studies like [14] considered other techniques such as LSI and multi-word methods and investigated their performance in different tasks. Although tf-idf can still capture many characteristics of a document and boost the performance of many tasks such as topic modeling [15, 16], they still fail to detect the entire context of a document such as the semantic relation between words [17].

The paper of Mikolov et al. [8] introduced an entire new idea of textual representation where a document is analyzed at the word level and each word is represented based on its context, i.e.

it carries semantic features of the document. The context of the word is defined by the co-occurring words. The model is simply a neural network where the vectorial embedding of a word is determined by those of its surrounding words. Many variations of this approach have been later studied to, for instance, operate directly at the document level

[18, 19]. In addition, the combination of tf-idf and word2vec has been widely used and shown to bring significant improvement in many applications [10, 20]. As it will be illustrated in Section IV, we will make use of the combination of these two techniques to conduct a part of our experiments, and will show that they work very efficiently in capturing the technical side of documents.

Measuring the similarity between documents has been widely studied in the literature through different approaches. For instance, [21] used a phrase recognition approach to detect similar documents, and [22] employed radix tree to calculate similarities between web documents. In a different approach, [23] introduces a graph-based method to exploit the hierarchical relations in order to efficiently calculate the similarity between documents. Similarly, [24] proposes to represent documents as typed heterogeneous information networks and, following the notion of graphs, it computes the distance between documents for the task of document clustering.

Perhaps the most related studies to ours are [25] and [26]. In [25], the authors propose to use a word-based method to capture similar documents via PLSA. The document matching in this particular work is done based on the words appearing in the document. More precisely, the similarity between two documents is defined as linear combination of the cosine between the tf-idf vectors and the PLSA-based representations of the words. Although the objective of this paper is similar to our work in that they also rely on words-basis scoring, two main differences distinguish that research and ours. Firstly, we focus on keyphrases and pay more attention on the parts of the text where those keyphrases are used, while [25] relies on the entire content of the document. Secondly, the present study uses the context-based techniques, such as word2vec to particularly target the semantic aspects of a document. In other words, if two documents describe the similar subjects with different vocabulary, the purely tf-idf based or topic modeling based approaches fail to see their similarities, while context-based techniques are able to capture that. In a similar manner, [26] defines two documents to be similar if they have the same pieces of text such as sentences or paragraph, which has similar limitations as mentioned before. This current study, to the best of our knowledge, is the first one to propose technical content based similarity between documents using their semantic terminology.

Iii Framework

Graph-based techniques have been widely used in representing textual documents, where, in general, meaningful linguistic units of the text, such as paragraphs [27], sentences [28] or words [29], construct the nodes and the relation between them defines the edges. This relation could be of different natures depending on the task in mind. statistical (such as co-occurrence) and syntactic (such as noun-adjective relation) are two widely used kind of relations between nodes of the graph.

Following this line of thought, [11]

investigated the task of keyword extraction by representing a document as a

graph-of-words and retrieving the main core of the graph. In the following, we briefly explain how this task is accomplished and, then, use the results of this technique to rank the sentences and eventually embed a document.

Iii-a Graph-of-words

In [11], the authors propose to represent a document as a graph where nodes are terms of the document. Two nodes are then connected via an edge if they co-occur within a fixed-size window. More formally:

Definition 1

The graph-of-words of document is defined as where is the set of nodes that represents the terms of and is the set of edges which indicates the co-occurrence of the terms within a fixed-size sliding window of size .

Note that can be weighted or unweighted. In the weighted setting, the weight of is the number of times that and co-occur in the sliding window within a document. One can also consider a directed (weighted/unweighted) version of where the order of words in the document determines the direction of edges. In other words, if precedes in the document within the sliding window then the edge is added to .

With such representation of the document, [11] proposes a technique which focuses more on cohesiveness of the nodes, rather than classic methods, such as [30] and [28], which rely on the notion of centrality. To do that, the k-core approach [31] has been employed:

Definition 2

is called a k-core or a core of order of iff , and , , and is the maximal graph with such property. The core of maximum order is then called the main core of .

The intuition behind using the -core approach is to not only focus on the central nodes of the graph, but also pay attention to how connected the neighbors of the the node are, which is known as the cohesion of a graph. Following this notion, [11] proposes to use the -core approach for keyword extraction.

Basically, the idea is to, starting from the graph-of-words, calculate the -core of the graph and then take all the nodes of the main core, i.e. the core with the maximum order, as keywords. Via extensive experiments, the authors show that their approach outperforms the traditional methods such as HITS and PageRank [30, 28]. Additionally, unlike other techniques that need the number of keywords to extract, the size of the main core basically handles this issue. On top of that, this technique can effectively be used to detect keyphrases, i.e. composite keywords[32]. Once again, in such context, each term of the text is represented as a node in the graph and two connected nodes can potentially construct a keyphrase.

For example, in the sentence ”our platform is based advanced artificial intelligence techniques”, both ”artificial” and ”intelligence” can be considered as keywords, and ”artificial intelligence” can construct a keyphrase. Note that if the size of sliding window explained previously is larger than 2, then keyphrases like ”artificial techniques” and ”advanced intelligence” can also be extracted from the graph as keyphrases.

With that in mind, one can simply use the nodes of the main core to construct the keyphrases using different approaches, by for instance taking the words corresponding to the connected nodes. We use a slight modification of the k-core method in order to, first, extract the keyphrases of size 2, i.e. combination of two and only two terms, and, then, rank the sentences. Using those ranked sentences, we propose a technique to embed the document such that it encodes the main technical content of the document. The proposed approach is detailed in the following.

Iii-B Tde: Terminology-based Document Embedding

As explained above, the k-core, graph-based methods can efficiently be used to extract keywords or keyphrases, where keyphrases often better reflect the semantics of the document as they tend to reduce the noise significantly. Knowing that the objective of this study is to eventually embed a document w.r.t. its technical content through the keyphrases, we first establish a link between the embedded information in the graph-of-words and the resultant keyphrases. We then score the sentences based on the keyphrases they contain and, finally, calculate the embedding of the document via its scored sentences. In other words, the embedding of a document is a weighted average of the embedding of its sentences, where the weights are derived from the graph-of-words which has been explained above.

Let be the set of all cores of the graph-of-words of document where is the maximal order of the graph and is the maximal core. Also, let be the set of all keyphrases appearing in core . More formally, . Then, the embedding of the document , denoted as , can be calculated as the weighted average of the embedding of its sentences:

(1)

where is the set of sentences of , is the embedding of the sentence and is the score of sentence . Eq. (1) is actually a weighted average of the embeddings of all the sentences of the document. Here, the idea is to use the keyphrases of each sentence to determine its score (weight). We propose the following for calculating the score of each sentence of the document:

(2)

where the function returns a score (weight) for each keyphrase . This is mainly where the graph information is taken into account. The score of the keyphrase can be calculated using its properties found in the k-core setting, i.e. the degree of the edge connecting and , as well as the core where those two nodes (terms) appear:

(3)

where is a function that assigns a weight to each core such that cores get monotonically decreasing weights. Obviously the main core has the highest weight. In our experiments, we use the rational function:

(4)

where is the maximum order of the cores.

We implemented the graph-words as an undirected weighted graph where the number of co-occurrences of two words determines the weight of the edge linking them. One should note that the directed version has also been investigated and, according to our observations, was not as good as the undirected one. For the embedding of the sentences (Eq. (1)) we propose two different techniques which are further explained in the next Section. Finally, it should be noted that unlike [11], we use all the cores of the graph as it is shown in Eq. (2) where the more focus is still on the most important cores as expressed in Eq. (4).

Algorithm 1 illustrates the procedure of TDE regardless of the embedding chosen for the sentences of the document. Needless to say, in practice the algorithm can be accelerated if the loops at lines 7 and 8 are done via memory operations. This can be simply done by precomputing and storing all the possible keyphrases (all adjacent nodes of the graph) and their scores, in a dictionary-like data structure for instance.

1:Set containing all sentences of document , : keyphrases of each core
2:: the embedding of
3:
4:
5:for all  do
6:     
7:     for  to  do
8:         for all  do
9:              if  then
10:                     \\  Eqs. (3)-(4)
11:              end if
12:         end for
13:     end for
14:        \\  is the embedding of
15:     
16:end for
17:
18:RETURN
Algorithm 1 Terminology-based Document Embedding

Iv Experiments

Iv-a Baselines

To evaluate the validity of the proposed technique, we compare the method explained in Section III with two document embedding baselines. As mentioned in Section II, many studies investigate the problem of document embedding as an end-to-end problem where the document is embedded all at once, i.e. the embedding is learned at the document level. Accordingly, we chose, as our first baseline, the document embedding method investigated in [18] which is a robust improvement over the original doc2vec [33]. This baseline will be denoted as D2V. Although D2V is not designed particularly for the task that we are investigating, i.e. finding similar documents w.r.t. technical content, we still keep it as a baseline as it is one of the popular techniques to address the document embedding problem.

Following the ideas presented in [34] and [35], one can compute the document embedding by averaging the embedding of the words appearing in the document. The word embeddings can be learned via classical methods such word2vec explained in [8]. According to our experiments, tf-idf weighted average performs significantly better than simple averaging and, consequently, we chose to use it as the second baseline. This baseline is referred to as TWA (Tf-idf Weighted Average).

These two baselines are then compared to the graph-based method proposed in this paper and detailed in Section III. As mentioned in that section, a document is represented as a weighted average of the embeddings of the sentences appearing in it, where the weights are the score of sentences which, in turn, are calculated using the graph-based representation of the document (see Eqs. (1)-(4)). To be consistent with the abbreviation of the previous section, this technique will be denoted as TDE.

In TDE, the embedding of a sentence can be calculated in different ways. Here, we propose two ways of doing that: the first one consists of using the models which are trained to directly produce the sentence embedding; and the second one is to use the embeddings of the words forming the sentence. For the former, the state-of-the-art sentence embedding technique, described in [36] is employed. For the latter, we perform a tf-idf weighted average of the words constructing the sentence to calculate the embedding of the sentence. One should note that as in a sentence it is rarely the case that we have repetition of words, the calculation is almost an idf weighted averaging. To avoid an ambiguity, the first approach is denoted as TDE (for sent2vec) and the second as TDE (for idf weighted). In the following, we explain our dataset from which the embeddings of words, sentences and documents are learned.

Iv-B Dataset, embedding models and preprocessing

We crawled websites of around 68K startups (all around the world, with no constraint on the domain of business) with a total number of 3.4M webpages. After filtering those without sufficient textual information or non-English ones, we ended up with around 43K startups and 2.8M pages. We only used the English sentences of the pages for training our models and performing the evaluations.

To train the word2vec, we used gensim111https://radimrehurek.com/gensim/ on the sentences extracted from the 2.8M pages mentioned above, with a minimum count of words and window size equal to five. The number of sentences extracted reached to 950K. We used the authors’ implementation222https://github.com/epfml/sent2vec of [36] to train the sent2vec model.

Furthermore, keeping in mind that the objective of this study is to investigate document similarities based on the technical content, not all parts of a document are of interest. Knowing that each document in our dataset is the combination of all the textual content of a startup’s website, many parts of that can possibly be considered as noise. For instance, the pages describing privacy policies or legal information must be ignored before performing any document embedding process. As a result, we used multiple classifiers, trained using thousands of pages, to filter out such contents. Accordingly, as the first preprocessing step, we trained three separate SVM classifiers to filter out the pages with privacy, legal information or cookies information.

Iv-C Evaluation

To fairly evaluate the performance of the above-mentioned baselines, TWA and D2V, with that of the proposed ones, TDE and TDE, we adapted the following strategy: we selected a set of 100 documents (startups) from four different domains: medical, agriculture, energy and biology. Evidently, the texts of these documents have not been used for training the above-mentioned models.

For each test case and for each method, we extracted the 5 most similar startups (using cosine similarity between the indexed embedding of each method and the entire dataset). The results are then combined, shuffled and assigned to the corresponding domain expert for evaluation. The experts then give a score to each of the startups from the score set {1, 2, 3, 4, 5} where 1 denotes the least relevant and 5 denotes the most relevant. Note that the experts have been clearly informed that the combined list is not in any way ordered/ranked and the outputs of a test case should be evaluated independently. Obviously, the results do not necessarily contain all the possible scores.

To assess the performance of the proposed approach, we use NDCG (Normalized Discounted Cumulative Gain) which perfectly matches our experimental settings. We report here two values, namely NDCG@1 and NDCG@5, to investigate the behavior of each technique. In the following, we discuss the results of our experiments and illustrate the efficiency of the proposed methods.

Iv-D Results

Table I shows the results of our experiments on all four approaches detailed in Section IV-A

using NDCG@1 and NDCG@5, compared via t-test at 5% for the significance test. As one can notice, the performance of

D2V falls way below other methods with both NDCG@1 and NDCG@5. That could be explained by the fact that D2V needs long documents to be trained properly, and since our documents are only English sentences of only a part of a website (see the filtering procedures explained in Section IV-B), they could be as short as few sentences. As a consequence, D2V has difficulties to project the context into the numerical space. As a result, we will not investigate this technique in our discussion below.

D2V TWA TDE TDE
NDCG@1 0.26 0.54 0.63 0.69
NDCG@5 0.24 0.60 0.60 0.65
TABLE I: NDCG@1 and NDCG@5 on all four methods. The best approach is shown in bold.

The second baseline, i.e. TWA, performs well w.r.t. the proposed methods; as one can observe, it achieves the same NDCG@5 compared to TDE. However, if sent2vec is used to embed the sentences, then one can outperform TWA by 8% in terms of NDCG@5 via TDE

When it comes to NDCG@1, both graph-based variants outperform the baselines significantly: TDE is around 16% better that TWA and TDE has a better NDCG@1 value by 27% w.r.t. TWA.

These experiments show that the proposed technique, be it with the tf-idf weighting or with the sent2vec method, outperform the baselines. In other words, they are able to capture better similar documents when the technical content is of interest. Additionally, according to the results reported on NDCG@1, the proposed methods can find way better results when finding the most relevant document is the task in mind. This could be particularly interesting as in many IR tasks, such as search engines, the first retrieved document plays a very important role in the further processing steps [37].

V Conclusion

In this paper, we have studied the problem of finding similar documents with respect to technical content. To do that, we proposed a hybrid approach which employs the graph representation techniques and sentence embedding. Using the graph-of-words representation of a document and computing the cores of the graph, we first extract the keyphrases of the document of all cores of the graph. We then use this information to assign a score to each sentence; once the scores are calculated, we use the embedding of each sentence and its score to calculate the embedding of the document. We proposed two approaches to embed sentence: (i) sent2vec, a state-of-the-art technique for sentence embedding, and (ii) a tf-idf weighted average of the words appearing in the sentence. We then compared those to two baselines, doc2vec, an existing method in the state-of-the-art to embed documents, and a tf-idf weighted average of the words appearing in the document.

As dataset, we used 2.8M webpages of 43K startups that we crawled from the web, where we considered the combination of all webpages of a startup as a document. To evaluate, we asked human experts to score the output of all techniques. Using the NDCG metric, we illustrated that the proposed metric can outperform the baselines up to 27% and, as a result, can provide better similar documents when technical content is concerned.

Acknowledgments

The authors would like to thank the domain experts of Skopai, Adeline Tarantini, Olivier Berengario, Elie Gehin and Guillaume Emery who spend a considerable amount of time to carefully evaluate the results.

References

  • [1] S. Aghabozorgi, A. S. Shirkhorshidi, and T. Y. Wah, “Time-series clustering–a decade review,” Information Systems, vol. 53, pp. 16–38, 2015.
  • [2] B. Wang, J. Zhu, E. Pierson, D. Ramazzotti, and S. Batzoglou, “Visualization and analysis of single-cell rna-seq data by kernel-based similarity learning,” Nature methods, vol. 14, no. 4, p. 414, 2017.
  • [3] G. Nikolentzos, P. Meladianos, F. Rousseau, Y. Stavrakas, and M. Vazirgiannis, “Shortest-path graph kernels for document similarity,” in

    Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

    , 2017, pp. 1890–1900.
  • [4] Y. Song and D. Roth, “Unsupervised sparse vector densification for short text similarity,” in Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2015, pp. 1275–1280.
  • [5] D. Guan, H. Yang, and N. Goharian, “Effective structured query formulation for session search,” DTIC Document, Tech. Rep., 2012.
  • [6] P. Bafna, D. Pramod, and A. Vaidya, “Document clustering: Tf-idf approach,” in 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT).   IEEE, 2016, pp. 61–66.
  • [7] Z. Zhu, M. Li, L. Chen, and Z. Yang, “Building comparable corpora based on bilingual lda model,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), vol. 2, 2013, pp. 278–282.
  • [8] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013.
  • [9]

    P. Wang, J. Xu, B. Xu, C. Liu, H. Zhang, F. Wang, and H. Hao, “Semantic clustering and convolutional neural network for short text categorization,” in

    Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), vol. 2, 2015, pp. 352–357.
  • [10]

    J. Lilleberg, Y. Zhu, and Y. Zhang, “Support vector machines and word2vec for text classification with semantic features,” in

    2015 IEEE 14th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC).   IEEE, 2015, pp. 136–140.
  • [11] F. Rousseau and M. Vazirgiannis, “Main core retention on graph-of-words for single-document keyword extraction,” in European Conference on Information Retrieval.   Springer, 2015, pp. 382–393.
  • [12]

    B. Trstenjak, S. Mikac, and D. Donko, “Knn with tf-idf based framework for text categorization,”

    Procedia Engineering, vol. 69, pp. 1356–1364, 2014.
  • [13] J. M. Schultz and M. Liberman, “Topic detection and tracking using idf-weighted cosine coefficient,” in Proceedings of the DARPA broadcast news workshop.   San Francisco: Morgan Kaufmann, 1999, pp. 189–192.
  • [14] W. Zhang, T. Yoshida, and X. Tang, “A comparative study of tf* idf, LSI and multi-words for text classification,” Expert Systems with Applications, vol. 38, no. 3, pp. 2758–2765, 2011.
  • [15] L. Hong and B. D. Davison, “Empirical study of topic modeling in twitter,” in Proceedings of the first workshop on social media analytics.   acm, 2010, pp. 80–88.
  • [16] R. Mehrotra, S. Sanner, W. Buntine, and L. Xie, “Improving lda topic models for microblogs via tweet pooling and automatic labeling,” in Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval.   ACM, 2013, pp. 889–892.
  • [17] M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler, “Movieqa: Understanding stories in movies through question-answering,” in

    Proceedings of the IEEE conference on computer vision and pattern recognition

    , 2016, pp. 4631–4640.
  • [18] J. H. Lau and T. Baldwin, “An empirical evaluation of doc2vec with practical insights into document embedding generation,” arXiv preprint arXiv:1607.05368, 2016.
  • [19]

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in

    Advances in neural information processing systems, 2013, pp. 3111–3119.
  • [20]

    J. Acosta, N. Lamaute, M. Luo, E. Finkelstein, and C. Andreea, “Sentiment analysis of twitter messages using word2vec,”

    Proceedings of Student-Faculty Research Day, CSIS, Pace University, p. 7, 2017.
  • [21] J. W. Cooper, A. R. Coden, and E. W. Brown, “Detecting similar documents using salient terms,” in Proceedings of the eleventh international conference on Information and knowledge management.   ACM, 2002, pp. 245–251.
  • [22] A. Pereira and N. Ziviani, “Syntactic similarity of web documents,” in Proceedings of the IEEE/LEOS 3rd International Conference on Numerical Simulation of Semiconductor Optoelectronic Devices (IEEE Cat. No. 03EX726).   IEEE, 2003, pp. 194–200.
  • [23] C. Paul, A. Rettinger, A. Mogadala, C. A. Knoblock, and P. Szekely, “Efficient graph-based document similarity,” in European Semantic Web Conference.   Springer, 2016, pp. 334–349.
  • [24] C. Wang, Y. Song, H. Li, M. Zhang, and J. Han, “Knowsim: A document similarity measure on structured heterogeneous information networks,” in 2015 IEEE International Conference on Data Mining.   IEEE, 2015, pp. 1015–1020.
  • [25] T. Brants and R. Stolle, “Finding similar documents in document collections,” in Using Semantics for Information Retrieval and Filtering: State of the Art and Future Research. Workshop at LREC-2002, 2002.
  • [26] J. W. Cooper, A. R. Coden, and E. W. Brown, “A novel method for detecting similar documents,” in Proceedings of the 35th Annual Hawaii International Conference on System Sciences.   IEEE, 2002, pp. 1153–1159.
  • [27]

    H. Balinsky, A. Balinsky, and S. J. Simske, “Automatic text summarization and small-world networks,” in

    Proceedings of the 11th ACM symposium on Document engineering.   ACM, 2011, pp. 175–184.
  • [28] R. Mihalcea and P. Tarau, “Textrank: Bringing order into text,” in Proceedings of the 2004 conference on empirical methods in natural language processing, 2004.
  • [29] A. Schenker, “Graph-theoretic techniques for web content mining,” 2003.
  • [30] M. Litvak and M. Last, “Graph-based keyword extraction for single-document summarization,” in Proceedings of the workshop on Multi-source Multilingual Information Extraction and Summarization.   Association for Computational Linguistics, 2008, pp. 17–24.
  • [31] S. B. Seidman, “Network structure and minimum degree,” Social networks, vol. 5, no. 3, pp. 269–287, 1983.
  • [32] F. Rousseau, “Ph.D. Dessertation, http://frncsrss.github.io/papers/rousseau-dissertation.pdf,” 2015.
  • [33] Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in International conference on machine learning, 2014, pp. 1188–1196.
  • [34] J. Wieting, M. Bansal, K. Gimpel, and K. Livescu, “Charagram: Embedding words and sentences via character n-grams,” arXiv preprint arXiv:1607.02789, 2016.
  • [35] S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski, “A latent variable model approach to pmi-based word embeddings,” Transactions of the Association for Computational Linguistics, vol. 4, pp. 385–399, 2016.
  • [36]

    M. Pagliardini, P. Gupta, and M. Jaggi, “Unsupervised learning of sentence embeddings using compositional n-gram features,” in

    Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), vol. 1, 2018, pp. 528–540.
  • [37] T. Joachims, L. A. Granka, B. Pan, H. Hembrooke, and G. Gay, “Accurately interpreting clickthrough data as implicit feedback,” in Sigir, vol. 5, 2005, pp. 154–161.