Semantic Regularities in Document Representations

Recent work exhibited that distributed word representations are good at capturing linguistic regularities in language. This allows vector-oriented reasoning based on simple linear algebra between words. Since many different methods have been proposed for learning document representations, it is natural to ask whether there is also linear structure in these learned representations to allow similar reasoning at document level. To answer this question, we design a new document analogy task for testing the semantic regularities in document representations, and conduct empirical evaluations over several state-of-the-art document representation models. The results reveal that neural embedding based document representations work better on this analogy task than conventional methods, and we provide some preliminary explanations over these observations.



There are no comments yet.


page 1

page 2

page 3

page 4


Efficient Vector Representation for Documents through Corruption

We present an efficient document representation learning framework, Docu...

Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec

Distributed dense word vectors have been shown to be effective at captur...

Contextualized Word Representations for Reading Comprehension

Reading a document and extracting an answer to a question about its cont...

Semantic Document Distance Measures and Unsupervised Document Revision Detection

In this paper, we model the document revision detection problem as a min...

Proof of the tree module property for exceptional representations of the quiver 𝔼_6

This document (together with the ancillary file e6_proof.pdf) is an appe...

Explaining Away Syntactic Structure in Semantic Document Representations

Most generative document models act on bag-of-words input in an attempt ...

SIGNet: Scalable Embeddings for Signed Networks

Recent successes in word embedding and document embedding have motivated...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Recently, Tomas:Linguistic discovered that word representations learned by a recursive neural net (RNN) as well as by related log-linear models [Mikolov et al.2013b] can capture the linguistic regularities in language, which allows easy solutions to analogy questions of the form “Beijing:China as Paris:    ” using simple linear algebra. With this word analogy task, a flurry of subsequent work exhibited that similar linear structure can also be revealed from representations learned from other methods [Mnih and Kavukcuoglu2013, Pennington et al.2014, Levy and Goldberg2014b].

Besides word representation, document representation is also a fundamental and critical problem in natural language processing. Over the past decades, various methods have been proposed to represent the document as a vector, including Bag of Words (BOW) 

[Harris1954], Latent Semantic Indexing (LSI) [Deerwester et al.1990], Non-negative Matrix Factorization (NMF) [Lee and Seung1999]) and Latent Dirichlet Allocation (LDA) [Blei et al.2003]. Recently, there is a rising enthusiasm for applying the neural embedding methods to representing the documents [Srivastava et al.2013, Le and Mikolov2014].

It is, therefore, natural to ask whether there is also linear structure in these learned document representations to allow similar reasoning at document level. For example, given three articles talking about naive bayes, logistic regression, and hidden markov model, is it possible to find the article about conditional random fields as the solution to the document analogy question “naive bayes: logistic regression as hidden Markov model:

    ” (i.e., document pairs explaining generative-discriminative model relations)? Obviously, such reasoning is much more complex in semantics and cannot be achieved by simple retrieval or classification based on lexical information. Representation with such linear structure would be useful for many semantic processing applications, e.g., it may help controversial search [Dori-Hacohen et al.2015] by discovering document pairs talking about opposite facts on controversial topics with some seed pairs, or help non-local corpus navigation and paper recommendation together with word vectors [Dai et al.2014].

For this purpose, we introduce a new document analogy task for evaluating the semantic regularities in document representations. Since it is non-trivial to directly label the analogy questions over documents, we leverage the existing word/phrase semantic analogy test set and map the words/phrases in these questions to Wikipedia articles through title matching. In this way, we obtain a large labeled analogy test set over documents. The task is then to test whether different document representations over the Wikipedia articles can find the right answers to these semantic analogy questions.

Based on this test set, we evaluate several existing state-of-the-art document representations and show that neural embedding based models can achieve better performance than conventional models. The major contributions of this paper include: 1) the introduction of a new document analogy task with benchmark dataset for evaluating document representations; 2) empirical comparison among state-of-the-art models and preliminary explanations over the results.

2 Measuring Semantic Regularities

Relation Count Example
capital-common-countries 506 beijing: china paris: france
capital-world 3991 bangkok: thailand cario: egypt
currency 88 europe: euro india: rupee
city-in-state 277 houston: texas miami: florida
family 56 boy: girl man:woman
newspapers 20 chicago: chicago tribune houston: houston chronicle
ice hockey 462 boston: boston bruins los angeles: los angeles kings
basketball 306 chicago: chicago bulls dallas: dallas mavericks
airlines 306 canada: air canada italy: alitalia
people-companies 100 bill gates: microsoft larry page: google
Table 1: Details of the test set, where the words/phases in example refers to Wikipedia articles.

2.1 A Document Analogy Test Set

We propose to create a document analogy test set so that we can quantitatively evaluate how well different document representations capture semantic regularities. Following the idea of word analogy task, we try to build a test set of analogy questions of the form “ is to as is to     ”, where are the identities of the documents. However, it is not trivial to directly label the relations between two arbitrary documents due to the diversity in topics. Fortunately, we found that each Wikipedia page is a concise document describing one specific concept, and thus the relations between the documents can be explained by their corresponding concepts. Therefore, we can convert the task of labeling between documents into that between concepts (which are of words or phrases), where we already have a large labeled data set from Mikolov:Distributed.

Based on the idea above, we build a document analogy test set using Wikipedia and existing word and phrase analogy test set. Specifically, we adopt the publicly available April 2010 dump of Wikipedia111 [Shaoul and Westbury2010], which has been widely used in [Huang et al.2012, Luong et al.2013, Neelakantan et al.2014]. The corpus contains 3035070 articles and about billion tokens. We then collect all the existing word and phrase analogy test sets and match the words/phrases in questions to Wikipedia page titles. Note here we do not take syntactic analogy questions of words into consideration because the relations between documents are usually semantic. By resolving the ambiguity in matching, we finally obtain 6112 analogy questions over Wikipedia documents. Table 1 shows the details of the test set.

2.2 Analogy Reasoning

In this work, we adopt the same vector offset method [Mikolov et al.2013c] for analogy reasoning. To answer the questions like “a is to b as c is to     ”, we try to find a document with vector , which is the closest to

according to the cosine similarity:


where and are the normalized document vectors. The question is judged as correctly answered only if

is exactly the answer document in the evaluation set. The evaluation metric for this task is the percentage of questions answered correctly.

3 Models

In this section, we briefly summarize the models used in this paper. Before that, we first list the notations.

Let denote a corpus of documents over the word vocabulary . Let be a document-word matrix, where entry in denotes the weight of the -th word in the -th document .

Bag of Words (BOW) model treats a document as a bag (multiset) of its words. It represents a document as a vector , where denotes the weight of the -th word in the -th document . The most popular weighting scheme for is TF-IDF [Jones1972]

. However, The BOW model suffers from the sparsity and curse of dimensionality due to treating individual word as distinct feature.

Matrix Factorization methods attempt to tackle the limitation of BOW model through learning a low-dimensional vector for document by factorizing the document-word matrix .

Deerwester:Indexing applied truncated Singular Value Decomposition (SVD) to document-word matrix, namely Latent Semantic Indexing (LSI). LSI approximates

by setting all but the largest singular values in to (), as

Hence one might think of the rows of as representations for documents in the latent space.

An alternative way is factorizing into two non-negative matrices [Lee and Seung1999],

where the rows of can be seen as the representations of documents.

Unlike LSI which may have negative entries, NMF has better interpretability with the non-negative constraint.

Topic Models are also very popular in document representation fields because of their good interpretability, generalization ability and extensibility. The most representative work is the Latent Dirichlet Allocation (LDA) model introduced by Blei:Latent. It represents the documents as distributions over latent topics, where each topic is characterized by a distribution over words.

Neural Embedding models have attracted much attention in text representations due to its breakthrough in statistical language model [Bengio et al.2003].

The Paragraph Vector models are first introduced in [Le and Mikolov2014] for document representation. The Distributed Memory Model of Paragraph Vectors (PV-DM) captures the representation of a document via inserting a document vector to the continuous bag-of-words (CBOW) model [Mikolov et al.2013a]. A simpler model can be obtained by replacing the input word vector with document vector in Skip Gram (SG) model, which is called “Distributed Bag of Words version of Paragraph Vector” (PV-DBOW).

Bag of Word Embeddings (BOWE) model tries to represent the document as a linear combination of word vectors, where the word vectors can be obtained by tools like Word2Vec or GloVe. The low-dimensional representations of documents in BOWE can be written as

where denotes the BOW representation of documents.

4 Experiments

capital-common-countries 0.0 23.12 9.29 23.72 60.87 54.15 83.0
capital-world 0.8 9.97 5.06 9.15 43.62 42.65 67.53
currency 0.0 0.0 0.0 0.0 4.55 3.41 14.77
city-in-state 0.0 7.94 6.50 4.33 33.57 34.30 51.26
family 19.64 5.36 1.79 14.29 21.43 21.43 19.64
newspapers 5.0 25.0 10.0 10.0 5.0 50.0 40.0
ice hockey 0.0 3.68 2.16 0.0 12.12 20.13 33.33
basketball 0.0 4.25 1.63 0.0 10.13 14.71 38.56
airlines 11.76 9.15 2.94 2.61 12.42 20.26 42.48
people-companies 2.0 1.0 0.0 0.0 6.0 12.0 2.0
total 1.34 9.88 4.81 8.43 37.47 37.76 60.42
Table 2: Results on the document analogy task under dimension 100. Bold scores are the best.

In this section, we first describe our experimental settings including the corpus, hyper-parameter selections, and specifications for different document representation methods. Then we compare these methods on document analogy task and discuss the results.

4.1 Corpus and Preprocessing

The corpus used to learn document representations in this experiment is the same Wikipedia April 2010 dump as described in section 2.1. In preprocessing, we lowercase the corpus, remove pure digit words, non-English characters and the words occur less than 20 times.

4.2 Experimental Settings

The baseline methods used in this paper including BOW with TF-IDF weight, LSI, NMF, LDA, PV-DM, PV-DBOW, and BOWE. For BOW, LSI and LDA, we use the popular python topic model library gensim222

. For NMF, we choose the python machine learning library

scikit learn333 We implement PV-DM and PV-DBOW models in C++ due to Quoc:Distributed have not released source codes of PV models. For word embeddings in BOWE, we use CBOW in the Word2Vec tool444 The negative sampling method is adopted to take the place of the hierarchal softmax since we found the former always achieves better performance. The learning rate is linearly decayed to as described in [Mikolov et al.2013a], where the initial learning rate of PV-DM and CBOW model is 0.05, and PV-DBOW is 0.025. We set context window size as 10 and use 10 negative samples.

4.3 Results

In Table 2, we compare the results of 100-dimensional document vectors from all the methods on different subtasks of document analogy. As we can see, among all the methods, BOW is almost the worst. This demonstrates the weakness of simple vector space model on capturing semantic regularities.

Neural embedding models such as PV-DM and PV-DBOW perform much better than conventional latent models such as LSI, NMF, and LDA. This is quite amazing since PV models can also be viewed as implicit matrix factorization according to the explanations on Word2Vec [Levy and Goldberg2014a]. A major difference is that conventional latent models usually work on matrix with each entry standing for the frequency or TF-IDF of a word in a document, while PV models factorize a shifted pointwise mutual information (shifted-PMI) matrix. As discussed in [Arora et al.2015], PMI is a key factor why Word2Vec can work well for word analogy task. We guess this might also be a major factor to explain the gap between PV models and other latent models. Therefore, we conducted further experiments on LSI. As a result, we find that the total accuracy of LSI can achieve 15.53% with PMI matrix (about 57% performance gain over LSI with TF-IDF matrix). The results indicate that PMI plays an important role in revealing linear structure in document representations.

A surprising result is that the simple BOWE model performs significantly better than any other methods on almost all the subtasks (-value ). There might be two possible reasons for the result. Firstly, when learning word vectors alone with Word2Vec, one can achieve very high scores on word analogy tasks [Mikolov et al.2013c]. BOWE thus benefits from the strong linear structures in word vectors by directly using word vectors as the representation of a document. Secondly, the calculation of Euclidean distance555Note that dot product between normalized vectors in analogy reasoning is equivalent to Euclidean distance. between documents under BOWE is equivalent to using a relaxed Word Mover’s Distance [Kusner et al.2015], which has been shown strong performance in measuring document distance.

We also conduct the experiments on different dimensions as shown in Table 3. Similar trending can be found as that in Table 2.

Model 50 100 150 200
BOW 1.34 1.34 1.34 1.34
LSI 4.19 9.88 17.0 21.81
NMF 1.59 4.81 8.75 11.85
LDA 3.52 8.43 10.39 10.45
PV-DM 25.62 37.47 37.71 36.03
PV-DBOW 25.33 37.76 40.61 39.09
BOWE 42.05 60.42 66.74 69.49
Table 3: Results on the document analogy task under different dimensions.

5 Conclusion

In this paper, we introduce a new document analogy task for quantitatively evaluating how well different document representations capture semantic regularities. Based on the introduced benchmark dataset, we conduct empirical comparisons among several state-of-the-art document representation methods. The results reveal that neural embedding based document representations work better on this analogy task. We provide some preliminary explanations on these observations, leaving the inherent differences of these models to be further investigated in the future. With this benchmark dataset, it would also be easier for us to develop new document representation models and to compare with existing methods.


  • [Arora et al.2015] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2015. Random walks on context spaces: Towards an explanation of the mysteries of semantic word embeddings. CoRR, abs/1502.03520.
  • [Bengio et al.2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137–1155, March.
  • [Blei et al.2003] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022, March.
  • [Dai et al.2014] Andrew M Dai, Christopher Olah, Quoc V Le, and Greg S Corrado. 2014. Document embedding with paragraph vectors.

    NIPS Deep Learning Workshop

  • [Deerwester et al.1990] Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407.
  • [Dori-Hacohen et al.2015] Shiri Dori-Hacohen, Elad Yom-Tov, and James Allan. 2015. Navigating controversy as a complex search task. In Proceedings of the First International Workshop on Supporting Complex Search Tasks co-located with the 37th European Conference on Information Retrieval(ECIR 2015). Elsevier, March.
  • [Harris1954] Zellig Harris. 1954. Distributional structure. Word, 10(23):146–162.
  • [Huang et al.2012] Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL ’12, pages 873–882, Stroudsburg, PA, USA. Association for Computational Linguistics.
  • [Jones1972] Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–21.
  • [Kusner et al.2015] Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32th International Conference on Machine Learning (ICML-15), ICML ’15. ACM, New York, NY, USA, July.
  • [Le and Mikolov2014] Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Tony Jebara and Eric P. Xing, editors, Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1188–1196. JMLR Workshop and Conference Proceedings.
  • [Lee and Seung1999] Daniel D. Lee and H. Sebastian Seung. 1999. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788–791, october.
  • [Levy and Goldberg2014a] Omer Levy and Yoav Goldberg. 2014a. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems 27, pages 2177–2185. Curran Associates, Inc., Montreal, Quebec, Canada.
  • [Levy and Goldberg2014b] Omer Levy and Yoav Goldberg, 2014b. Proceedings of the Eighteenth Conference on Computational Natural Language Learning, chapter Linguistic Regularities in Sparse and Explicit Word Representations, pages 171–180. Association for Computational Linguistics.
  • [Luong et al.2013] Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013.

    Better word representations with recursive neural networks for morphology.

    In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 104–113. Association for Computational Linguistics.
  • [Mikolov et al.2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.

    Efficient estimation of word representations in vector space.

    In Proceedings of Workshop of ICLR.
  • [Mikolov et al.2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc.
  • [Mikolov et al.2013c] Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2013). Association for Computational Linguistics, May.
  • [Mnih and Kavukcuoglu2013] Andriy Mnih and Koray Kavukcuoglu. 2013.

    Learning word embeddings efficiently with noise-contrastive estimation.

    In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 2265–2273. Curran Associates, Inc.
  • [Neelakantan et al.2014] Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1059–1069, Doha, Qatar, October. Association for Computational Linguistics.
  • [Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543.
  • [Shaoul and Westbury2010] Cyrus Shaoul and Chris Westbury. 2010. The westbury lab wikipedia corpus. Edmonton, AB: University of Alberta.
  • [Srivastava et al.2013] Nitish Srivastava, Ruslan Salakhutdinov, and Geoffrey E. Hinton. 2013.

    Modeling documents with deep boltzmann machines.


    Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence

    , pages 616–625, Seattle, USA, August.