"The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders

06/18/2015 ∙ by Rémi Lebret, et al. ∙ 0

Recently, there has been a lot of effort to represent words in continuous vector spaces. Those representations have been shown to capture both semantic and syntactic information about words. However, distributed representations of phrases remain a challenge. We introduce a novel model that jointly learns word vector representations and their summation. Word representations are learnt using the word co-occurrence statistical information. To embed sequences of words (i.e. phrases) with different sizes into a common semantic space, we propose to average word vector representations. In contrast with previous methods which reported a posteriori some compositionality aspects by simple summation, we simultaneously train words to sum, while keeping the maximum information from the original vectors. We evaluate the quality of the word representations on several classical word evaluation tasks, and we introduce a novel task to evaluate the quality of the phrase representations. While our distributed representations compete with other methods of learning word representations on word evaluations, we show that they give better performance on the phrase evaluation. Such representations of phrases could be interesting for many tasks in natural language processing.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Human language “makes infinite use of finite means” (Humboldt, 1836). A large number of sentences can be generated from a finite set of words. Thus there has been a lot of effort to capture the meaning of words. Some approaches are based on distributional word representations (Lund & Burgess, 1996; Patel et al., 1998), others are based on distributed representations (Bengio, 2008; Collobert et al., 2011; Mnih & Kavukcuoglu, 2013; Mikolov et al., 2013b; Lebret & Collobert, 2014; Pennington et al., 2014; Levy & Goldberg, 2014) where the meaning of a word is encoded as a vector computed from co-occurrence statistics of a word and its neighboring words. Finally, distributed representations emerged as the solution to many natural language processing (NLP) tasks (Turney & Pantel, 2010; Collobert et al., 2011).

Given these representations of words in a vector space, techniques for combining them have been proposed to get representations of phrases or sentences. These compositional models involve vector addition or multiplication (Mitchell & Lapata, 2010). Such simple compositions have shown to perform competitively on the paraphrase detection and phrase similarity tasks (Blacoe & Lapata, 2012). More sophisticated approaches use techniques from logic, category theory, and quantum information (Clark et al., 2008). Others use the syntactic relations between words to treat certain words as functions and other as arguments such as adjective-noun composition (Baroni & Zamparelli, 2010) or noun-verb composition (Grefenstette et al., 2013)

. Recursive neural network model for semantic compositionality has also been proposed 

(Socher et al., 2012), where each word has a matrix-vector representation: the vector captures its meaning (as it is initialized with a pre-trained distributed representation), while the matrix learns throught a parse tree how it modifies the meaning of the other word that it combines with. Many recent works are based on distributed representations of phrases to tackle a wide range of application in NLP: machine translation (Bahdanau et al., 2015), constituency parsing (Legrand & Collobert, 2015)

, sentiment analysis 

(Socher et al., 2013), or image captioning (Lebret et al., 2015). There is therefore a clear need for distributed word representations that can be easily extrapolated to meaningful phrase representations.

We argue that distributed representation and composition must go hand in hand, i.e., they must be mutually learned. We present a model that learns to capture meaning of words in distributed representations using a low-rank approximation of a large word co-occurrence matrix. We choose to stochastically perform this low-rank approximation which enables the model to simultaneously train these representations to compose for producing representations of phrases (see Figure 1). As composition function, we choose a simple weighted addition for its simplicity and for enabling sequences of words with different lengths to be representated in a common vector space. Aside from generating distributed representations of words and phrases, this model gives an encoding function (represented by a matrix) which can be used to encode new words or even phrases based on their co-occurrence counts. This offers two different alternatives for phrase representations: (1) representation for a query phrase can be inferred by averaging vector representations of its words (only if they all were in the training set), or (2) by using its word co-occurrence statistics.

Evaluation on the popular word similarity and analogy tasks demonstrate the capability of our joint model for capturing good distributed representations. We then introduce a novel task for evaluating phrase representations. Given a phrase representation, the objective is to retrieve the words that compose the phrase. We compare our model against other state-of-the-art methods for distributed word representations which capture meaningful linear substructures (Mikolov et al., 2013a; Pennington et al., 2014). We show that our model achieves similar performance on word evaluation tasks, but that it outperforms other methods on the phrase evaluation task.

2 Related Works

In the literature, two major model families exist for learning distributed word representations: the count-based methods and the predictive-based methods.

The count-based methods consist of using the statistical information contained in large corpora of unlabeled text to build large matrices by simply counting words (word co-coocurrence statistics). The rows correspond to words or terms, and the columns correspond to a local context. The context can be documents, such as in latent semantic analysis (LSA) (Deerwester et al., 1990); or other words (Lund & Burgess, 1996)

. To generate low-dimensional word representations, a low-rank approximation of these large matrices is performed, mainly with a singular value decomposition (SVD). Many authors proposed to improve this model with different transformations for the matrix of counts, such as positive pointwise mutual information (PPMI) 

(Bullinaria & Levy, 2007; Levy & Goldberg, 2014)

, or a square root of the co-occurrence probabilities in the form of Hellinger PCA 

(Lebret & Collobert, 2014). Instead of using the co-occurrence probabilities, (Pennington et al., 2014) suggest that word vector representations should be learnt with ratios of co-occurrence probabilities. For this purpose, they introduce a log-bilinear regression model that combines both global matrix factorization and local context window methods.

The predictive-based model has first been introduced as a neural probabilistic language model (Bengio et al., 2003). A neural network architecture is trained to predict the next word given a window of preceding words, where words are representated by low-dimensional vector. Since, some variations of this architecture have been proposed. (Collobert et al., 2011) train a language model to discriminate a two-class classification task: if the word in the middle of the input window is related to its context or not. More recently, the need of full neural architectures has been questioned (Mnih & Kavukcuoglu, 2013; Mikolov et al., 2013a). Mikolov et al. (2013a

) propose two predictive-based log-linear models for learning distributed representations of words: (i) the continous bag-of-words model (CBOW), where the objective is to correctly classify the current (middle) word given a symmetric window of context words around it; (ii) the skip-gram model, where instead of predicting the current word based on the context, it tries to maximize classification of a word based on another word in the same sentence. In 

Mikolov et al. (2013b), the authors also introduce a data-driven approach for learning phrases, where the phrases are treated as individual tokens during the training.

In this paper, we leverage both families: (i) we use the statistical information for learning distributed word representations by approximating the Hellinger PCA with an autoencoder network; (ii) we jointly learn to predict the words that compose a given phrase.

3 A Joint Model

Some prior works have designed models to learn word representations (Mnih & Kavukcuoglu, 2013; Mikolov et al., 2013b; Lebret & Collobert, 2014), while others have proposed models to compose these word representations (Mitchell & Lapata, 2010; Socher et al., 2012). We propose instead to jointly learn word representations and their composition by simple summation.

3.1 Learning Word Representations w.r.t. the Hellinger Distance

As words occurring in similar contexts tend to have similar meanings (Harris, 1954), word co-occurrence statistics are generally used to embed similar words into a common vector space (Turney & Pantel, 2010). Common approaches calculate the frequencies, apply some transformations (tf-idf, PPMI), reduce the dimensionality, and calculate the similarities. More recently, Lebret & Collobert (2014) proposed a novel method based on a Hellinger PCA of the word co-occurrence matrix. They showed that word representations can be learnt even with a reasonable number of context words. Inspired by this work, we propose to stochastically perform this low-rank approximation. For this purpose, we use an autoencoder with only linear activations to find an optimal solution related to the Hellinger PCA (Bourlard & Kamp, 1988). Replacing the PCA by an autoencoder allows us to learn jointly a cost function which constrains the word information to be kept by summation.

3.1.1 Word Co-Occurrence Probabilities

“You shall know a word by the company it keeps” (Firth, 1957). Keeping this famous quote in mind, word co-occurrence probabilities are computed by counting the number of times each context word (where ) occurs around a word :


where is the number of times a context word occurs in the surrounding of the word . A multinomial distribution of classes (words) is thus obtained for each word :


3.1.2 Hellinger Distance

Similarities between words can be derived by computing a distance between their corresponding word distributions. Several distances (or metrics) over discrete distributions exist, such as the Bhattacharyya distance, the Hellinger distance or Kullback-Leibler divergence. We chose here the Hellinger distance for its simplicity and symmetry property (as it is a true distance). Considering two discrete probability distributions

and , the Hellinger distance is formally defined as:


which is directly related to the Euclidean norm of the difference of the square root vectors:


Note that it makes more sense to take the Hellinger distance rather than the Euclidean distance for comparing discrete distributions, as and are unit vectors according to the Hellinger distance ( and are units vector according to the norm).

3.1.3 Autoencoder

An autoencoder is employed to represent words in a lower dimensional space. It takes a distribution as input, encodes it in a more compact representation, and is trained to reconstruct its own input from that representation:


where is the output of the network, is the encoding function which maps distributions in a -dimension (with ), and is the decoding function. is a distributed representation that captures the main factors of variation in the data as the Hellinger PCA does (Bourlard & Kamp, 1988). Here, encoder and decoder are both linear layers.

3.2 Learning to Sum Word Representations

Interesting compositionality properties have been observed from models based on the addition of representations (Mikolov et al., 2013b). An exhaustive comparison of different composition functions has indeed revealed that an additive model performs well on pre-trained word representations (Mitchell & Lapata, 2010)

. Because our word representations are learnt from linear operations, the inherent structure of these representations is linear. To combine a sequence of words into a common vector space, we then simply apply an element-wise addition of their vector representations. This approach makes sense and works well when the meaning of a text is literally “the sum of its parts”. This is usually the case with noun and verb phrase chunks. For example, into phrases such as “the red cat” or “struggle to deal”, each word independently has its proper meaning. Distributed representations for such phrase chunks must retain information from the individual words. An objective function is thus defined to learn how to combine the word vector representations, while keeping the maximum information from the original vectors. An operation as simple as a weighted sum will probably fail for sequences where individual words act as operators that modify the meaning of another word, or for multiword expressions. Other more complex functions could be chosen to also include such cases, but we choose to propose a much simpler model (i.e., averaging the word representations) to get phrase chunk representations with unsupervised learning. In this paper, we therefore focus on noun and verb phrase chunks.

3.2.1 Additive Model

We define a phrase chunk of words, with a set of phrase chunks. By feeding all into the autoencoder, a representation of each word is obtained:


By an element-wise addition, a representation of the phrase chunk can be calculated as:

Figure 1: Architecture for the joint learning of word representations and their summation. Considering the noun phrase the red cat, each word is represented as the square root of its co-occurrence probability distribution . These are the inputs given to an autoencoder which encodes them in a lower dimension . These new representations are then given to a decoder which is trained to reconstruct the initial inputs. This is the first objective function. The second objective is to keep information when words are summed. All are summed together to represent in the same space as . A dot product between the phrase representation and all the other word representations from the dictionary is calculated. These scores are trained to be high for words that appear in and low for the others.

3.2.2 Training

In predictive-based model, such as the Skip-gram model, the objective is to maximize the likelihood of a word based on other words in the same sequence. Instead, our training is slightly different in the sense that we aim at discriminating whether words are in the phrase chunk or not. An objective function is thus defined to encourage words which appear in the chunk to give high scores when calculating the dot product between and . On the other hand, these scores must be low for words that do not appear in the chunk. We train this problem with a ranking-type cost:


Note that due to the large size of , a negative sampling approach can be used to speed up the training. In Equation 8, the whole dictionary is thus replaced by a subset with randomly chosen negative samples . A new set is randomly picked at each iteration during the training.

3.3 Joint Learning

In contrast with other methods which have subsequently found nice compositionality properties by simple summation, the novelty of our method is the explicit learning of word representations suitable for summation. The system is then designed to force words with similar context to be close in a -dimensional space, while these dimensions are learnt to be combined with other related words. This joint learning is illustrated in Figure 1. The whole system is trained by minimizing both objective functions (5) and (8

) over the training data using stochastic gradient descent.

4 Experiments

4.1 Datasets

4.1.1 Building Word Representation over Large Corpora

Our English corpus is composed of the entire English Wikipedia111Available at http://download.wikimedia.org. We took the January 2014 version. (where all MediaWiki markups have been removed). We consider lower case words to limit the number of words in the dictionary. Additionally, all occurrences of sequences of numbers within a word are replaced with the string “NUMBER”. The resulting text is tokenized using the Stanford tokenizer222Available at http://nlp.stanford.edu/software/tokenizer.shtml. The data set contains about 1.6 billion words. As dictionary , we consider all the words within our corpus which appear at least one hundred times. This results in a 191,268 words dictionary. Only the 10,000 most frequent words within this dictionary were used as context words to calculate the word co-occurrence probabilities. A symmetric context window of ten words around each word is used to obtain the multinomial distribution . We chose to encode words in a 100-dimensional vector.

4.1.2 Summing Words for Phrase Representation

To learn the summation of words that appear frequently together, we choose to consider only the noun and verb phrase chunks to build . We extract these chunks with a phrase chunking approach by using the SENNA software333Available at http://ml.nec-labs.com/senna/. By retaining only the phrase chunks appearing at least ten times, this results in 1,823,259 noun phrase chunks and 255,232 verb phrase chunks, for a total of 2,078,491 phrase chunks. We divided this set of phrases into three sets: 1,000 phrases for validation, 5,000 phrases for testing, and the rest for training (2,072,491 phrases). An unsupervised framework requires a large amount of data. Because our primary focus is to provide good word representations, validation and testing sets are intentionally kept small to retain as much phrases as possible in the training set.

4.2 Other Methods

We compare our distributed representations with other available models for computing vector representations of words: (1) the GloVe model which is also based on co-occurrence statistics of corpora (Pennington et al., 2014)444Code available at http://www-nlp.stanford.edu/software/glove.tar.gz., (2) the continuous bag-of-words (CBOW) and the skip-gram (SG) architectures which learn representations from prediction-based models (Mikolov et al., 2013b)555Code available at http://word2vec.googlecode.com/svn/trunk/.. The same corpus and dictionary as the ones described in Section 4.1.1

are used to train 100-dimensional word vector representations. We use a symmetric context window of ten words, and the default values set by the authors for the other hyperparameters. To see the improvement compared to a standalone SVD, we generate word representations with a truncated SVD of the matrix

, where each row of is a distribution , .

4.3 Evaluating Word Representations

The first objective of the model is to learn distributed representations which capture both syntactic and semantic informations about words. To evaluate the quality of these representations, we used both analogy and similarity tasks.

4.3.1 Word analogies

The word analogy task consists of questions like, “a is to b as c is to ?”. It was introduced in Mikolov et al. (2013a) and contains 19,544 such questions, divided into a semantic subset and a syntactic subset. The 8,869 semantic questions are analogies about places, like “Bern is to Switzerland as Paris is to ?”, or family relationship, like “uncle is to aunt as boy is to ?”. The 10,675 syntactic questions are grammatical analogies, involving plural and adjectives forms, superlatives, verb tenses, etc. To correctly answer the question, the model should uniquely identify the missing term, with only an exact correspondence counted as a correct match.

4.3.2 Word Similarities

We also evaluate our model on a variety of word similarity tasks. These include the WordSimilarity-353 Test Collection (WS-353) (Finkelstein et al., 2001), the Rubenstein and Goodenough dataset (RG-65) (Rubenstein & Goodenough, 1965), and the Stanford Rare Word (RW) (Luong et al., 2013). They all contain sets of English word pairs along with human-assigned similarity judgements. WS-353 and RG-65 datasets contain 353 and 65 word pairs respectively. Those are relatively common word pairs, like computer:internet or football:tennis. The RW dataset differs from these two datasets, since it contains 2,034 pairs where one of the word is rare or morphologically complex, such as brigadier:general or cognizance:knowing.

4.3.3 Results

CBOW 0.57 0.47 0.32 53.5 22.7
Skip-gram 0.62 0.49 0.39 66.7 53.8
GloVe 0.56 0.50 0.36 79.7 75.0
SVD 0.43 0.39 0.27 52.3 34.1
Our model 0.62 0.49 0.39 69.4 43.0
Table 1: Word representations evaluation on both similarity and analogy tasks. Comparison of performance across all models with 100-dimensional word vector representations. For all models, a symmetric context window of ten words is used. Spearman rank correlation is reported on word similarity tasks. Accuracy is reported on word analogy tasks.

Results reported in Table 1 show that our model gives similar results than other state-of-the-art methods on word similarity tasks. However, there is a significant performance boost between the low-rank approximation of with a SVD and this same approximation with our joint model. This shows that combining a count-based model with a predictive-based approach helps for generating better word representations. Performance on word analogy tasks show that our joint model competes with others on the syntactic questions, but that it gives a lower accuracy on semantic questions. One possible explanation is that less common words are involved in semantic questions compared to syntactic questions. Among the four words that make a semantic question, one of them is, in average, the most frequent word in , while it is the for a syntactic question. Compared to other methods which take the whole dictionary as context dictionary, we consider only a small subset of it ( contains only the most frequent words of ). A larger context dictionary would certainly help to improve performance on this task666It has not been explored due to limitations in hardware resources. It would be easily computable with a cluster of CPU..

4.4 Evaluating Phrase Representations

As a second objective, we aim at learning to sum word representations to generate phrase representations while keeping the original information coming from the words. We thus introduce a novel task to evaluate the phrase representations.

4.4.1 Description of the Task

As dataset, we use the collection of test phrases described in Section 4.1.2. It contains 5000 phrases (noun phrases and verb phrases) extracted from Wikipedia with a chunking approach. Amoung them, 2244, 2030 and 547 are, respectively, composed of two, three and four words. The remaining 179 are composed of at least five words with a maximum of eight words. For a given phrase of words, the objective is to retrieve the words from its distributed representation . Scores between the phrase and all the possible words are calculated using the dot product between their distributed representations , as illustrated in Figure 1. The top scores are considered as the words composing the phrase .

4.4.2 Results

Figure 2: Recall@1 based on the number of words per phrases. Comparison of performance across all models with 100-dimensional word vector representations.

To evaluate whether words making a given phrase can be retrieved from the distributed phrase representation, we use Recall @, which measures the fraction of times a correct word was found among the top results. is proportional to the number of words per phrase, e.g. for a 3-word phrase with a Recall@5, the correct words are found amoung the top 15 results. Higher Recall @ means better retrieval performance. Since we care most about the top-ranked retrieved results, the Recall @ with small are more important.

R@1 R@5 R@10
CBOW 11.33 29.56 38.46
Skip-gram 7.96 22.26 30.04
GloVe 54.97 79.97 86.54
SVD 17.42 32.87 40.72
Our model 64.22 91.72 95.85
Table 2: Phrase representations evaluation. Comparison of performance across all models with 100-dimensional phrase vector representations on word retrieval. R@ is Recall@, with .
Encoding Function Averaging Words
american airlines braniff airlines american airways
aloha airlines pan american airlines
braniff airways american eagle airlines
jetblue airways north american airlines
braniff international airways american overseas airlines
chicago bulls denver nuggets chicago colts
seattle supersonics chicago hornets
cleveland cavaliers chicago stags
boston celtics buffalo bulls
dallas mavericks chicago cardinals
home plate right fielder the home plate umpire
center fielders the home plate area
the outfield fence the home leg
leadoff batter the ball home
the infield the diamond state base ball club
president of the united states president coolidge the united states president
president eisenhower the united states presidency
u.s. president dwight eisenhower the first united states secretary
president truman the united states minister
president reagan the first united states senator
Table 3: Examples of phrases and five of their ten nearest phrases from the collection of phrases. Representations for the collection of phrases have been computed by averaging the word representations. Query phrase representations are inferred using the two different alternatives: (1) with the encoding function using counts from a symmetric window of ten context words around the query phrase, (2) by averaging the representations of the words that compose the query phrase. All distributed representations are 100-dimensional vectors.

Results reported in Table 2 show that our distributed word representations can be averaged together to produce meaningful phrase representations, since the words are retrieved with a high recall. Our model significantly outperforms others methods on this task. In Figure 2, a more detailed analysis of results reveals that the GloVe model competes with ours for the 2-word phrases. However GloVe’s representations cannot maintain this performance for longer phrases. It is probably not too surprising as this model is trained using ratios of co-occurrence probabilities for two target words. Consequently, it well learns linear substructures for pairs of words, which probably also explains its good performance on word analogy tasks. In contrast, our joint model can learn more complex substructures which make possible the aggregation of multiple words within a low-dimensional vector space.

4.5 Inferring New Phrase Representations

Representations for new phrases can thus be generated by simply averaging its word representations, assuming that the words are in the dictionary . Consider that the dictionary tends to grow exponentially with , it gives a nice framework to produce the huge variety of possible sequence of words in a timely and efficient manner with low memory consumption, unlike other methods. Relying on word co-occurrence statistics to represent words in vector space also provides a framework to easily generate representations for unseen words. This is another advantage compared to methods focused on learning distributed word representations (such as CBOW, Skip-gram and GloVe models), where the whole system needs to be trained again to learn representations for these new constituents. To infer a representation for a new word , one only needs to count its context words over a large corpus of text to build the distribution . This nice feature can be extrapolated to phrases, which gives another alternative for generating phrase representations. Table 3 presents some examples of phrases, where we use both alternatives to compute their distributed representations. It can be seen that both alternatives give distinct representations. For instance, by using the encoding function , our model infers a representation for the entity Chicago Bulls which is close to other NBA teams, like the Denver Nuggets or the Seattle Supersonics. By averaging the representations of both words Chicago and Bulls, our model infers a representation which is close to other Chicago’s sport teams. Both representations are meaningful, but they carry different information. Relying on co-occurrence statistics gives entities that occur in a similar context, while the summation tries to find entities containing the maximum amount of similar information. This also works with longer phrases, such as President of the United States. The first alternative gives men who served as president, when the second gives related positions.

5 Conclusion

We introduce a model that combines both count-based methods and predictive-based methods for generating distributed representations of words and phrases. Using a chunking approach, a collection of noun phrases and verb phrases is extracted from Wikipedia. For a given -word phrase, we train our model to generate a low-dimensional representation for each word based on its co-occurrence probability distribution. These representations are averaged together to generate a distributed phrase representation in the same semantic space. Thanks to an autoencoder approach, we can simultaneously train the model to retrieve the original words from the phrase representation, and therefore learn complex linear substructures. When compared to state-of-the-art methods on some classical word evaluation tasks, the competitive results show that our joint model produces meaningful word representations. Performance on a novel task for evaluating phrase representations confirm the ability of our model to learn complex substructures, which make possible the aggregation of multiple words within a low-dimensional vector space. Better still, inference of new phrase representations is also easily feasible when relying on counts. Some quantitative examples demonstrate that both alternatives can give different but meaningful information about phrases. The word representations and the collection of phrases used in these experiments are available online, here: http://www.lebret.ch/words/.


This work was supported by the HASLER foundation through the grant “Information and Communication Technology for a Better World 2020” (SmartWorld).


  • Bahdanau et al. (2015) Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3th International Conference on Learning Representations (ICLR), 2015.
  • Baroni & Zamparelli (2010) Baroni, M. and Zamparelli, R. Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space. In Proceedings of the EMNLP, pp. 1183–1193, 2010.
  • Bengio (2008) Bengio, Y. Neural net language models. Scholarpedia, 3(1):3881, 2008.
  • Bengio et al. (2003) Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. A neural probabilistic language model.

    The Journal of Machine Learning Research

    , 3:1137–1155, 2003.
  • Blacoe & Lapata (2012) Blacoe, W. and Lapata, M. A Comparison of Vector-based Representations for Semantic Composition. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pp. 546–556. Association for Computational Linguistics, 2012.
  • Bourlard & Kamp (1988) Bourlard, H. and Kamp, Y.

    Auto-association by multilayer perceptrons and singular value decomposition.

    Biological cybernetics, 59(4-5):291–294, 1988.
  • Bullinaria & Levy (2007) Bullinaria, J. A. and Levy, J. P. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39(3):510–526, 2007.
  • Clark et al. (2008) Clark, S., Coecke, B., and Sadrzadeh, M. A compositional distributional model of meaning. In Proceedings of the Second Quantum Interaction Symposium (QI-2008), pp. 133–140, 2008.
  • Collobert et al. (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. Natural Language Processing (Almost) from Scratch. The Journal of Machine Learning Research, 12:2493–2537, 2011.
  • Deerwester et al. (1990) Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407, 1990.
  • Finkelstein et al. (2001) Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., and Ruppin, E. Placing Search in Context: The Concept Revisited. In Proceedings of the 10th international conference on World Wide Web, pp. 406–414. ACM, 2001.
  • Firth (1957) Firth, J. R. A Synopsis of Linguistic Theory 1930-55. 1957.
  • Grefenstette et al. (2013) Grefenstette, E., Dinu, G., Zhang, Y., Sadrzadeh, M., and Baroni, M. Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers, chapter Multi-Step Regression Learning for Compositional Distributional Semantics, pp. 131–142. Association for Computational Linguistics, 2013.
  • Harris (1954) Harris, Z. S. Distributional structure. Word, 1954.
  • Humboldt (1836) Humboldt, W. Über die Verschiedenheit des menschlichen Sprachbaues: Und ihren Einfluss auf die geistige Entwickelung des Menschengeschlechts. Druckerei der Königlichen Akademie der Wissenschaften, 1836.
  • Lebret & Collobert (2014) Lebret, R. and Collobert, R. Word Embeddings through Hellinger PCA. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 482–490. Association for Computational Linguistics, April 2014.
  • Lebret et al. (2015) Lebret, R., Pinheiro, P. H. O., and Collobert, R. Phrase-based image captioning. In Proceedings of the 32st International Conference on Machine Learning (ICML), 2015.
  • Legrand & Collobert (2015) Legrand, J. and Collobert, R. Joint rnn-based greedy parsing and word composition. In Proceedings of the 3th International Conference on Learning Representations (ICLR), 2015.
  • Levy & Goldberg (2014) Levy, O. and Goldberg, Y. Neural Word Embedding as Implicit Matrix Factorization. In Advances in Neural Information Processing Systems, pp. 2177–2185. 2014.
  • Lund & Burgess (1996) Lund, K. and Burgess, C. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2):203–208, 1996.
  • Luong et al. (2013) Luong, M., Socher, R., and Manning, C. D. Better Word Representations with Recursive Neural Networks for Morphology. In CoNLL, 2013.
  • Mikolov et al. (2013a) Mikolov, T., Chen, K., Corrado, G., and Dean, J.

    Efficient Estimation of Word Representations in Vector Space.

    In Proceedings of Workshop at International Conference on Learning Representations (ICLR 2013), 2013a.
  • Mikolov et al. (2013b) Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, pp. 3111–3119. 2013b.
  • Mitchell & Lapata (2010) Mitchell, J. and Lapata, M. Composition in Distributional Models of Semantics. Cognitive Science, 34(8):1388–1429, 2010.
  • Mnih & Kavukcuoglu (2013) Mnih, A. and Kavukcuoglu, K.

    Learning word embeddings efficiently with noise-contrastive estimation.

    In Advances in Neural Information Processing Systems, pp. 2265–2273, 2013.
  • Patel et al. (1998) Patel, M., Bullinaria, J. A., and Levy, J. P. Extracting Semantic Representations from Large Text Corpora. In 4th Neural Computation and Psychology Workshop, London, 9–11 April 1997, pp. 199–212. Springer, 1998.
  • Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C. D. GloVe: Global Vectors for Word Representation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), volume 12, 2014.
  • Rubenstein & Goodenough (1965) Rubenstein, H. and Goodenough, J. B. Contextual Correlates of Synonymy. Communications of the ACM, 8(10):627–633, 1965.
  • Socher et al. (2012) Socher, R., Huval, B., Manning, C., and Ng, A. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1201–1211. Association for Computational Linguistics, 2012.
  • Socher et al. (2013) Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y, and Potts, C. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), volume 1631, pp. 1642. Association for Computational Linguistics, 2013.
  • Turney & Pantel (2010) Turney, P. and Pantel, P. From Frequency to Meaning: Vector Space Models of Semantics.

    Journal of Artificial Intelligence Research

    , 37(1):141–188, 2010.