English Out-of-Vocabulary Lexical Evaluation Task

04/11/2018 ∙ by Han Wang, et al. ∙ Texas A&M University 0

Unlike previous unknown nouns tagging task (Curran, 2005) (Ciaramita and Johnson, 2003), this is the first attempt to focus on out-of- vocabulary(OOV) lexical evaluation tasks that does not require any prior knowledge. The OOV words are words that only appear in test samples. The goal of tasks is to pro- vide solutions for OOV lexical classification and predication. The tasks require annotators to conclude the attributes of the OOV words based on their related contexts. Then, we uti- lize unsupervised word embedding methods such as Word2Vec(Mikolov et al., 2013) and Word2GM (Athiwaratkun and Wilson, 2017) to perform the baseline experiments on the cat- egorical classification task and OOV words at- tribute prediction tasks.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

The evolution of modern English language brings new words in and eliminates old words out. Thus out-of-vocabulary (OOV) handling is an inevitable challenge among nearly all natural language processing topics: In cross-lingual translation, the quality of translation is heavily dependent on the identification of OOV words. In speech-to-text translation, detecting and modelling the OOV words can significantly improve the lexical completeness of the input. In semantic analysis, a majority of OOV words are proper names (PN) which are important for discovering the contextual concept relatedness. In social network sentiment analysis, new words are created every day on the Internet, thus the ability of handling OOV words can also contribute to the robustness of the model. Moreover, the concept of OOV handling is closely related with the emerging ”zero shot learning (ZSL)”


in other machine learning studies.

Therefore, it is indispensable to investigate the metrics of evaluating the OOV classifications and predictions. Although in NLP study there are various semantic evaluation tasks for computational semantic analysis such as [6, 7, 8, 9, 10], the majority of current evaluation tasks focus on word sense disambiguation and text simplification. [11, 9] describes their work to capture major semantic inferences across applications. Finding a valid substitution with the given context has been proven to be effective in question answering, abstract extracting, etc. The task proposed in this paper concentrates more on the dominant OOV semantic prediction. The main difference between the other semantic simplification tasks is that OOV words may not impede the performance in their tasks, while in our proposed task, the outcome is directly related with the OOV words.

In previous unknown nouns supersense tagging, [1] and [2], rule-based models were proposed within the scope of WordNet. The main limitations are:

  • The models are heavily dependent on some particular features of the unknown words such as the part of speech tagging and chunking, morphological analysis and even gramatical relation extraction.

  • Supersense method is based on WordNet hierarchy, called lexicographer files, which contains redundant prior knowledge.

  • These models cannot handle words that do not exist in WordNet. However, the vocabulary of Wikipedia we used is ten times larger than that of WordNet.

The task proposed in this paper jumps out from the above limitations. First, both training and testing data we utilized are from Wikipedia, which require no prior knowledge. Second, the baseline experiments are purely unsupervised and are based on vector-space rather than predefined rules.

Word Sense Disambiguation(WSD) is to identify the meanings of ambiguous words. We apply WSD in our baseline experiments and compare it with non-WSD models.

Ii Related Work


proposed supersense to classify common nouns that extends named entity classification. Based on supersense,

[1] improved the rule-based model by adding hand-coded unseen nouns tagging within the scope of WordNet. [12] built the text simplification system closely related with WordNet to obtain the word pairs as hypernym or synonym. [7] compiled the corpus based on the lexical substitution at SemEval-2007 with manually annotation. However, OOV prediction task is similar with lexical simplification whilst the difference is obvious. First, it is required to acquire prior knowledge as much as better for text simplification. Few OOV words in our task exist in WordNet, not to mention the creation of the thesaurus. With a never-seen word, the key to understand it is to familiarize with the given context. Second, most of the test words and the candidates in lexical substitution tasks such as [6]are daily words. Thus, the demand for comprehension related to the whole context is less essential than ours.

OOV prediction obstructs the text representation because learning representation in word is the cornerstone for the further text understanding in human sense. In the past, one-hot encoding

[13] has been used for the word indexing because of the simplicity. Nevertheless, the limitation of the one-hot encoding is apparent. This algorithm produced a sparse matrix for each word representation, but the computational power is much weak, which means it is impossible for the applications with big data. Besides, this method cannot extract the relationship between words, let alone the sentences. Compared with one-hot encoding, word embedding [14][15] utilizes semantic and syntactic information, which can extract more relationship than one-hot encoding.

However, word embedding also has been constrained by the poor hardware development in previous decades and the algorithm’s high time complexity. Recently, [3] proposed two models (skip-gram and continuous bag-of-words) named Word2Vec for effective learning representation. In addition, those methods succeed to represent word from sparse vectors into dense vectors, which favored for next training or clustering. [16] compared the detailed performance between sparse and dense vectors in short text classification.

Those models mentioned above suffers a common weakness, each word is mapped to a unique vector. In fact, numerous words can represent multiple senses in different conversations. Consequently, [17] proposed multi-prototype model for the polysemy. After that, other related models [18, 19, 20, 4, 21]

have been inspired such as probabilistic model, Gaussian and Gaussian mixture models and so on.

Iii Task Statement

OOV handling in natural language processing is still a sophisticated task due to several reasons. Lack of definition or explanation about the particular OOV word is one of the obstacles to accurately predict the meanings of OOV words. Another aspect is, newly created words usually have a comparably lower frequency than normal words, and sometimes they are eliminated when the minimal count is performed in preprocessing, Therefore, recognition of OOV highly depends on the quality of corpora. Given the fact that no corpus in use is the latest real-world corpus, it is essentially helpful to take the OOV classification and prediction capacity into consideration as a robust metric of NLP models. We hence propose our OOV classification and prediction tasks. Each task contains three parts, an OOV word, context and attributes. The given context is as close as a description to the OOV word and it can be either long or short regarding the rarity and difficulty of the OOV word. We assigned several attributes to each of the OOV word based on the corresponding context. The attributes can be the topic of the context, the potential or prior characteristics about the OOV word. Attributes are provided by independent annotators. We invited five individuals in USA, they are pursuing master or PhD degrees and are all fluent but non-native English speakers. Each context is concluded by three random annotators and we intersect the common attributes.

Another contribution in this paper is that we introduced a category classification task of OOV words with context. When we analyzed the rare words in Wikipedia English corpus, we found that many of the words are chemical and medical science terminologies with obscure contexts, which could not be comprehensively understood by human annotators. In order to make the proposed task more generic, we discarded those contexts, and hand-picked 5 categories: Greek mythology, locations, animals, plants and technology.

Iii-a Data source

The corpus was collected from Wikipedia dumps
(https://dumps.wikimedia.org/backup-index.html). The raw XML format corpus contains various types of pre-defined categories, each category has an unambiguous structure tree with its children nodes representing the sub-categories, which is similar to WordNet (https://wordnet.princeton.edu/). Wikipedia category tree has much more nodes compared with WordNet. The majority of the selected OOV words actually do not exist in WordNet.

Iii-B Data selection

Regarding the rules of OOV sampling, firstly words with both too high and too low occurrence were eliminated because we need to limit the data size and those eliminated words are usually less informative, which can be treated as stopwords. A reasonable data size is vital in terms of performance and speed optimization for both training and testing. Secondly, an OOV word is restricted to appear only in given context, and this is also our condition to retrieve such an OOV. An advantage to utilize Wikipedia as the data source is its sufficient entries satisfying the requirements.

After the occurrence filtering, there yet exists numerous OOV candidates. Thus we only select some particular but convictive categories. As mentioned above, according to statistical analysis, those five categories contain abundant OOV words and their given contexts are readable. Oppositely, in the category of medicine and chemistry, the context is not readily comprehensible for our annotators.

The last step for data selection is the attribute extraction. Each OOV word and its context is designated to three random annotators. After the individual conclusion, an attribute intersection is performed to acquire final results.


Word: arachis


Arachis is a genus of about 70 species of annual and perennial flowering plants in the pea family (Fabaceae), native to South America, and was recently assigned to the informal monophyletic Pterocarpus clade of the Dalbergieae.

This OOV’s category is plant, while the attributes are pea, flower and fabaceae, which can all be found in context.

Word: winwebsec


Winwebsec is a category of malware that targets the users of Windows operating systems and produces fake claims as genuine anti-malware software, then demands payment to provide fixes to fictitious problems.

This OOV’s category is technology, while its attributes are malware, adware, spyware, and the context contains only malware.

The whole data set for the two tasks can be accessed from (https://github.com/hwangtamu/OOVLexical)

Iii-C Task 1

The first task is to test if the positive examples are correctly classified and the negative examples are correctly excluded from their corresponding categories. More specifically, the ground truth is set as the higher level Wikipedia category names from which the entries were taken.


where represents rank of a correct prediction and is the total number of test samples.

Iii-D Task 2

The second task is to test if top semantic predictions of the OOV words hit the human annotated attributes. Each OOV word can have up to 5 annotated share-weighted attributes, therefore we proposed a different scoring criterion than Task 1.


where is the top prediction set of the th test sample and is the annotated set of th test sample.

We provide details of baseline experiments in next chapters, with the and scores reported.

Iv Experiment

We used Word2Vec [3] and Word2GM [4] as baseline models of the tasks. The data set we utilized for training is a subset of the Wikipedia corpus with M tokens, and K unique tokens. The low frequency words (appear less than 5 times) are removed from the training set, leaving a vocabulary of K to out models.

Iv-a Word2Vec

The skip-gram Word2Vec model can efficiently project a vocabulary of words into a finite dimensional vector space by maximizing


where and are vector representations for context and word respectively [22].

One limitation of Word2Vec is that each word can only be represented as one single dot in , and many English words have multiple meanings in use.

The Word2Vec model we trained has 50 dimensions, and in order to compare the performance, we also used a pretrained Word2Vec model from Facebook’s FastText (https://fasttext.cc/docs/en/english-vectors.html). It was trained with 16B tokens, has 300 dimensions and contains 1M most frequent words.

Iv-B Word2GM

As an attempt to solve the limitation of Word2Vec mentioned above, the word embedding with Gaussian mixtures (word2gm) [4] tried to represent each word with a Gaussian mixture in high-dimensional space :


The objective function, derived from Word2Gauss [20] is to minimize the max-margin ranking:


where the term is from negative sampling. Several metrics can be applied to measure the similarity of words:

Fig. 1: An example of multi-sense words

Maximum cosine similarity


where ,

are the index of the Gaussian distribution in its Gaussian mixture. Note the cosine similarity is equivalent to the normalized Euclidean distance

. We use this metric to evaluate the models.

Expected likelihood [23]:




Note that KL-divergence is an asymmetric measure of similarity between two distributions.

Iv-C Training

We used similar hyper-parameters in Word2Vec and Word2GM training, in order to make the results comparable with each other. Both models applied skip-gram context window , space dimensions and only contained words with word frequency . Training a Word2Vec model is fairly straightforward.

It is yet unclear how to properly choose the number of Gaussians per word , the boundaries of and and their initial values, thus we simply borrowed the model from the original paper of Word2GM [4]. During training, we observed that the vector space is dense in terms of the radius and covariance matrices , and it might affect the convergence speed of the adaptive gradient descent algorithm we used.

Iv-D Testing

We examined the task performance on several models. Firstly, we trained a Word2Vec model with 250M English Wikipedia corpus. Secondly, we trained a Word2GM model with the same Wikipedia corpus and a Word2GM model with a trimmed 100M English Wikipedia corpus since training Word2GM models is quite time-consuming and memory-expensive, and we’re currently investigating how to optimize the Gaussian mixture embedding algorithm. Finally, we compared these models with a pre-trained Word2Vec model from Facebook FastText.

V Evaluation

Model Accuracy Score
w2v-250m 0.64 1.55
w2v-fb 0.68 1.66
w2gm-250m 0.54 2.09
w2gm-100m 0.50 2.25
TABLE I: Task 1 Evaluation
Model Score
w2v-250m 4.8%
w2v-fb 3.0%
w2gm-250m 3.2%
w2gm-100m 2.2%
TABLE II: Task 2 Evaluation

As shown in Table 1 and Table 2 the Word2Vec models generally out-perform Word2GM models. The accuracy was calculated based on the best 1 prediction per OOV word while the score takes all the predictions into consideration. In Task 2, only the 5 best predictions for each OOV word were scored since making more prediction attempts would drop the scores. The result of Task 2 indicates that the unsupervised model outputs are still far from how humans use English. Therefore, we want these results to become the baseline of the tasks.

Vi Conclusions

This is the first attempt to address the issues of OOV lexical prediction in NLP tasks with unstructured data and given no prior knowledge. Hence, we propose two tasks for OOV classification and prediction, then we create baseline results with several models based on Word2Vec and Word2GM algorithms. Our result has shown that, OOV lexical prediction is still challenging with unsupervised word embedding models.


  • [1] James R Curran, “Supersense tagging of unknown nouns using semantic similarity,” in Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 2005, pp. 26–33.
  • [2] Massimiliano Ciaramita and Mark Johnson, “Supersense tagging of unknown nouns in wordnet,” in Proceedings of the 2003 conference on Empirical methods in natural language processing. Association for Computational Linguistics, 2003, pp. 168–175.
  • [3] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean,

    Distributed representations of words and phrases and their compositionality,”

    in Advances in neural information processing systems, 2013, pp. 3111–3119.
  • [4] Ben Athiwaratkun and Andrew Gordon Wilson, “Multimodal word distributions,” in Conference of the Association for Computational Linguistics (ACL), 2017.
  • [5] Ziming Zhang and Venkatesh Saligrama, “Zero-shot learning via semantic similarity embedding,” in

    Proceedings of the IEEE International Conference on Computer Vision

    , 2015, pp. 4166–4174.
  • [6] Diana McCarthy and Roberto Navigli, “Semeval-2007 task 10: English lexical substitution task,” in Proceedings of the 4th International Workshop on Semantic Evaluations. Association for Computational Linguistics, 2007, pp. 48–53.
  • [7] Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea, “Semeval-2012 task 1: English lexical simplification,” in Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation. Association for Computational Linguistics, 2012, pp. 347–355.
  • [8] Ye Wang and Mi Lu, “A self-adaptive algorithm to defeat text-based CAPTCHA,” in Industrial Technology (ICIT), 2016 IEEE International Conference on. IEEE, 2016, pp. 720–725.
  • [9] Ye Wang, Yuanjiang Huang, Wu Zheng, Zhi Zhou, Debin Liu, and Mi Lu,

    “Combining convolutional neural network and self-adaptive algorithm to defeat synthetic multi-digit text-based CAPTCHA,”

    in Industrial Technology (ICIT), 2017 IEEE International Conference on. IEEE, 2017, pp. 980–985.
  • [10] Advaith Siddharthan, “Syntactic simplification and text cohesion,” Research on Language & Computation, vol. 4, no. 1, pp. 77–109, 2006.
  • [11] Ido Dagan, Oren Glickman, and Bernardo Magnini, “The pascal recognising textual entailment challenge,” in Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177–190. Springer, 2006.
  • [12] Or Biran, Samuel Brody, and Noémie Elhadad, “Putting it simply: a context-aware approach to lexical simplification,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computational Linguistics, 2011, pp. 496–501.
  • [13] Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze, “Boolean retrieval,” Introduction to information retrieval, pp. 1–18, 2008.
  • [14] DE Rumelhart, “David e. rumelhart, geoffrey e. hinton, and ronald j. williams,” Nature, vol. 323, pp. 533–536, 1986.
  • [15] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, “A neural probabilistic language model,” Journal of machine learning research, vol. 3, no. Feb, pp. 1137–1155, 2003.
  • [16] Ye Wang, Zhi Zhou, Shan Jin, Debin Liu, and Mi Lu, “Comparisons and selections of features and classifiers for short text classification,” in IOP Conference Series: Materials Science and Engineering. IOP Publishing, 2017, vol. 261, p. 012018.
  • [17] Joseph Reisinger and Raymond J Mooney, “Multi-prototype vector-space models of word meaning,” in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2010, pp. 109–117.
  • [18] Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng, “Improving word representations via global context and multiple word prototypes,” in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 2012, pp. 873–882.
  • [19] Fei Tian, Hanjun Dai, Jiang Bian, Bin Gao, Rui Zhang, Enhong Chen, and Tie-Yan Liu, “A probabilistic model for learning multi-prototype word embeddings.,” in COLING, 2014, pp. 151–160.
  • [20] Luke Vilnis and Andrew McCallum, “Word representations via gaussian embedding,” arXiv preprint arXiv:1412.6623, 2014.
  • [21] Ye Wang and Mi Lu, “An optimized system to solve text-based CAPTCHA,” arXiv preprint arXiv:1806.07202, 2018.
  • [22] Yoav Goldberg and Omer Levy, “word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method,” arXiv preprint arXiv:1402.3722, 2014.
  • [23] Tony Jebara, Risi Kondor, and Andrew Howard,

    Probability product kernels,”

    Journal of Machine Learning Research, vol. 5, no. Jul, pp. 819–844, 2004.