Multilingual Word Embeddings using Multigraphs

12/14/2016 ∙ by Radu Soricut, et al. ∙ Google 0

We present a family of neural-network--inspired models for computing continuous word representations, specifically designed to exploit both monolingual and multilingual text. This framework allows us to perform unsupervised training of embeddings that exhibit higher accuracy on syntactic and semantic compositionality, as well as multilingual semantic similarity, compared to previous models trained in an unsupervised fashion. We also show that such multilingual embeddings, optimized for semantic similarity, can improve the performance of statistical machine translation with respect to how it handles words not present in the parallel data.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Word embeddings are representations that use vectors to represent word surface forms. They are known to be useful in improving the performance of many NLP tasks, from sequence labeling 

[Huang et al.2014, Turian et al.2010], to language modeling [Bengio et al.2003, Mnih et al.2009], to parsing [Finkel et al.2008, Bansal et al.2014] and morphological analysis [Soricut and Och2015, Cotterell et al.2016]. Cross-lingual tasks (parsing, retrieval, translation) have been particularly attractive as applications for word embeddings, see [Upadhyay et al.2016] for a good overview. Continuous word embeddings use real-valued vectors, and are typically induced either via neural networks [Bengio et al.2003, Mnih et al.2009, Socher et al.2011a], or neural-network–inspired models [Mikolov et al.2013a, Levy and Goldberg2014, Pennington et al.2014, Yu and Dredze2014, Ling et al.2015], designed to learn word embeddings in an unsupervised manner.

The embeddings that result from training such models exhibit certain desirable properties. One such property is syntactic compositionality, i.e., the extent to which morpho-syntactic properties (such as pluralization, past-tense, etc.) can be represented as vectors in the same embedding space, using simple vector arithmetic to encode the application of this property, e.g., . Another property is semantic compositionality, i.e., the extent to which semantic properties (such as “maleness”, “royalty”, etc.) can be represented as vectors in the same embedding space, using simple vector arithmetic to encode the application of such a property, e.g.,  [Mikolov et al.2013c]. Another desired property, semantic similarity, means that words with similar meaning are represented as vectors that are close (cosine-wise) in the embedding space. Together with semantic compositionality, it allows for manipulating semantic concepts as points in such an embedding space. The extension of the semantic similarity property to spaces that embed words from more than one language results in multilingual semantic similarity, i.e., the extent to which words that are translations of each other are embedded to vectors that are close in the embedding space [Mikolov et al.2013b, Gouws et al.2015, Luong et al.2015].

Figure 1: Multigraph with three types of edges

In this paper, we present a unified framework for designing neural-network–inspired embedding models using multigraphs. The new framework introduces the flexibility to exploit both monolingual and bilingual text, without and with annotations (e.g., syntactic dependencies, word alignments). Furthermore, it unifies existing models such as the SkipGram model of Mikolov et al. mikolov-et-al:2013a, the Dependency embedding model of Levy and Goldberg levy-goldberg:2014, and the BiSkip model of Luong et al. luong-et-al:2015 as particular instances of this framework. Our empirical evaluations show that this framework allows us to build models that yield embeddings with significantly higher accuracy on syntactic compositionality, semantic compositionality, and multilingual semantic similarity. We find that both syntactic and semantic compositionality accuracy of certain embedding spaces improves for a given language due to the presence of words from other language(s), embedded by the same model in a common embedding space.

We also show that multilingual embedding spaces optimized for semantic similarity improve the end-to-end performance of a statistical MT system for English-Spanish, English-French, and English-Czech translations. In order to isolate the impact of this method on end-to-end translation performance, we perform these experiments on non-neural, phrase-based MT systems with standard dense features [Koehn et al.2003], rather than the more recently-proposed neural MT systems [Wu et al.2016, Zhou et al.2016]. The point of this evaluation is not to improve state-of-the-art in MT; rather, it allows for a simple way to evaluate how good the translations induced by our multilingual embeddings are. To this end, we measure the impact of using our approach on handling words that are not present in the parallel data on which our MT systems are trained. Potential translations for such words are automatically identified from our multilingual embeddings, and are subsequently used at decode time to produce improved translations for test sentences containing such words. Although the method used to measure the quality of the induced translations (based on enhancing phrase-tables) cannot be directly used for neural MT systems, the mapping of surface strings to their embeddings in a common multilingual embedding space can be used directly in fully-neural MT systems, as an orthogonal method to previously proposed ones, e.g. sub-word unit translations [Sennrich et al.2016, Wu et al.2016].

2 Multigraph-based Embedding Models

We describe here multigraph-based embedding models, using a formalism that extends on the SkipGram model [Mikolov et al.2013a]. This extension is achieved by replacing the definition of context in SkipGram with the formal definition of multigraph neighborhoods. We first revisit the SkipGram model, then formally define the notion of multigraph and multigraph-based embedding models.

2.1 The SkipGram Embedding Models

The SkipGram neural embedding model of Mikolov et al. mikolov-et-al:2013a uses a vocabulary and an -dimensional real-valued space . For each word , a vector is associated as the embedding of the word , learned as follows.

We define to be the set of context words of , which is a window of words to the left and right of . is the complement for . For example, in the English text (1st line) of Figure 1, .

The training objective is that for all words in the training data, the dot product is maximized for while minimized for . Formally, this is defined as:

where . Due to its large size, is approximated via sampling: for each , we draw samples according to the distribution ( is the unigram distribution, the normalization constant), where is a model hyper-parameter.

2.2 Multigraph-based Embedding Models

The SkipGram embedding models only make use of the context windows. However, there is much more information in both monolingual and bilingual text, e.g. syntactic dependencies and word alignments. For example, in Figure 1, besides the dotted edges connecting neighbors in the text surface (denoted as edges), the dashed edges connect words that are in a syntactic dependency relation (denoted as edges), and solid edges connect nodes that represent translation-aligned words (denoted as edges).

In order to make use of all three types of information, we first define a set of labels , and a distance function by assigning a distance to each label. We use notation to represent , , . If any of the labels are mapped to distance 0, we omit it from the notation; for instance, means , , .

Next, we show how to incorporate those edges with different labels and distance functions into the same embedding model using multigraphs.

2.2.1 Multigraphs

A multigraph is a graph which is allowed to have multiple edges between the same nodes. We use here the particular notion of an undirected multigraph with identifiable edges (i.e., any edge has an identity that helps it being distinguished from a different edge that may share the same end nodes). Formally, an undirected multigraph with identifiable edges is a 5-tuple , where is a set of vertices or nodes; is a set of edges; is a set of labels; assigns end nodes to edges; and assigns identity labels to edges.

We use notation to represent the neighborhood of node in as given by a distance function . is defined as:

where denotes the edges of a length- path from to .

Using the example from Figure 1, we have , because there is a length-2 path with two -type edges , with and . On the other hand, , because the path contains an -type edge and a -type edge , hence while .

2.2.2 Objective Function

Training the embedding variables is based on the neighborhoods for all words in the multigraph. The general form of the objective function can take any arbitrary function defined on and , where is the complement of . We use here a similar objective function to the one used by the SkipGram model:


where . Due to its large size, is approximated similarly with : for each , we draw samples according to the unigram 111The unigram vocabulary over multilingual text is considered to be the union over all language-tagged word types. distribution , where is a model hyper-parameter.

2.2.3 Discussion

Given sentence-level parallel data with word alignments, Equation 1 can be used to learn parameters for a model such as . Under this model, the graph neighborhood for in Figure 1 is . The neighborhood for is .

If syntactic dependency relations are available (for one or both languages in the parallel data), we can also learn parameters for model . For the multigraph in Figure 1, the graph neighborhood for is .

We can also create multigraph models for which the set of labels collapses some of the labels. For instance, collapsing labels T and A leads to a multigraph model such as , under which both text-based edges and alignment-based edges are traversed without differentiation (up to distance 3). In this case, the graph neighborhood for is . For , it is

Note the  90% overlap between and . Since the objective function from Equation 1 imposes that words that appear in similar contexts have similar embeddings, it follows that, under a model like , the embeddings for and should be similar. This leads to embeddings for words in multiple languages that keeps translations close to each other. On the other hand, models like and do not have this property, but nevertheless give rise to embeddings for words in multiple languages with properties that we analyze and quantify in Section 4.

3 Related Work

Distributed word representations have been used recently for tackling various tasks such as language modeling [Mnih and Hinton2007, Mikolov and Zweig2012], paraphrase detection [Socher et al.2011a]

, sentiment analysis 

[Socher et al.2011b], syntactic parsing [Collobert2011, Faruqui and Dyer2014, Guo et al.2015, Huang et al.2015], and a multitude of cross-lingual tasks [Upadhyay et al.2016].

The introduction of the CBOW and SkipGram embedding models [Mikolov et al.2013a] has boosted this research direction. These models are simple and easy to implement, and can be trained orders of magnitude faster than previous models. Subsequent research has proposed models that take advantage of additional textual information, such as syntactic dependencies [Levy and Goldberg2014, Bansal et al.2014], global statistics [Pennington et al.2014], or parallel data [Mikolov et al.2013b, Gouws et al.2015, Luong et al.2015, Mogadala and Rettinger2016]. Prior knowledge can also be incorporated to achieve improved lexical embeddings by modifying the objective function while allowing for the exploitation of existing resources such as WordNet [Yu and Dredze2014], or by modifying the model architecture while targeting specific tasks [Ling et al.2015].

Our paper describes a mechanism which unifies the way context signals from the training data are exploited. It exploits the information available in parallel data in an on-line training fashion (bi/multi-lingual training), compared to the off-line matrix transformation proposal of Mikolov et al. mikolov-et-al:2013c. The BilBOWA model [Gouws et al.2015] uses a parallel bag-of-word representation for the parallel data, while the BiSkip model [Luong et al.2015] achieves bilingual training by exploiting word alignments. Some of these proposals can be formulated as particular instances under our multigraph framework. For instance, the context window (with window size ) of the SkipGram model is equivalent to model in the multigraph formulation. The dependency-based embedding model of Levy and Goldberg levy-goldberg:2014 is equivalent to when word and context vocabulary are the same 222 We also ignore the collapse of preposition-based dependencies, which makes their model be between and .. The BiSkip model [Luong et al.2015] with a window of size is equivalent to model in our multigraph formulation.

In addition to subsuming some of the previously-proposed methods, our approach comes with a mathematical foundation (in terms of multigraphs) for incorporating information from both monolingual and parallel data. This formulation allows us to understand and justify, from a formal perspective, some of the empirical results obtained by some of these models. Moreover, our method allows for the exploitation of signals obtained via both unsupervised (e.g. raw text, parallel text with unsupervised alignments) and supervised learning (e.g., syntactic dependencies), while building a common embedding over arbitrary many languages, simply by treating the training data as a multigraph over potentially multiple languages, linked together via multiple bilingual alignments.

A related approach for inducing multilingual embeddings is based on neural networks for automatic translation, either in conjunction with a phrase-table [Devlin et al.2014, Setiawan et al.2015], or a fully neural approach [Sutskever et al.2014, Cho et al.2014, Wu et al.2016, Zhou et al.2016]

. These approaches use signals similar to ours when exploiting parallel training data, but the resulting embeddings are optimized for translation accuracy (according to the loss-function definition of these models, usually using a maximum-likelihood objective 

[Sutskever et al.2014, Cho et al.2014]

or a reinforcement-learning–inspired objective 

[Wu et al.2016]). In addition, they do not directly allow for the exploitation of both parallel and monolingual data simultaneously at train time, or the exploitation of additional sources of linguistic information (such as syntactic dependencies).

Because our approach enables us to exploit both monolingual and parallel data simultaneously, the resulting distributed representation can be successfully used to learn translations for terms that appear in the monolingual data only (Section 

4.3). This represents the neural word-embedding equivalent of a long line of research based on word-surface patterns, starting with earlier attempts [Rapp1995], and continuing with [Koehn and Knight2002, Garera et al.2009], and complemented by approaches based on probabilistic models [Haghighi et al.2008]. Our approach has the advantages of achieving this effect in a completely unsupervised fashion, without exploiting surface patterns, and benefiting from the smoothness properties associated with continuous word representations.

Test Name Input Reference Size
Word-Rel Syn car - child + children cars 10,675
Word-Rel Sem policeman - groom + bride policewoman 8,869
Mitchell elderly & woman, black & hair 1.6 324
amount & reduce, cost & cut 6.6
Stanford-C … is a celebrated jazz clarinetist …, 2,003
… an English rock band … 8.2
Stanford-R amorphous, inorganic 1.9 2,034
belligerence, hostility 8.7
Mikolov Transl. EnEs strained tenso 1,000
Mikolov Transl. EsEn inteligente clever 1,000
Table 1: Descriptions of the standard test sets used in our evaluations.
Word-Rel Syn Word-Rel Sem Mitchell Stanford-C Stanford-R
Model Acc@1 Spearman
(SkipGram) 68.4[67.5, 69.2] 76.7[75.8, 77.6] 64.0[58.0, 70.8] 66.5[63.9, 69.2] 42.6[39.0, 46.3]
(Dependency) 67.2[66.3, 68.1] 46.4[45.4, 47.4] 65.3[59.8, 71.6] 66.7[64.3, 69.5] 41.1[37.4, 44.8]
72.6[71.8, 73.5] 76.2[75.3, 77.0] 65.3[59.8, 71.6] 68.3[65.9, 70.9] 45.2[41.7, 48.8]
Table 2: Evaluation for models , , , trained on English Wikipedia (1Bw). is significantly better on Word-Rel Syn, while all other tests stay at similar levels.

4 Empirical Results

In this section, we present empirical evaluations aimed at answering several questions: How do multigraph embeddings, such as , , , differ from regular SkipGram embeddings (i.e., ? What leverage do we get from using multilingual data? How can we use multigraph embeddings to accomplish non-trivial end-to-end tasks (such as automatic translation)? We provide answers for each of these questions next.

4.1 Data

As training data, we use publicly available parallel and monolingual data. For parallel data, we use the Spanish-English, French-English, German-English, Russian-English, and Czech-English data released as part of the WMT-2015 shared task [Bojar et al.2015]. For monolingual data, we use the LDC Gigaword collections for English (5 Bw) [Graff and Cieri2003], Spanish (1.4 Bw) [Mendonça et al.2011], and French (1.1 Bw) [Graff et al.2011]. For monolingual English, we also use the Wikipedia data (1 Bw) [Shaoul and Westbury2010]. This setup allows us to directly compare with previous published results that use the same data sets.

As test data, we use several standard, publicly released test sets, see Table 1. We use the Semantic-Syntactic Word Relationship dataset introduced in [Mikolov et al.2013c] by reporting results separately on the syntactic and the semantic parts. The role of the Word-Rel Syn part of this dataset is to quantify the syntactic compositionality property of an embedding space; the role of the Word-Rel Sem part is to quantify the semantic compositionality property of an embedding space. We also use the Mitchell dataset [Mitchell and Lapata2010], consisting of pairs of two-word phrases and a human similarity judgment on the scale of 1-7; the Stanford English in-context word-similarity (Stanford-C) dataset [Huang et al.2012], consisting of sentence pairs and a human similarity judgment on the scale of 1-10; the Stanford English Rare-Word (Stanford-R) dataset [Luong et al.2013], consisting of word pairs and human similarity judgments, containing a higher degree of English morphology compared to other word-similarity datasets; the Mikolov translation dataset [Mikolov et al.2013b]

, consisting of English-Spanish word pairs that are translations of each other (single reference). All confidence intervals reported are at 95% confidence level.

4.2 Intrinsic Evaluations

As evaluation metrics, we use Acc@1 (the percent of cases for which the 1-highest cosine exactly matches the reference), Acc@5 (the percent of cases for which one of the 5-highest cosine exactly matches the reference), and Spearman correlation (how well cosine-based rankings match human-score–based rankings).

Two Edge Types are Better than One:

The first multigraph model we evaluate is model . This model formally subsumes both the SkipGram model (via neighborhoods) and the Dependency model (via neighborhoods). Table 2 presents the performance of this model across five different testsets, against models and . For training we used English Wikipedia (1B words), and a dependency parser similar with the one described in [Petrov2012].

The results show that model is capable of combining the strength of both the model and the model, and achieves significantly improved performance on syntactic compositionality (72.6% Acc@1 on Word-Rel Syn, compared to 68.3% for SkipGram). On all other tests, the performance does not degrade compared to the best achieved by one of the single-edge type models: 76.2% Acc@1 on Word-Rel Sem, compared to 76.7% for SkipGram; 68.3 Spearman on Stanford-C, compared to 66.5 for SkipGram, and 65.7 reported by [Huang et al.2012] on the same training set; 45.2 Spearman on Stanford-R, compared to 42.6 for SkipGram, and 34.4 reported by [Luong et al.2013] on the same training set.

Improved syntactic compositionality:

To investigate in more detail the syntactic compositionality property, we train multigraph models of type (SkipGram) and on various language-pairs, using the WMT parallel datasets, as follows: 1LP=EnEs; 2LPs=EnEs+EnDe; 4LPs=EnEs+EnDe+EnRu+EnCz. The results in Table 3 show the positive impact on the syntactic compositionality due to the presence of the other language(s).

Word-Rel Syn Word-Rel Sem
Model Acc@1
[1LP] 26.9[26.1, 27.7] 2.5[2.2, 2.9]
[1LP] 37.8[36.9, 38.6] 18.3[17.4, 19.1]
[1LP] 51.6[50.7, 52.5] 5.9[5.4, 6.4]
[2LPs] 48.0[47.1, 48.9] 33.0[32.0, 34.0]
[2LPs] 57.9[57.0, 58.8] 11.3[10.6, 11.9]
[4LPs] 46.8[45.8, 47.7] 39.6[38.6, 40.6]
[4LPs] 59.4[58.4, 60.2] 12.1[11.4, 12.8]
Table 3: Model versus on Word-Rel (Syn/Sem); multilingual setups with WMT data.

Note that Word-Rel Syn measures properties of embeddings for English words only. Yet the multilingual aspect of the embeddings directly impacts the syntactic compositionality of English embeddings (59.3% Acc@1 on Word-Rel Syn with the 4LPs combination, versus 46.8% Acc@1 when using only the English side of the same training data for ). Also note that this is not due only to simply having more tokens/events when training the models: for instance, [1LP] is trained on roughly the same number of tokens as [2LPs] (the former has tokens in English and Spanish, while the latter has only English tokens but on twice more data), yet the Acc@1 of the former, at 51.6%, is statistically-significantly better compared to the latter, at 48.0%.

Another significant finding from Table 3 is that the improved syntactic compositionality of embedding spaces comes at the expense of semantic compositionality (12.1% Acc@1 on Word-Rel Sem with the 4LPs combination, compared to 39.6% Acc@1 for ). Poor semantic compositionality appears to be a consequence of multigraph neighborhoods being too local; we show in later results how to mitigate this issue.

Improved semantic compositionality:

We use the same multilingual training setup to investigate in more detail the semantic compositionality property as well. We train multigraph models of type on the same combination of WMT parallel datasets as above.

Word-Rel Syn Word-Rel Sem
Model Acc@1
[1LP] 37.8[36.9, 38.6] 18.3[17.4, 19.1]
[1LP] 40.9[40.0, 41.8] 30.7[29.8, 31.7]
[2LPs] 48.0[47.1, 48.9] 33.0[32.0, 34.0]
[2LPs] 48.1[47.2, 49.0] 43.1[42.0, 44.0]
[4LPs] 46.8[45.8, 47.7] 39.6[38.6, 40.6]
[4LPs] 48.2[47.3, 49.1] 46.4[45.3, 47.4]
Table 4: Model versus on Word-Rel (Syn/Sem); multilingual setups with WMT data.

The results in Table 4 show a significant positive impact, this time on the semantic compositionality of the resulting embeddings space, due to the presence of the other language(s). We emphasize that the Word-Rel Sem dataset measures properties of embeddings for English words only, but the multilingual aspect of the embeddings directly impacts the semantic compositionality of the English embeddings (Word-Rel Sem at 46.4% Acc@1 under the 4LPs condition, versus 39.6% for SkipGram).

For this model, the improved semantic compositionality also comes with a statistically-significant increase in syntactic compositionality (Word-Rel Syn 48.2% Acc@1 under the 4LPs condition, versus 46.8% for SkipGram). However, this result is significantly below the one obtained by the model on the same training data on syntactic compositionality (59.3% Acc@1 on Word-Rel Syn, see Table 3).

Word-Rel Syn Word-Rel Sem Mitchell Stanford-C Stanford-R
Model Acc@1 Spearman
(SkipGram) [En+4LPs] 67.8[67.0, 68.6] 74.4[73.4, 75.3] 68.4[62.7, 74.3] 67.4[64.9, 70.1] 44.8[41.3, 48.4]
[En]+[4LPs] 72.0[71.2, 72.8] 78.3[77.4, 79.2] 66.4[60.9, 72.9] 68.6[66.1, 71.3] 46.6[43.1, 50.1]
Table 5: Multigraph embeddings in a monolingual plus (multi) bilingual setup. The monolingual data [En] is Wikipedia (1Bw), the parallel data (4LPs=EnEs+EnDe+EnRu+EnCz) is WMT-2015 (1Bw on English side).

Improved syntactic+semantic compositionality

It is also possible to train embeddings that achieve superior performance on both syntactic and semantic compositionality. To this end, we train a combined model that runs on multigraphs on monolingual English data (Wikipedia 1Bw), plus multigraphs on parallel data; we denote the resulting model [En]+[4LPs].

As the evaluation in Table 5 shows, [En]+[4LPs] registers both the improved syntactic compositionality of and the improved semantic compositionality of . This model significantly improves on testsets impacted by syntactic compositionality (72.0% Acc@1 on Word-Rel Syn versus 68.5% for SkipGram) and semantic compositionality (78.3% Acc@1 on Word-Rel Sem versus 74.4% for SkipGram). We make the conjecture here that better compositionality properties can improve the end-performance of techniques that explicitly exploit them, such as morphology induction [Soricut and Och2015].

Improved multilingual embeddings:

We also investigate the extent to which multigraph-based embeddings encode multilingual semantic similarity. The intuition for this property is that words in various languages that are translations of each other should embed to points that are close to each other in the common embedding space.

Mikolov Translation Test
Model EnEs EsEn
Acc@1 Acc@1
 [Mikolov et al.2013b] 33.0[–,–] 35.0[–,–]
 [Gouws et al.2015] 39.0[–,–] 44.0[–,–]
[EnEs] 57.3[54.3, 60.3] 58.2[55.2, 61.0]
Table 6: Evaluations of model on a intrinsic translation accuracy task. Training data for all models is English-Spanish WMT-2015.

The results in Table 6 show the superiority of embeddings compared to embeddings obtained with previously-proposed approaches: the model of Mikolov et al. mikolov-et-al:2013c, and the BilBOWA model of Gouws et al. gouws-et-al:2015. At 57.3% Acc@1 on EnEs and 58.2% Acc@1 on EsEn, model outperforms by a large margin the best results reported on these test sets. The advantage of the model is that it can fully exploit the information made available by word-level alignments during on-line training: under the objective function from Eq. 1 using multigraph-based neighborhoods, words aligned in the training data embed to points that have high cosines in the common embedding space.

Mikolov Translation Test
Model Acc@1 Acc@5
[EnEs] 57.3[54.3, 60.3] 86.3[84.3,88.4]
[2LPs] 51.6[48.7, 54.5] 90.0[88.1, 91.7]
[4LPs] 49.3[46.2, 52.2] 90.6[88.9, 92.3]
Table 7: Intrinsic translation accuracy with model ; multilingual setups with WMT-2015 data.

We also test what happens when we train our models on multiple bilingual training sets. In addition to training the model on the English-Spanish parallel data only, we also train it using two bilingual training sets (2LPs setup, for EnEs+EnDe), and four bilingual training sets (4LPs setup, for EnEs+EnDe+EnRu+EnCz). The figures in Table 7 show the results of such a setup. Note that we tag words from different languages with language tags (see Figure 1), and compute cosine ranks (i.e., Acc@1 and Acc@5) considering the embedded points from the desired target language only. The results show lower translation performance on Acc@1 on the English-Spanish test set: from 57.3% Acc@1 for [EnEs] under the 1LP (English-Spanish) condition, to 51.6% Acc@1 under the 2LPs condition, to 49.4% Acc@1 under the 4LPs condition. Our interpretation is that multiple bilingual events tend to pull the embedding parameters for English words in different directions, leading to a decrease in Acc@1 translation accuracy. However, under Acc@5 this trend is reversed, and accuracy numbers improve when adding more bilingual training data. This is an indication that, even though the addition of multiple languages inserts some additional noise to the embeddings (for instance, due to polysemy), multiple bilingual datasets contribute positively to the creation of a common embedding space that is geared toward translation invariance.

4.3 Extrinsic Evaluation: Machine Translation

In this section, we use multigraph embeddings to improve automatic translation between a source language and a target language . Multigraph model - uses: text-based edges on a monolingual corpus (), text-and-alignment–based edges on a parallel corpus - (-); and text-based edges on a monolingual corpus (333 The two monolingual corpora should be from a fairly comparable domain.. The choice of the neighborhood creation is based on the results of the previous evaluations, using a model optimized for multilingual semantic similarity between and .

4.3.1 Inducing Translations from Non-parallel Data

The empirical results we report here exploit the properties of our multilingual embeddings for automatic translation purposes. The intuition we intend to exploit is that words that are out-of-vocabulary for the source-side of the parallel corpora (we denote them pOOV, for ’parallel Out-of-Vocabulary’) are embedded in the same multilingual embedding space as the words in the vocabulary of the target language (both from the parallel-target and monolingual-target corpora). Under a multilingual embedding space optimized for semantic similarity, pOOV source words and their highest-cosine target-language neighbors tend to be semantically related (for highest-cosine above a cosine threshold ); therefore, such pairs should provide potentially correct translations, even for word-pairs not directly observed in the parallel data.

To ground our intuition, we first present the results of a small evaluation done using WMT English-Spanish data. We use a set of 100 word-pairs of the form , randomly extracted according to the following set of criteria: is an En pOOV in [EnEs], with a count of at least 100 in [En]; is the 1-highest (cosine-wise) Es neighbor of in [En][EnEs][Es], with (we used ). We ask human evaluators to rate each pair using one of the following ratings: (well translated, with no/little semantic loss; syntactic differences are OK); (translation is a hypernym (less specific); e.g.: ”flower” is a hypernym for ”daffodil”); (translation is a hyponym (more specific); e.g.: ”catholic” is a hyponym for ”churchgoer”); (translation has other/no semantic relation with the source word).

We present in Table 8 the results of this evaluation. The evaluation was done using two highly-skilled bilingual linguists, with (rare) disagreements resolved in a second pass. The results indicate that, in about 50% of the cases, the translation proposed for the pOOV words are reasonable (a hypernym is often a reasonable choice as a translation).

Rating Example Perc.
boozing alcoholismo 15%
lamprey peces 35%
preternatural magia 4%
unladylike arrogante 46%
Table 8: Semantic-relation evaluation of English-Spanish pOOV (for [EnEs]) translations.

4.3.2 End-to-end Performance Impact using Translations from Non-parallel Data

The results above are corroborated by quantitative evaluation results done using end-to-end MT systems. To evaluate the impact of our technique in isolation, we use as base condition a non-neural, phrase-based MT system. We favor such a system in this evaluation simply because it allows a straightforward way to evaluate how good the induced translations for the pOOV words are.

Under our base condition, we use phrase-based MT systems trained on WMT data on three - language pairs: English-Spanish, English-French, and English-Czech. When possible, we use the publicly available monolingual and parallel data mentioned in Section 4.1: the LDC Gigaword corpora for English, Spanish, and French; and the WMT-2015 parallel corpora for EnEs, EnFr, and EnCz. For monolingual Czech, we use a collection of 2.5 Bw of Czech news harvested from the web and automatically cleaned (boiler-plate removed, formatting removed, encoding made consistent, etc.). For each language, the target language model is trained on a monolingual corpus that subsumes both the target-parallel and the target-monolingual corpora.

Under the test condition, we use the same system plus additional phrase-tables that contain entries of the form that meet the selection criteria from Section 4.3.1. For each language pair, we also create special test sets (see Table 9), extracted from the English Gigaword to contain at least 1 pOOV word per sentence. On these test sets, the decoder has the opportunity to use as a possible translation for , in contrast to the base condition in which the only choice of the decoder is to use in the target translation.

Model pOOV Test Scores
Sentences Change-rate no-pOOV-transl pOOV-transl Delta [confidence]
[En][EnEs][Es] 1010 25.5% 2.67 2.98 +0.32 [0.19, 0.42]
[En][EnFr][Fr] 1001 62.2% 2.81 2.96 +0.14 [0.06, 0.22]
[En][EnCz][Cz] 1004 20.7% 1.94 2.19 +0.25 [0.13, 0.36]
Example #1
Source: The two sides have had a testy relationship since Singapore was expelled from the Malaysian Federation.
MT-Spanish Base: Las dos partes han mantenido una relación testy desde Singapur fue expulsada de la Federación de Malasia.
MT-Spanish Test: Las dos partes han tenido relaciones tensas desde Singapur fue expulsada de la Federación de Malasia.
Example #2
Source: They doubted whether he could play the ultimate game, with the unique pressure it presented, with the
same elan he displayed during the season as NFL Player of the Year.
MT-Spanish Base: Ellos dudaron si podría jugar el partido final, con la única presión que presenta, con la misma Elan se mostrará
durante la temporada de la NFL como el Jugador del Año.
MT-Spanish Test: Ellos dudaron si podría jugar el partido final, con la única presión que presenta, con la misma creatividad que
aparece durante la temporada de la NFL como el Jugador del Año.
Table 9: Translation performance of multigraph bilingual models, using test sets from the English Gigaword containing sentences with parallel-OOV (pOOV) terms; compares base condition (no pOOV translation) against test condition (pOOV translations). The evaluation uses a scale from 0:useless to 6:perfect. For instance, Example #2 gets a score of 3 for the Base translation and a 5 for the Test-condition translation.

As the results in Table 9 show, the systems under the test condition score significantly higher compared to the systems under the base condition, in a side-by-side, randomized blind comparison. This evaluation is done using a pool of professional translators that see base-condition translations and test-condition translations side-by-side, in a randomized position, without knowledge of what change in the MT system is being evaluated; each evaluator is allowed to score only 1% of the evaluation entries, to reduce bias; the scale used is from 0 (useless) to 6 (perfect). The values of the Delta numbers in Table 9 are statistically significant at 95%-confidence using bootstrap resampling [Koehn2004]. The results indicate that the translations induced using - contribute significantly to the translation accuracy of sentences containing pOOV terms. Moreover, the two examples in Table 9 illustrate how our approach is orthogonal to other approaches proposed for dealing with rare or OOV words, such as sub-word unit translations [Sennrich et al.2016]. Whereas for Example #1 one can argue that a rare word like ’testy’ could be decomposed as ’test’+’y’ and subsequently translated correctly using sub-word translations, this cannot happen with word ’elan’ in Example #2. The proposed translation under the test condition, ’creatividad’, may not be a perfect rendering of the word meaning for ’elan’, but it is superior to copying the word in the target translation (as under the base condition), or to any potential sub-word–based translation.

5 Conclusions and Future Work

This article introduces a novel framework for building continuous word representations as -dimensional real-valued vectors. This framework utilizes multigraphs as an appropriate mathematical tool for inducing word representations from plain or annotated, monolingual or multilingual text. It also helps with unifying and generalizing several previously-proposed word embedding models [Mikolov et al.2013a, Levy and Goldberg2014, Luong et al.2015].

We empirically show that this framework allows us to build models that yield word embeddings with significantly higher accuracy on syntactic and semantic compositionality, as well as multilingual semantic similarity. We also show that the latter improvement leads to better translation lexicons for words that do not appear in the parallel training data. The resulting translations are evaluated for end-to-end automatic translation accuracy, and succeed in significantly improving the performance of an automatic translation system.

The family of models defined by our framework is much larger than what we presented here. By choosing different neighborhood or objective functions, some of these models may further improve the performance on syntactic/semantic compositionality or similarity, while others may have other properties yet to be discovered. For instance, machine translation could potentially benefit from a thorough investigation study on the impact of various multigraph-based embeddings in pretraining, which can incorporate a variety of signals (e.g., dependency information in source and/or target context).

In general, the high accuracy on the intrinsic tasks that some of these models exhibit, combined with their attractive computational costs, makes them prime candidates for further exploring their properties and devising mechanisms to exploit them in end-to-end applications.


  • [Bansal et al.2014] Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of ACL, pages 809–815.
  • [Bengio et al.2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model.

    Journal of Machine Learning Research

    , 3:1137–1155.
  • [Bojar et al.2015] Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1–46.
  • [Cho et al.2014] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of EMNLP, pages 1724–1734.
  • [Collobert2011] Ronan Collobert. 2011. Deep learning for efficient discriminative parsing. In

    Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics

    , pages 224–232.
  • [Cotterell et al.2016] Ryan Cotterell, Hinrich Schütze, and Jason Eisner. 2016. Morphological smoothing and extrapolation of word embeddings. In Proceedings of the ACL.
  • [Devlin et al.2014] Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of ACL.
  • [Faruqui and Dyer2014] Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of EACL.
  • [Finkel et al.2008] Jenny Rose Finkel, Alex Kleeman, and Christopher D. Manning. 2008. Efficient, feature-based, conditional random field parsing. In Proceedings of ACL, pages 959–967.
  • [Garera et al.2009] Nikesh Garera, Chris Callison-Burch, and David Yarowsky. 2009. Improving translation lexicon induction from monolingual corpora via dependency contexts and part-of-speech equivalences. In Proceedings of CoNLL, pages 129–137.
  • [Gouws et al.2015] Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. Bilbowa: Fast bilingual distributed representations without word alignments. In Proceedings of ICML.
  • [Graff and Cieri2003] David Graff and Christopher Cieri. 2003. English Gigaword Fifth Edition LDC2003T05. In Linguistic Data Consortium, Philadelphia.
  • [Graff et al.2011] David Graff, Ângelo Mendonça, and Denise DiPersio. 2011. French Gigaword Third Edition LDC2011T10. In Linguistic Data Consortium, Philadelphia.
  • [Guo et al.2015] Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proceedings of the ACL, pages 1234–1244.
  • [Haghighi et al.2008] Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL, pages 771–779.
  • [Huang et al.2012] Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873–882.
  • [Huang et al.2014] Fei Huang, Arun Ahuja, Doug Downey, Yi Yang, Yuhong Guo, and Alexander Yates. 2014.

    Learning representations for weakly supervised natural language processing tasks.

    Computational Linguistics, 40(1):85–120.
  • [Huang et al.2015] Kejun Huang, Matt Gardner, Evangelos Papalexakis, Christos Faloutsos, Nikos Sidiropoulos, Tom Mitchell, Partha P. Talukdar, and Xiao Fu. 2015. Translation invariant word embeddings. In Proceedings of EMNLP, pages 1084–1088.
  • [Koehn and Knight2002] Philipp Koehn and Kevin Knight. 2002. Learning a translation lexicon from monolingual corpora. In Proceedings of the ACL-02 Workshop on Unsupervised Lexical Acquisition - Volume 9, pages 9–16.
  • [Koehn et al.2003] Philipp Koehn, Franz Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of NAACL.
  • [Koehn2004] Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP, pages 388––395.
  • [Levy and Goldberg2014] Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL, pages 302–308.
  • [Ling et al.2015] Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In Proceedings of NAACL, pages 1299–1304.
  • [Luong et al.2013] Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of CoNLL.
  • [Luong et al.2015] Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of NAACL.
  • [Mendonça et al.2011] Ângelo Mendonça, Daniel Jaquette, David Graff, and Denise DiPersio. 2011. Spanish Gigaword Third Edition LDC2011T12. In Linguistic Data Consortium, Philadelphia.
  • [Mikolov and Zweig2012] Tomas Mikolov and Geoffrey Zweig. 2012.

    Context dependent recurrent neural network language model.

    In IEEE Spoken Language Technology Workshop (SLT), pages 234–239.
  • [Mikolov et al.2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeff Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.
  • [Mikolov et al.2013b] Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. In CoRR, volume abs/1309.4168.
  • [Mikolov et al.2013c] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of NAACL-HLT, pages 746–751.
  • [Mitchell and Lapata2010] Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429.
  • [Mnih and Hinton2007] Andriy Mnih and Geoffrey E. Hinton. 2007. Three new graphical models for statistical language modelling. In Machine Learning, Proceedings of the Twenty-Fourth International Conference, pages 641–648.
  • [Mnih et al.2009] Andriy Mnih, Zhang Yuecheng, and Geoffrey E. Hinton. 2009. Improving a statistical language model through non-linear prediction. Neurocomputing, 72(7-9):1414–1418.
  • [Mogadala and Rettinger2016] Aditya Mogadala and Achim Rettinger. 2016. Bilingual word embeddings from parallel and non-parallel corpora for cross-language text classification. In Proceedings of the NAACL.
  • [Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP.
  • [Petrov2012] Slav Petrov. 2012. Coarse-to-Fine Natural Language Processing. Theory and Applications of Natural Language Processing. Springer.
  • [Rapp1995] Reinhard Rapp. 1995. Identifying word translations in non-parallel texts. In Proceedings of ACL, pages 320–322.
  • [Sennrich et al.2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the ACL.
  • [Setiawan et al.2015] Hendra Setiawan, Zhongqiang Huang, Jacob Devlin, Thomas Lamar, Rabih Zbib, Richard M. Schwartz, and John Makhoul. 2015.

    Statistical machine translation features with multitask tensor networks.

    In Proceedings of ACL, pages 31–41.
  • [Shaoul and Westbury2010] Cyrus Shaoul and Chris Westbury. 2010. The Westbury lab Wikipedia corpus.
  • [Socher et al.2011a] Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011a.

    Dynamic pooling and unfolding recursive autoencoders for paraphrase detection.

    In NIPS, pages 801–809.
  • [Socher et al.2011b] Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of EMNLP, pages 151–161.
  • [Soricut and Och2015] Radu Soricut and Franz Joseph Och. 2015. Unsupervised morphology induction using word embeddings. In Proceedings of NAACL.
  • [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc.
  • [Turian et al.2010] Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010.

    Word representations: A simple and general method for semi-supervised learning.

    In Proceedings of the ACL, pages 384–394.
  • [Upadhyay et al.2016] Shyam Upadhyay, Manaal Faruqui, Chris Dyer, and Dan Roth. 2016. Cross-lingual models of word embeddings: An empirical comparison. In Proceedings of the ACL.
  • [Wu et al.2016] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
  • [Yu and Dredze2014] Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In Association for Computational Linguistics (ACL).
  • [Zhou et al.2016] Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep recurrent models with fast-forward connections for neural machine translation. Transactions of the Association for Computational Linguistics, 4:371––383.