Weakly-Supervised Concept-based Adversarial Learning for Cross-lingual Word Embeddings

04/20/2019 ∙ by Haozhou Wang, et al. ∙ 0

Distributed representations of words which map each word to a continuous vector have proven useful in capturing important linguistic information not only in a single language but also across different languages. Current unsupervised adversarial approaches show that it is possible to build a mapping matrix that align two sets of monolingual word embeddings together without high quality parallel data such as a dictionary or a sentence-aligned corpus. However, without post refinement, the performance of these methods' preliminary mapping is not good, leading to poor performance for typologically distant languages. In this paper, we propose a weakly-supervised adversarial training method to overcome this limitation, based on the intuition that mapping across languages is better done at the concept level than at the word level. We propose a concept-based adversarial training method which for most languages improves the performance of previous unsupervised adversarial methods, especially for typologically distant language pairs.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Distributed representations of words which map each word to a continuous vector have proven useful in capturing important linguistic information. Vectors of words that are semantically or syntactically similar have been shown to be close to each other in the same space Mikolov et al. (2013a, c); Pennington et al. (2014)

, making them widely useful in many natural language processing tasks such as machine translation and parsing

Bansal et al. (2014); Mi et al. (2016), both in a single languages but also across different languages.

Mikolov et al. (2013b) first observed that the geometric positions of similar words in different languages were related by a linear relation. Zou et al. (2013) showed that a cross-lingually shared word embedding space is more useful than a monolingual space in an end-to-end machine translation task. However, traditional methods for mapping two monolingual word embeddings require high quality aligned sentences or dictionaries Faruqui and Dyer (2014); Ammar et al. (2016b).

Reducing the need for parallel data, then, has become the main issue for cross-lingual word embedding mapping. Some recent work aiming at reducing resources has shown competitive cross-lingual mappings across similar languages, using a pseudo-dictionary, such as identical character strings between two languages Smith et al. (2017), or a simple list of numerals Artetxe et al. (2017).

In a more general method, Zhang et al. (2017) have shown that learning mappings across languages via adversarial training Goodfellow et al. (2014) can avoid using bilingual evidence. This generality comes at the expense of performance. To overcome this limitation, Lample et al. (2018)

refine the preliminary mapping matrix trained by generative adversarial networks (GANs) and obtain a model that is again comparable to supervised models for several language pairs. Despite these big improvements, the performance of these refined GAN models depends largely on the quality of the preliminary mappings. It is probably for this reason that these models still do not show satisfactory performance for typologically distant languages.

We observe that some cross-lingual resources, such as document-aligned data, are not as expensive as high-quality dictionaries. For example, Wikipedia provides thousands of aligned articles in most languages. Based on this observation, in this paper, we explore new augmented models of the adversarial framework that rely on these readily available cross-lingual resources and we develop a weakly supervised adversarial method for learning cross-lingual word embedding mappings.

The main novelty of our method lies in the use of Wikipedia concept-aligned article pairs in the discriminator, which encourages the generator to align words from different languages which are used to describe the same concept. This technique is based on the intuition that the right level of lexical alignment between languages is not the word but the concept. Concepts are then defined as the distribution of words in their respective Wikipedia article, and these distributions are aligned cross-linguistically by the adversarial framework.

The results of our experiments on bilingual lexicon induction show that the preliminary mappings (without post-refinement) trained by our proposed multi-discriminator model are much better than unsupervised adversarial training methods. When we include the post-refinement of

Lample et al. (2018), in most cases, our models are comparable or even better than our dictionary-based supervised baselines, and considerably improve the performance of the method of Lample et al. (2018), especially for distant language pairs, such as Chinese-English and Finnish-English.111We provide our definition of similar and distant in section 5. Compared to previous state-of-the-art, our method still shows competitive performance.

2 Models

The basic idea of mapping two sets of pre-trained monolingual word embeddings together was first proposed by Mikolov et al. (2013b). They use a small dictionary of pairs of words obtained from Google Translate to learn a transformation matrix that projects the embeddings of the source language words onto the embeddings of their translation words in the target language:


The trained matrix can then be used for detecting the translation for any source language word by simply searching a word whose embedding is nearest to . Recent work by Smith et al. (2017) has shown that enforcing the matrix to be orthogonal can effectively improve the results of mapping.

2.1 GANs for cross-lingual word embeddings

Since the core of this linear mapping is derived from a dictionary, the quality and size of the dictionary can considerably affect the result. Recent attempts by Zhang et al. (2017) and Lample et al. (2018) have shown that, even without dictionary or any other cross-lingual resource, training the transformation matrix is still possible using the GAN framework Goodfellow et al. (2014). The standard GAN framework plays a min-max game between two models: a generative model and a discriminative model . The generator learns from the source data distribution and tries to generate new samples that appear drawn from the distribution of target data. The discriminator discriminates the generated samples from the target data.

If we adapt the standard GAN framework to the goal of mapping cross-lingual word embeddings, the objective of the generator is to learn the transformation matrix that maps the source language embeddings to the target language embedding space, and the discriminator

, usually a neural network classifier, detects whether the input is from the target language, giving us the objective:


denote the probability that came from rather than . The inputs of generator are the embeddings sampled from the distribution of source language word embeddings, , and the inputs of discriminator are sampled from both the real target language word embeddings and the generated target language word embeddings . Both and

are trained simultaneously by stochastic gradient descent. A good transformation matrix

can make similar to , so that can no longer distinguish between them. However, this kind of similarity is at the distribution level. Good monolingual word embeddings normally have a large distribution, and without any post-processing the mappings are usually very rough when we look at specific words Lample et al. (2018).

2.2 Concept-based GANs for cross-lingual word embeddings

Under the assumption that words in different languages are similar when they are used to describe the same topic or concept Søgaard et al. (2015), it seems possible to use concept-aligned documents to improve the mapping performance and make the generated embeddings of the source language words closer to the embeddings of their similar words in the target language. Differently from dictionaries and sentence-aligned corpora, cross-lingual concept-aligned documents are much more readily available. For example, Wikipedia contains more than 40-million articles in more than 200 languages222https://en.wikipedia.org/wiki/List_of_Wikipedias. Therefore, many articles of different languages can be linked together by the same concept.

Given two monolingual word embeddings and , our work consists in using a set of Wikipedia concept-aligned article pairs to reshape the transformation matrix . represents the article of concept in the source language and represents the article of the same concept in the target language.

Figure 1: Architecture of our concept-based multi-discriminator model

As shown in Figure 1, the generator uses the transformation matrix to map the input embeddings from the source to the target language, differently from the original GANs, the input of the generator and the input of the discriminator on concept are not sampled from the whole distribution of or , but from their sub-distribution conditioned on the concept , denoted and . represents the embedding of the shared concept . In this paper, we use the embedding of the title of in the target language. For titles consisting of multiple words, we average the embeddings of its words.

Instead of determining whether its input follows the whole target language distribution , the objective of the discriminator now becomes to judge whether its input follows the distribution . Because is smaller than and the proportion of similar words in is higher than in , the embeddings of the input source words have a greater chance of being trained to align with the embeddings of their similar words in the target language.

Our multi-discriminator concept-based model does not completely remove the usual discriminator on language that determines whether its input follows the real target language distribution or the generated target language distribution, . The structure of is very simple, but it is useful when the concept-based discriminator is not stable. The objective function of the multi-discriminator model, shown in equation (3), combines all these elements.


3 Shared Components

Both the GAN and the concept-based GAN models use a variety of further methods that have been shown to improve results.

3.1 Orthogonalization

Previous studies Xing et al. (2015); Smith et al. (2017) show that enforcing the mapping matrix to be orthogonal can improve the performance and make the adversarial training more stable. In this work, we perform the same update step proposed by Cisse et al. (2017) to approximate setting

to an orthogonal matrix:


According to Lample et al. (2018) and Chen and Cardie (2018), when setting to less than 0.01, the orthogonalization usually performs well.

3.2 Post-refinement

Previous work has shown that the core cross-lingual mapping can be improved by refining it by bootstrapping from a dictionary extracted from the learned mapping itself Lample et al. (2018). Since this refinement process is not the focus of this work, we perform the same refinement procedure as Lample et al. (2018) .

After the core mapping matrix is learned, we retrieve the closest target words for the ten thousand most frequent source words. Then, we retrieve back the nearest target words and keep the mutual nearest word pairs for our preliminary dictionary. Finally, we update our preliminary mapping matrix using the objective in equation 1. Moreover, following Xing et al. (2015) and Artetxe et al. (2016), we force the refined matrix

to be orthogonal by using singular value decomposition (SVD):


3.3 Cross-Domain Similarity Local Scaling

Radovanović et al. (2010) and Dinu et al. (2015) demostrated that standard nearest neighbour is not very effective in high-dimensional spaces. Some vectors in the target space often appear as nearest neighbours for many source vectors. Consequently, the target similar words retrieved for a given source word can be noisy. The work of Lample et al. (2018) showed that using cross-domain similarity local scaling (CSLS) to retrieve target similar words is more accurate than standard nearest neighbours. Instead of just considering the similarity between the source word and its neighbours in the target language, CSLS also takes into account the similarity between the target word and its neighbours in the source language:


where represents the mean similarity between a source embedding and its neighbours in the target language, and represent the mean similarity between a target embedding and its neighbours in the source language. In this work, we use CSLS to build dictionary for our post refinement.

de en 0.16M 1.17M 0.73M
es en 0.15M 0.61M 0.65M
fi en 0.17M 0.35M 0.17M
fr en 0.16M 0.63M 0.71M
it en 0.15M 0.56M 0.71M
ru en 0.14M 0.89M 0.56M
tr en 0.01M 0.24M 0.19M
zh en 0.06M 0.39M 0.35M
Table 1: Statistics of Wikipedia concept-aligned articles for our experimentations (M=millions).

4 Training

Since our model is trained in a weakly-supervised fashion, the sampling and model selection procedures are important.

4.1 Sampling Procedure

As the statistics show in Table 1, even after filtering, the vocabulary of concept-aligned articles is still large. But it has been shown that the embeddings of frequent words are the most informative and useful Luong et al. (2013); Lample et al. (2018), so we only keep the one-hundred thousand most frequent words for learning .

For each training step, then, the input word of our generator is randomly sampled from the vocabulary that is common both to the source monolingual word embedding and the source Wikipedia concept-aligned articles. After the input source word is sampled, we sample a concept according to the frequency of in each source article of the ensemble of concepts. Then we uniformly sample a target word from the sub-vocabulary of the target article of concept .333We use uniform sampling for the target words because the distribution of the sampled sub-vocabulary is very small. Sampling according to its local frequency in the target article would significantly increase the sampling probability of words like “the” and “that”, which have less semantic information.

4.2 Model Selection

It is not as difficult to select the best model for weakly-supervised learning as it is for unsupervised learning. In our task, the first criterion is the average of the cosine similarity between the embeddings of the concept title in the source and the target languages. Moreover, we find that if we record the average embeddings of the ten most frequent words in the source and target languages for each aligned article pair, the average cosine similarity between these two sets of embeddings is also a good indication for selecting the best model. As shown in Figure

2, both of these two criteria correlate well with the word translation evaluation task that we describe in section 5. In this paper, we use the average similarity of cross-lingual title as the criterion for selecting the best model.

Figure 2: Model selection criteria

4.3 Other Details

For our generator, the mapping matrix is initialized with a random orthogonal matrix. Our two discriminators and

are two multi-layer perceptron classifiers with different hidden layer sizes (1024 for

and 2048 for

), and a ReLU function as the activation function. In this paper, we set the number of hidden layers to 2 for both

and . In practice, one hidden layer also performs very well; complex classifiers often make the training of models more difficult.

5 Experiments

Since the task of cross-lingual word embedding mapping is situated in a rich context of related work, in this section, we experimentally evaluate our proposal on the task of Bilingual Lexicon Induction (BLI). This task evaluates directly the bilingual mapping ability of a cross-lingual word embedding model. For each language pair, we retrieve the best translations for source words in the test data, and we report the accuracy.

More precisely, for a given source language word, we map its embedding onto the target language and retrieve the closest word. If this closest word is included in the correct translations in the evaluation dictionary, we consider that it is a correct case. In this paper, we report the results that use CSLS to retrieve translations.

Different previous pieces of work on bilingual lexicon induction use different datasets. We choose two from the publicly available ones, selected so as to have a comprehensive evaluation of our method.


The dataset provided by Lample et al. (2018) contains high quality dictionaries for more than 150 language pairs. For each language pair, it has a training dictionary of 5000 words and an evaluation dictionary of 1500 words. This dataset allows us to have a better understanding of the performance of our method on many different language pairs. We choose nine languages for testing and compare our method to our supervised and unsupervised baselines described in section 5.1: English(en), German(de), Finish(fi), French(fr), Spanish(es), Italien(it), Russian(ru), Turkish(tr) and Chinese(zh). We classify similar and distant languages based on a combination of structural (directional dependency distance), as proposed and measured in Chen and Gerdes (2017) and lexical properties, as measured by the clustering of current large-scale multilingual sentence embeddings.444See for example the clustering in https://code.fb.com/ai-research/laser-multilingual-sentence-embeddings/. We consider en de, en fr, en es, en it as similar language pairs and en fi, en ru, en tr, en zh as distant language pairs.


Differently from BLI-1, the dataset of Dinu et al. (2015) and its extensions provided by Artetxe et al. (2017, 2018b) only consists of dictionaries of four language pairs trained on a Europarl parallel corpus. Each dictionary has a training set of 5000 entries and a test set of 1500 entries. Compared to BLI-1, this dataset is much noisier and the entries are selected from different frequency ranges. However, BLI-2 has been widely used for testing by previous methods Faruqui and Dyer (2014); Dinu et al. (2015); Xing et al. (2015); Artetxe et al. (2016); Zhang et al. (2016); Artetxe et al. (2017); Smith et al. (2017); Lample et al. (2018); Artetxe et al. (2018a, b).555Most of these methods have been tested by Artetxe et al. (2018b) by using their own implementations. Using BLI-2 allows us to have a direct comparison with the state-of-the-art.

5.1 Baselines:

As the objective of this work is to improve the performance of unsupervised GANs by using our weakly-supervised concept-based model, we choose the model of Lample et al. (2018) (called MUSE below), as our unsupervised baseline, since it is a typical standard GANs model. We evaluate the models both in a setting without refinement and a setting with refinement. The procedure of refinement is described in section 3.2 and 3.3.

Moreover, it is important to evaluate whether our model is comparable to previous supervised models. We use two different dictionary-based systems, VecMap proposed by Artetxe et al. (2017) and Procrustes, provided by Lample et al. (2018), as our supervised baselines. Previous experiments have already showed that these two systems are strong baselines.

Supervised Without refinement With refinement
VecMap Procrustes MUSE
de-en 72.5 72.0 55.3 66.3 72.2 72.4
en-de 72.5 72.1 59.2 68.2 72.5 73.4
es-en 83.4 82.9 78.1 78.0 83.1 83.6
en-es 81.9 81.5 75.5 74.9 81.1 80.9
fr-en 81.5 82.2 72.2 75.6 81.2 83.3
en-fr 81.3 81.3 77.8 78.3 82.2 82.2
it-en 77.7 79.1 61.0 63.3 76.4 76.2
en-it 77.2 77.5 63.3 64.3 77.4 78.2
avg. 78.5 78.6 67.8 71.1 78.3 78.8
fi-en 55.9 56.2 0.00 42.8 28.6 53.5
en-fi 39.5 39.3 0.00 35.2 21.0 43.0
ru-en 60.5 61.3 43.8 54.3 49.9 58.9
en-ru 48.0 50.9 28.7 43.2 37.6 51.1
tr-en 55.7 58.0 12.2 42.1 25.5 54.7
en-tr 37.3 37.1 26.2 33.1 39.4 41.4
zh-en 45.0 41.2 26.9 32.1 30.8 42.0
en-zh 45.4 52.1 29.4 37.7 33.0 48.0
avg. 48.4 49.5 20.9 40.1 33.2 49.1
Table 2: Results of bilingual lexicon induction (accuracy % P@1) for similar and distant language pairs on the dataset BLI-1. Procrustes AND MUSE represent the supervised and unsupervised model of Lample et al. (2018), Artetxe et al. (2017) . Word translations are retrieved by using CSLS. Bold face indicates the best result overall and italics indicate the best result between the two columns without refinement.

5.2 Data

Monolingual Word Embeddings:

The quality of monolingual word embeddings has a considerable impact on cross-lingual embeddings Lample et al. (2018). Compared to CBOW and Skip-gram embeddings, FastText Bojanowski et al. (2017) capture syntactic information better. The ideal situation would be to use FastText embeddings for both the BLI-1 and BLI-2 dataset. However, much previous work uses CBOW embeddings, so we use different monolingual word embeddings for BLI-1 and for BLI-2.

For BLI-1, we use FastText666https://github.com/facebookresearch/fastText to train our monolingual word embedding models with 300 dimensions for each language and default settings. The training corpus come from a Wikipedia dump 777https://dumps.wikimedia.org/. For European languages, words are lower-cased and tokenized by the scripts of Moses Koehn et al. (2007)888http://www.statmt.org/moses/. For Chinese, we first use OpenCC999https://github.com/BYVoid/OpenCC to convert traditional characters to simplified characters and then use Jieba101010https://github.com/fxsjy/jieba to perform tokenization. For each language, we only keep the words that appear more than five times.

For BLI-2, following the work of Artetxe et al. (2018a)111111In the work of Artetxe et al. (2018a)  the authors reimplemented many previous methods described in table 3, we use their pretrained CBOW embeddings of 300 dimensions. For English, Italian and German, the models are trained on the WacKy corpus. The Finnish model is trained from Common Crawl and the Spanish model is trained from WMT News Crawl.

Concept Aligned Data:

For concept-aligned articles, we use the Linguatools Wikipedia comparable corpus.121212http://linguatools.org/tools/corpora/wikipedia-comparable-corpora/ The statistics of our concept-aligned data are reported in Table 1.

5.3 Results and Discussion

Table 2 summarizes the results of bilingual lexicon induction on the BLI-1 dataset. We can see that with post-refinement, our method achieves the best performance on eight language pairs. Table 3 illustrates the results of bilingual lexicon induction on the BLI-2 dataset. Although our method (with refinement) does not achieve the best results, the gap between our method and the best state-of-the-art Artetxe et al. (2018a) is very small, and in most cases, our method is better than previous supervised methods.

Comparison with unsupervised GANs:

As we have mentioned before, the preliminary mappings trained by the method of Lample et al. (2018) perform well for some similar language pairs, such as Spanish to English and French to English. After refinement, their unsupervised GANs models can reach the same level as supervised models for these similar language pairs. However, the biggest drawback of standard GANs is that they exhibit poor performance for distant language pairs. The results from Table 2 clearly confirm this. For example, without refinement, the mapping trained by the unsupervised GAN method can only correctly predict 12% of the words from Turkish to English. Given that the quality of preliminary mappings can seriously affect the effect of refinement, the low-quality preliminary mappings for distant language pairs severely limits the improvements brought by post-refinement. Notice that the method of Lample et al. (2018) scores a null result for English to Finnish on both BLI-1 and BLI-2, indicating that totally unsupervised adversarial training can yields rather unpredictable results.

Compared to the method of Lample et al. (2018), the improvement brought by our method is apparent. Our concept-based GAN models perform better than their unsupervised models on almost every language pairs for both BLI-1 and BLI-2 dataset. For those languages were the better results are not achieved, the gaps are very small. Figure 3 reports the error reduction rate brought by our method for bilingual lexicon induction on the BLI-1 dataset. It can be clearly seen that the improvement brought by our method is more significant on distant language pairs than on similar language pairs.

Figure 3: Average error reduction of our method compared to unsupervised adversarial method for bilingual lexicon induction on BLI-1 dataset (Lample et al., 2018)

. Since the Finnish-English pair is an outlier for the unsupervised method, we report both the average with and without this pair.

Comparison with supervised baselines:

Our two dictionary-based supervised baselines are strong. In most cases, the preliminary mappings trained by our concept-based model are not comparable with them, but the gab is small. After post-refinement, our method becomes comparable with these supervised methods. For some language pairs, such as French-English and and for English to Finnish, our method performs better.

Comparison with the state-of-the-art:

From the results shown in Table 3, we can see that in most cases, our method works better than previous supervised and unsupervised approaches. However, the performance of Artetxe et al. (2018a) is very strong and their method always works better than ours. Two potential reasons may cause this difference: First, their self-learning framework iteratively fine-tunes the transformation until convergence, while our refinement just runs several iterations.131313As optimisation of the refinement is not the objective of this paper, we follow the work of Lample et al. (2018) and run only five iterations for post-refinement Second, their framework consists of many optimization steps, such as symmetric re-weighting of vectors, steps that we do not have.

Impact of number of concepts:

To understand whether the number of aligned concepts affects our method, we trained our concept-based models on a range of Chinese-English concept-aligned article pairs, from 550 to 10’000. We test them on BLI-1 dataset. Following the trend of performance change shown in Figure 4, we see that when the number of shared concepts reaches 2500, the improvement in accuracy slows down, and is already very close to the result of the model trained from the total number of concepts (Table 1), thus indicating the direction for further future optimizations.

Method en-de en-es en-fi en-it
Supervised Mikolov et al. (2013b) 35.0 27.3 25.9 34.9
Faruqui and Dyer (2014) 37.1 26.8 27.6 38.4
Dinu et al. (2015) 38.9 30.4 29.1 37.7
Xing et al. (2015) 41.3 31.2 28.2 36.9
Artetxe et al. (2016) 41.9 31.4 30.6 39.3
Zhang et al. (2016) 40.8 31.1 28.2 36.7
Artetxe et al. (2017) 40.9 - 28.7 39.7
Smith et al. (2017) 43.3 35.1 29.4 43.1
Artetxe et al. (2018b) 44.1 32.9 44.1 45.3
Unsupervised Zhang et al. (2017) 0.00 0.00 0.00 0.00
Lample et al. (2018) 46.8 35.4 0.38 45.2
Artetxe et al. (2018a) 48.2 37.3 32.6 48.1
Weakly-supervised Our method 47.7 37.2 30.8 46.4
Table 3: Results of bilingual lexicon induction (accuracy % P@1) on BLI-2 dataset, all the results of previous methods come from the paper of Artetxe et al. (2018a)

6 Related Work

Sharing a word embedding space across different languages has proven useful for many cross-lingual tasks, such as machine translation Zou et al. (2013) and cross-lingual dependency parsing Jiang et al. (2015, 2016); Ammar et al. (2016a). Generally, such spaces can be trained directly from bilingual sentence aligned or document aligned text Hermann and Blunsom (2014); Chandar A P et al. (2014); Søgaard et al. (2015); Vulić and Moens (2013). However the performance of directly trained models is limited by their vocabulary size.

Instead of training shared embedding space directly, the work of Mikolov et al. (2013b) has shown that we can also combine two monolingual spaces by applying a linear mapping matrix. The matrix is trained by minimizing the sum of squared Euclidean distances between source and target words of a dictionary. This simple approach has been improved upon in several ways: using canonical correlation analysis to map source and target embeddings Faruqui and Dyer (2014); or by forcing the mapping matrix to be orthogonal Xing et al. (2015).

Recently, efforts have concentrated on how to limit or avoid reliance on dictionaries. Good results were achieved with some drastically minimal techniques. Zhang et al. (2016) achieved good results at bilingual POS tagging, but not bilingual lexicon induction, using only ten word pairs to build a coarse orthonormal mapping between source and target monolingual embeddings. The work of Smith et al. (2017) has shown that a singular value decomposition (SVD) method can produce a competitive cross-lingual mapping by using identical character strings across languages. Artetxe et al. (2017, 2018b) proposed a self-learning framework, which iteratively trains its cross-lingual mapping by using dictionaries trained in previous rounds. The initial dictionary of the self-learning can be reduced to 25 word pairs or even only a list of numerals and still have competitive performance. Furthermore, Artetxe et al. (2018a) extend their self-learning framework to unsupervised models, and build the state-of-the-art for bilingual lexicon induction. Instead of using a pre-build dictionary for initialization, they sort the value of the word vectors in both the source and the target distribution, treat two vectors that have similar permutations as possible translations and use them as the initialization dictionary. Additionally, their unsupervised framework also includes many optimization augmentations, such as stochastic dictionary induction, symmetric re-weighting, among others.

Theoretically, employing GANs for training cross-lingual word embedding is also a promising way to avoid the use of bilingual evidence. As far as we know, Barone (2016) was the first attempt at this approach, but the performance of their model is not competitive. Zhang et al. (2017) enforce the mapping matrix to be orthogonal during the adversarial training and achieve a good performance on bilingual lexicon induction. The main drawback of their approach is that the vocabularies of their training data are small, and the performance of their models drops significantly when they use large training data. The recent model proposed by Lample et al. (2018) is so far the most successful and becomes competitive with previous supervised approaches through a strong CSLS-based refinement to the core mapping matrix trained by GANs. Even in this case, though, without refinement, the core mappings are not as good as hoped for some distant language pairs. More recently, Chen and Cardie (2018) extends the work of Lample et al. (2018) from bilingual setting to multi-lingual setting, instead of training crosslingual word embeddings for only one language pair, their approach allows us to train crosslingual word embeddings for many language pairs at the same time. Another recent piece of work which is similar to Lample et al. (2018) comes from Xu et al. (2018). Their approach can be divided into 2 steps: first, using Wasserstein GAN Arjovsky et al. (2017) to train a preliminary mapping between two monolingual distribution and then minimizing the Sinkhorn Distance across distributions. Although their method performs better than Lample et al. (2018) in several tasks, the improvement mainly comes from the second step, showing that the problem of how to train a better preliminary mapping has not been resolved.

Figure 4: Accuracy of Chinese-English bilingual lexicon induction task for models trained from different concept numbers.

7 Conclusions

In this paper, we propose a weakly-supervised adversarial training method for cross-lingual word embedding mapping. Our approach is based on the intuition that mapping across distant languages is better done at the concept level than at the word level. The method improves the performance over previous unsupervised adversarial methods in many cases, especially for distant languages.


  • Ammar et al. (2016a) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016a. Many Languages, One Parser. Transactions of the Association for Computational Linguistics, 4:431–444.
  • Ammar et al. (2016b) Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. 2016b. Massively Multilingual Word Embeddings. arXiv preprint arXiv:1309.4168.
  • Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein GAN. arXiv preprint arXiv:1309.4168.
  • Artetxe et al. (2016) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance . In 2016 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas, USA. Association for Computational Linguistics.
  • Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In 55th Annual Meeting of the Association for Computational Linguistics, pages 451–462, Vancouver, Canada.
  • Artetxe et al. (2018a) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia. Association for Computational Linguistics.
  • Artetxe et al. (2018b) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b.

    Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations.


    Thirty-Second Conference on Artificial Intelligence

    , pages 5012–5019.
  • Bansal et al. (2014) Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring Continuous Word Representations for Dependency Parsing. In 52nd Annual Meeting of the Association for Computational Linguistics, pages 809–815, Baltimore, Maryland, USA.
  • Barone (2016) Antonio Valerio Miceli Barone. 2016.

    Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders.

    In 1st Workshop on Representation Learning for NLP, pages 121–126, Berlin, Germany.
  • Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135–146.
  • Chandar A P et al. (2014) Sarath Chandar A P, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. 2014. An Autoencoder Approach to Learning Bilingual Word Representations. In 2014 Advances in Neural Information Processing Systems, pages 1853–1861, Montréal Canada.
  • Chen and Cardie (2018) Xilun Chen and Claire Cardie. 2018. Unsupervised Multilingual Word Embeddings. In 2018 Conference on Empirical Methods in Natural Language Processing, pages 261–270. Association for Computational Linguistics.
  • Chen and Gerdes (2017) Xinying Chen and Kim Gerdes. 2017. Classifying languages by dependency structure. typologies of delexicalized universal dependency treebanks. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 54–63, Pisa, Italy. Linköping University Electronic Press.
  • Cisse et al. (2017) Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. 2017. Parseval Networks: Improving Robustness to Adversarial Examples. In

    34th International Conference on Machine Learning

    , pages 854–863, Sydney, Australia. PMLR.
  • Dinu et al. (2015) Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In 3rd International Conference on Learning Representations, pages 1–10, Toulon, France.
  • Faruqui and Dyer (2014) Manaal Faruqui and Chris Dyer. 2014. Improving Vector Space Word Representations Using Multilingual Correlation. In 14th Coference of the European Chapter of the Association for Computational Linguistics, pages 462–471, Gothenburg, Sweden.
  • Goodfellow et al. (2014) Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Networks. In Advances in Neural Information Processing Systems 27, pages 2672–2680, Montréal, Canada.
  • Hermann and Blunsom (2014) Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual Models for Compositional Distributed Semantics. In 52nd Annual Meeting of the Association for Computational Linguistics, pages 58–68, Baltimore, Maryland, USA.
  • Jiang et al. (2015) Guo Jiang, Che Wanxiang, Yarowsky David, Wang Haifeng, and Liu Ting. 2015. Cross-lingual dependency parsing based on distributed representations. In 53rd Annual Meeting of the Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing, pages 1234–1244, Beijing, China.
  • Jiang et al. (2016) Guo Jiang, Che Wanxiang, Yarowsky David, Wang Haifeng, and Liu Ting. 2016. A distributed representation-based framework for cross-lingual transfer parsing. Journal of Artificial Intelligence Research, 55:995–1023.
  • Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In 45th Annual Meeting of the Association for Computational Linguistics, Prague, Czech Republic.
  • Lample et al. (2018) Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In 6th International Conference on Learning Representations, pages 1–14, Vancouver, Canada.
  • Luong et al. (2013) Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In 17th Conference on Computational Natural Language Learning, pages 104–113, Sofia, Bulgaria.
  • Mi et al. (2016) Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016.

    Coverage Embedding Models for Neural Machine Translation.

    In 2016 Conference on Empirical Methods in Natural Language Processing, pages 955–960, Austin, Texas, USA.
  • Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1309.4168.
  • Mikolov et al. (2013b) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting Similarities among Languages for Machine Translation . arXiv preprint arXiv:1309.4168, (Computation and Language):1–10.
  • Mikolov et al. (2013c) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013c. Distributed Representations of Words and Phrases and their Compositionality. In 26th International Conference on Neural Information Processing Systems, pages 3111–3119, Lake Tahoe, Nevada, USA.
  • Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove - Global Vectors for Word Representation. In 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543, Doha, Qatar.
  • Radovanović et al. (2010) Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović. 2010.

    Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data.

    Journal of machine learning research, 11(Sep):2487–2531.
  • Smith et al. (2017) Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In 15th International Conference on Learning Representations, pages 1–10, Toulon, France.
  • Søgaard et al. (2015) Anders Søgaard, Zeljko Agić, Héctor Martínez Alonso, Barbara Plank, Bernd Bohnet, and Anders Johannsen. 2015. Inverted indexing for cross-lingual NLP. In 53rd Annual Meeting of the Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing, pages 1713–1722, Beijing, China.
  • Vulić and Moens (2013) Ivan Vulić and Marie-Francine Moens. 2013. A Study on Bootstrapping Bilingual Vector Spaces from Non-Parallel Data (and Nothing Else). In 2013 Conference on Empirical Methods in Natural Language Processing, pages 1613–1624, Seattle, Washington, USA.
  • Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation. In 2015 Annual Conference of the North American Chapter of the ACL, pages 1006–1011, Denver, Colorado, USA. Association for Computational Linguistics.
  • Xu et al. (2018) Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. 2018. Unsupervised Cross-lingual Transfer of Word Embedding Spaces. In 2018 Conference on Empirical Methods in Natural Language Processing, pages 2465–2474. Association for Computational Linguistics.
  • Zhang et al. (2017) Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial Training for Unsupervised Bilingual Lexicon Induction. In 55th Annual Meeting of the Association for Computational Linguistics, pages 1959–1970, Vancouver, Canada. Association for Computational Linguistics.
  • Zhang et al. (2016) Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. 2016. Ten Pairs to Tag - Multilingual POS Tagging via Coarse Mapping between Embeddings. In 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 12–17, San Diego, California, USA. Association for Computational Linguistics.
  • Zou et al. (2013) Will Y Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual Word Embeddings for Phrase-Based Machine Translation. In Conference on Empirical Methods in Natural Language Processing, pages 1393–1398, Seattle, Washington, USA.