Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

by   Mikel Artetxe, et al.

We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different language families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared BPE vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting sentence embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our approach sets a new state-of-the-art on zero-shot cross-lingual natural language inference for all the 14 languages in the XNLI dataset but one. We also achieve very competitive results in cross-lingual document classification (MLDoc dataset). Our sentence embeddings are also strong at parallel corpus mining, establishing a new state-of-the-art in the BUCC shared task for 3 of its 4 language pairs. Finally, we introduce a new test set of aligned sentences in 122 languages based on the Tatoeba corpus, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low-resource languages. Our PyTorch implementation, pre-trained encoder and the multilingual test set will be freely available.


page 1

page 2

page 3

page 4


XNLI: Evaluating Cross-lingual Sentence Representations

State-of-the-art natural language processing systems rely on supervision...

Bayesian multilingual topic model for zero-shot cross-lingual topic identification

This paper presents a Bayesian multilingual topic model for learning lan...

Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining

Existing models of multilingual sentence embeddings require large parall...

A Corpus for Multilingual Document Classification in Eight Languages

Cross-lingual document classification aims at training a document classi...

Emu: Enhancing Multilingual Sentence Embeddings with Semantic Specialization

We present Emu, a system that semantically enhances multilingual sentenc...

Explicit Alignment Objectives for Multilingual Bidirectional Encoders

Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) a...

OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval

Aligning parallel sentences in multilingual corpora is essential to cura...

1 Introduction

While the recent advent of deep learning has led to impressive progress in Natural Language Processing (NLP), these techniques are known to be particularly data hungry, limiting their applicability in many practical scenarios. An increasingly popular approach to alleviate this issue is to first learn general language representations on unlabeled data, which are then integrated in task-specific downstream systems. This approach was first popularized by word embeddings

(Mikolov et al., 2013b; Pennington et al., 2014), but has recently been superseded by sentence-level representations (Conneau et al., 2017; Peters et al., 2018; Devlin et al., 2018). Nevertheless, all these works learn a separate model for each language and are thus unable to leverage information across different languages, greatly limiting their potential performance for low-resource languages.

In this work, we are interested in universal language agnostic sentence embeddings

, that is, vector representations of sentences that are general with respect to two dimensions: the input language and the NLP task. The motivations for such a representation are multiple: the hope that languages with limited resources benefit from joint training over many languages, the desire to perform zero-shot transfer of an NLP model from one language (e.g. English) to another, and the possibility to handle code-switching. We achieve this by using a single encoder that can handle multiple languages, so that semantically similar sentences in different languages are close in the resulting embedding space.

Most research in multilingual NLP focuses on high-resource languages like Chinese, Arabic or major European languages, and is usually limited to a few (most often only two) languages. In contrast, we learn joint sentence representations for 93 different languages, including under-resourced and minority languages (see Tables 1 and 2). Our system is trained on freely available parallel texts only. The contributions of this paper are as follows:

  • We substantially improve on previous work to learn joint multilingual sentence representations. We learn one shared encoder that can handle 93 different languages. All languages are jointly embedded in a shared space, in contrast to most other works which usually consider separate English/foreign alignments. We cover 34 language families and 28 different scripts.

  • We outperform the state-of-the-art on zero-shot cross-lingual natural language inference (XNLI dataset) and classification (MLDoc dataset), bitext mining (BUCC dataset) and multilingual similarity search (Tatoeba dataset), for almost all considered languages. These results were obtained with a single pre-trained BiLSTM encoder for all 93 languages and tasks, without any fine-tuning.

  • We define a new test set based on the freely available Tatoeba corpus and provide baseline results for 122 languages. We report accuracy for multilingual similarity search on this test set, but the corpus could also be used for MT evaluation.

The remaining of this paper is organized as follows. In the next section, we first summarize related work. Section 3 then describe our approach in detail. All experimental results are given in Sections 4 and 5, and the paper concludes with a discussion and directions for future research. Dataset details and additional result analysis can be found in the appendix.

Figure 1: Architecture of our system to learn multilingual sentence embeddings.

2 Related work

Following the success of word embeddings (Mikolov et al., 2013b; Pennington et al., 2014)

, there has been an increasing interest in learning continuous vector representations of longer linguistic units like sentences. These sentence embeddings are commonly obtained using a Recurrent Neural Network (RNN) encoder, which is typically trained in an unsupervised way over large collections of unlabelled corpora. For instance, the skip-thought model of

Kiros et al. (2015) couple the encoder with an auxiliary decoder, and train the entire system end-to-end to predict the surrounding sentences over a large collection of books. It was later shown that more competitive results could be obtained by training the encoder over labeled Natural Language Inference (NLI) data (Conneau et al., 2017). This was recently extended to multitask learning, combining different training objectives like that of skip-thought, NLI and machine translation (Cer et al., 2018; Subramanian et al., 2018).

While the previous methods consider a single language at a time, multilingual representations have attracted a large attention in recent times. Most of this research focuses on cross-lingual word embeddings (Ruder et al., 2017), which are commonly learned jointly from parallel corpora (Gouws et al., 2015; Luong et al., 2015). An alternative approach that is becoming increasingly popular is to train word embeddings independently for each language over monolingual corpora, and then map them to a shared space based on a bilingual dictionary (Mikolov et al., 2013a; Artetxe et al., 2018a) or even in a fully unsupervised manner Conneau et al. (2018a); Artetxe et al. (2018b). Cross-lingual word embeddings are often used to build bag-of-word representations of longer linguistic units by taking their respective centroid (Klementiev et al., 2012). While this approach has the advantage of requiring a weak (or even no) cross-lingual signal, it has been shown that the resulting sentence embeddings works rather poorly in practical cross-lingual transfer settings (Conneau et al., 2018b).

A more competitive approach that we follow here is to use a sequence-to-sequence encoder-decoder architecture (Schwenk and Douze, 2017; Hassan et al., 2018)

. The full system is trained end-to-end on parallel corpora akin to neural machine translation: the encoder maps the source sequence into a fixed-length vector representation, which is used by the decoder to create the target sequence. This decoder is then discarded, and the encoder is kept to embed sentences in any of the training languages. While some proposals use a separate encoder for each language

(Schwenk and Douze, 2017), sharing a single encoder for all languages also gives strong results (Schwenk, 2018a).

Nevertheless, most existing work is either limited to few, rather close languages Schwenk and Douze (2017) or, more commonly, consider pairwise joint embeddings with English and one foreign language only España-Bonet et al. (2017); Guo et al. (2018). To the best of our knowledge, all existing work on learning multilingual representations for a large number of languages is limited to word embeddings (Ammar et al., 2016), ours being the first paper exploring massively multilingual sentence representations.

Finally, while all the previous approaches learn a fixed-length representation for each sentence, a recent research line has obtained very strong results using variable-length representations instead, consisting of contextualized embeddings of the words in the sentence (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2018). For that purpose, these methods train either an RNN or self-attentional encoder over unnanotated corpora using some form of language modeling. A classifier can then be learned on top of the resulting encoder, which is commonly further fine-tuned during this supervised training. Despite the strong performance of these approaches in monolingual settings, we argue that fixed-length approaches provide a more generic, flexible and compatible representation form for our multilingual scenario,111For instance, there is not always a one-to-one correspondence among words in different languages (e.g. a single word of a morphologically complex language might correspond to several words of a morphologically simple language), so having a separate vector for each word might not transfer as well across languages. and our model indeed outperforms the multilingual BERT model (Devlin et al., 2018) in zero-shot transfer (see Section 4.1).

3 Proposed method

We use a single, language agnostic BiLSTM encoder to build our sentence embeddings, which is coupled with an auxiliary decoder and trained over parallel corpora. From Section 3.1 to 3.3, we describe its architecture, our training strategy to scale to up to 93 languages, and the training data used for that purpose.

3.1 Architecture

Figure 1 illustrates the architecture of the proposed system, which is based on Schwenk (2018a)

. As it can be seen, sentence embeddings are obtained by applying a max-pooling operation over the output of a BiLSTM encoder. These sentence embeddings are used to initialize the decoder LSTM through a linear transformation, and are also concatenated to its input embeddings at every time step. Note that there is no other connection between the encoder and the decoder, as we want all relevant information of the input sequence to be captured by the sentence embedding.

We use a single encoder and decoder in our system, which are shared by all languages involved. For that purpose, we build a joint byte-pair encoding (BPE) vocabulary with 50k operations, which is learned on the concatenation of all training corpora. This way, the encoder has no explicit signal on what the input language is, encouraging it to learn language independent representations. In contrast, the decoder takes a language ID embedding that specifies the language to generate, which is concatenated to the input and sentence embeddings at every time step.

Scaling up to almost hundred languages, which use very different syntax, writing scripts and linguistic concepts, naturally calls for an encoder with sufficient capacity. In this paper, we limit our study to a stacked BiLSTM with 1 to 5 layers, each 512-dimensional. The resulting sentence representations (after concatenating both directions) are 1024 dimensional. The decoder has always one layer of dimension 2048. The input embedding size is set to 320, while the language ID embedding has 32 dimensions.

Details Training [-1pt] corpus [-2pt] size Tatoeba Error [%] Tatoeba [-1pt] test set [-2pt] size
ISO3 ISO2 Name Family Script en xx xx en
afr af Afrikaans Germanic Latin 67k 11.20 9.90 1000
sqi sq Albanian Albanian Latin 3.2M 1.80 2.30 1000
amh am Amharic Ethopian Ge’ez 88k 60.71 55.36 168
ara ar Arabic Arabic Arabic 8.2M 8.30 7.80 1000
aym ay Aymara Aymaran Latin 14k n/a n/a
aze az Azerbaijani Turkic Latin; Cyrillic; Persian 254k 44.10 23.90 1000
eus eu Basque Isolate Latin 1.2M 5.70 5.00 1000
ben bn Bengali Indo-Aryan Eastern-Nagari 913k 10.80 10.00 1000
ber ber Berber languages Berber Latin 62k 29.80 33.70 1000
nob nb Bokmål Norwegian Germanic Latin 4.1M 1.30 1.10 1000
bos bs Bosnian Slavic Latin 4.2M 3.95 3.11 354
bre br Breton Celtic Latin 29k 83.50 84.90 1000
bul bg Bulgarian Slavic Cyrillic 4.9M 4.50 5.40 1000
cat ca Catalan Romance Latin 813k 4.00 4.20 1000
cmn zh Chinese mandarin Chinese Chinese 8.3M 4.10 5.00 1000
swh sw (Coastal) Swahili Niger-Congo Latin 173k 45.64 39.23 390
hrv hr Croatian Slavic Latin 4.0M 2.80 2.70 1000
ces cs Czech Slavic Latin 5.5M 3.10 3.80 1000
dan da Danish Germanic Latin 7.9M 3.90 4.00 1000
nld nl Dutch Germanic Latin 8.4M 3.10 4.30 1000
eng en English Germanic Latin 2.6M n/a n/a
epo eo Esperanto constructed Latin 397k 2.70 2.80 1000
est et Estonian Uralic Latin 5.3M 3.20 3.40 1000
fin fi Finnish Uralic Latin 7.9M 3.70 3.70 1000
fra fr French Romance Latin 8.8M 4.40 4.30 1000
glg gl Galician Romance Latin 349k 4.60 4.40 1000
kat ka Georgian Kartvelian Georgian 296k 60.32 67.83 746
deu de German Germanic Latin 8.7M 0.90 1.00 1000
ell el Greek Hellenic Greek 6.5M 5.30 4.80 1000
hau ha Hausa Afro-Asiatic Latin; Arabic 127k n/a n/a
heb he Hebrew Semitic Hebrew 4.1M 8.10 7.60 1000
hin hi Hindi Indo-Aryan Devanagari 288k 5.80 4.80 1000
hun hu Hungarian Uralic Latin 5.3M 3.90 4.00 1000
isl is Icelandic Germanic Latin 2.0M 4.40 4.40 1000
ind id Indonesian Malayo-Polynesian Latin 4.3M 5.20 5.80 1000
pes ps Iranian Persian (Farsi) Iranian Persian 4.9M 7.20 6.00 1000
ita it Italian Romance Latin 8.3M 4.60 4.80 1000
jpn ja Japanese Japonic Kanjii 3.2M 3.90 5.40 1000
kab Kabyle Berber Latin (modified) 15k 39.10 44.70 1000
kor ko Korean Koreanic Hangul 1.4M 10.60 11.50 1000
kur ku Kurdish Iranian Latin; Persian 50k 80.24 85.37 410
lvs lv Latavian Baltic Latin 2.0M 4.50 4.70 1000
lat la Latin Romance Latin 19k 41.60 41.50 1000
lit lt Lithuanian Baltic Latin 3.2M 4.10 3.40 1000
nds Low German / Saxon Germanic Latin 12k 18.60 15.60 1000
mkd mk Macedonian Slavic Cyrillic 4.2M 5.20 5.40 1000
mlg mg Malagasy Malayo-Polynesian Latin 355k n/a n/a
zsm ms Malay Malayo-Polynesian Latin 2.9M 3.40 3.80 1000
mal ml Malayalam Dravidian Malayalam 373k 3.35 2.91 687
div dv Maldivian (Divehi) Indo-Aryan Thaana 90k n/a n/a
mar mr Marathi Indo-Aryan Devanagari 31k 9.00 8.00 1000
pol pl Polish Slavic Latin 5.5M 2.00 2.40 1000
por pt Portuguese Romance Latin 8.3M 4.70 4.90 1000
ron ro Romanian; Moldavian Romance Latin 4.9M 2.50 2.70 1000
rus ru Russian Slavic Cyrillic 9.3M 4.90 5.90 1000
srp sr Serbian Slavic Cyrillic; Latin 4.0M 4.30 5.00 1000
snd sd Sindhi Iranian Persian; Devanagari 91k n/a n/a
sin si Sinhala Indo-Aryan Sinhala 796k n/a n/a
slk sk Slovak Slavic Latin 5.2M 3.10 3.70 1000
slv sl Slovenian Slavic Latin 5.2M 4.50 3.77 823
som so Somali Cushitic Latin 85k n/a n/a
spa es Spanish Romance Latin 4.8M 1.90 2.10 1000
swe sv Swedish Germanic Latin 7.8M 3.60 3.20 1000
tgl tl Tagalog Malayo-Polynesian Latin 36k 47.40 51.50 1000
tgk tg Tajik Iranian Cyrillic 124k n/a n/a
tam ta Tamil Dravidian Tamil 42k 31.60 29.64 307
tat tt Tatar Turkic Cyrillic 119k 72.00 65.70 1000
tel te Telugu Dravidian Telugu 33k 18.38 22.22 234
tha th Thai Kra-Dai Thai 4.1M 4.93 4.20 548
tur tr Turkish Turkic Latin 5.7M 2.30 2.60 1000
uig ug Uighur Turkic Arabic 88k 59.90 49.60 1000
ukr uk Ukrainian Slavic Cyrillic 1.4M 5.80 5.10 1000
urd ur Urdu Indo-Aryan Arabic 746k 20.00 16.20 1000
uzb uz Uzbek Turkic Latin; Cyrillic 118k 82.24 80.37 428
vie vi Vietnamese Vietic Latin 4.0M 3.40 3.00 1000
Table 1: List of the 75 out of 93 languages used to trained the proposed model with at least 10k training examples, along with their language family, writing system, the resulting similarity error rate on the Tatoeba test set, and the number of sentences in it. Dashes denote language pairs excluded for containing less than 100 test sentences.
Details Training [-1pt] corpus [-2pt] size Tatoeba Error [%] Tatoeba [-1pt] test set [-2pt] size
ISO3 ISO2 Name Family Script en xx xx en
hye hy Armenian Armenian Armenian 6k 59.97 67.79 742
bel be Belarusian Slavic Cyrillic 5k 31.20 36.50 1000
mya my Burmese Sino-Tibetan Burmese 2k n/a n/a
dtp Central/Kadazan Dusun Malayo-Polynesian Latin 1k 92.10 93.50 1000
khm km Central Khmer Khmer Khmer 625 77.01 81.72 722
cbk Chavacano Creole, Romance Latin 1k 24.20 21.70 1000
kzj Coastal Kadazan Malayo-Polynesian Latin 560 91.60 94.10 1000
cor kw Cornish Celtic Latin 2k 91.90 93.20 1000
mhr Eastern Mari Uralic Cyrillic 1k 87.70 91.50 1000
ido io Ido constructed Latin 3k 17.40 15.20 1000
ina ia Interlingua constructed Latin 9k 5.40 4.10 1000
ile ie Interlingue constructed Latin 3k 14.70 12.80 1000
gle ga Irish Irish Latin 732 93.80 95.80 1000
kaz kk Kazakh Turkic Cyrillic 4k 80.17 82.61 575
lfn Lingua Franca Nova constructed Latin 2k 35.90 35.10 1000
oci oc Occitan (post 1500) Romance Latin 3k 39.20 38.40 1000
wuu Wu Chinese Chinese Chinese 2k 25.80 25.20 1000
yue Yue Chinese Chinese Chinese 4k 37.00 38.90 1000
Table 2: List of the 18 very low-resource languages included during training of the proposed model, along with their language family, writing system, the resulting similarity error rate on the Tatoeba test set, and the number of sentences in it. Dashes denote language pairs excluded for containing less than 100 test sentences.

3.2 Training strategies

In preceding work Schwenk and Douze (2017); Schwenk (2018a), each sentence at the input was jointly translated into all other languages. While this approach was shown to learn high-quality representations, it poses two obvious drawbacks when trying to scale to a large number of languages. First, it requires an N-way parallel corpus, which is difficult to obtain for all languages. Second, it has a quadratic cost with respect to the number of languages, making training prohibitively slow as the number of languages is increased. In our preliminary experiments, we observed that similar results can be obtained by using less target languages – two seem to be enough.222Note that, if we had a single target language, the only way to train the encoder for that language would be auto-encoding, which we observe to work poorly. Having two target languages avoids this problem. At the same time, we relax the requirement for N-way parallel corpora by considering independent alignments with the two target languages, e.g. we do not require each source sentence to be translated into the two target languages.

Training minimizes the cross-entropy loss on the training corpus, alternating over all combinations of the languages involved. For that purpose, we use Adam with a constant learning rate of 0.001 and dropout set to 0.1, and train for a fixed number of epochs. Our implementation is based on

fairseq,333 and we make use of its multi-GPU support to train on 16 NVIDIA V100 GPUs with a total batch size of 128,000 tokens. Unless otherwise specified, we train our model for 17 epochs, which takes about 5 days. Stopping training early decreases the overall performance only slightly.

3.3 Training data and pre-processing

As described in Section 3.2, training requires bitexts aligned with two target languages. We choose English and Spanish for that purpose, as most of the data is aligned with these languages.444Note that it is not necessary that all input languages are systematically aligned with both target languages. Once we have several languages with both alignments, the joint embedding is well conditioned, and we can add more languages with one alignment only, usually English. We collect training corpora for 93 input languages by combining the Europarl, United Nations, OpenSubtitles2018, Global Voices, Tanzil and Tatoeba corpus, which are all publicly available on the OPUS website555 Tiedemann (2012). Appendix A provides a more detailed description of this training data, while Tables 1 and 2 summarize the list of all languages used for training, their language family, writing script and the size of the bitexts. Our training data comprises a total of 223 million parallel sentences.

In preliminary experiments, we observed that the domain of the training data played a key role in the performance of our sentence embeddings in different tasks. Some tasks (BUCC, MLDoc) tend to perform better when the encoder is trained on long and formal sentences, whereas other tasks (XNLI, Tatoeba) benefit from training on shorter and more informal sentences. In an attempt to achieve a general purpose sentence encoder that performs well on all tasks, we aimed at balancing the size of training corpora with long and short sentences. For that purpose, we used at most two million sentences from OpenSubtitles, although more data is available for some languages.

All pre-processing is done with Moses tools:666 punctuation normalization, removing non-printing characters and tokenization. As the only exception, Chinese and Japanese texts were segmented with Jieba777 and Mecab,888 respectively. All the languages are kept in their original script with the exception of Greek, which we romanize into the Latin alphabet.

4 Experimental evaluation

In contrast with the well-established evaluation frameworks for English sentence representations Conneau et al. (2017); Wang et al. (2018)

, there is not yet a commonly accepted standard to evaluate multilingual sentence embeddings. The most notable effort in this regard is probably the XNLI corpus

Conneau et al. (2018b), an NLI test set similar to MultiNLI Williams et al. (2017) for which the premises and hypotheses were translated into 14 languages by professional translators. We train an NLI classifier on top of our multilingual sentence embedding using English training data, and evaluate its zero-shot transfer performance in the remaining languages (Section 4.1). So as to obtain a more complete picture of the behavior of our multilingual sentence representations, we also evaluate them in cross-lingual document classification (MLDoc, Section 4.2), and bitext mining (BUCC, Section 4.3). However, all these datasets only cover a subset of our 93 languages, so we also introduce a new test set for multilingual similarity search in 122 languages, including several languages for which we have no training data but whose language family is covered (Section 4.4). We remark that we use the same pre-trained BiLSTM encoder for all tasks and languages without any fine-tuning.

4.1 XNLI: cross-lingual NLI

fr es de el bg ru tr ar vi th zh hi sw ur
Zero-Shot Transfer, one NLI system for all languages:
Conneau et al. (2018c) X-BiLSTM 73.7 67.7 68.7 67.7 68.9 67.9 65.4 64.2 64.8 66.4 64.1 65.8 64.1 55.7 58.4
X-CBOW 64.5 60.3 60.7 61.0 60.5 60.4 57.8 58.7 57.5 58.8 56.9 58.8 56.3 50.4 52.2
BERT uncased Transformer 81.4 74.3 70.5 62.1 63.8 58.3
Proposed method BiLSTM 73.9 71.9 72.9 72.6 73.1 74.2 71.5 69.7 71.4 72.0 69.2 71.4 65.5 62.2 61.0
Translate test, one English NLI system:
Conneau et al. (2018c) BiLSTM 73.7 70.4 70.7 68.7 69.1 70.4 67.8 66.3 66.8 66.5 64.4 68.3 64.2 61.8 59.3
BERT uncased Transformer 81.4 74.9 74.4 70.4 70.1 62.1
Translate train, separate NLI systems for each language:
Conneau et al. (2018c) BiLSTM 73.7 68.3 68.8 66.5 66.4 67.4 66.5 64.5 65.8 66.0 62.8 67.0 62.1 58.2 56.6
BERT cased Transformer 81.9 77.8 75.9 70.7 68.9 76.6 61.6
Table 3: Accuracies on the test set of the XNLI cross-lingual natural language inference task. All results from Conneau et al. (2018c) correspond to max-pooling, which outperforms the last-state variant in all cases. Results which imply the use of MT do not use a multilingual model and are not directly comparable with zero-shot transfer. Overall best results are in bold, the best ones in each group are underlined.
Results for BERT Devlin et al. (2018) are extracted from its GitHub README99footnotemark: 9
Monolingual BERT model for Thai from

NLI has become a widely used task to evaluate sentence representations (Bowman et al., 2015; Williams et al., 2017). Given two sentences, a premise and a hypothesis, the task consists in deciding whether there is an entailment, contradiction or neutral relationship between them. XNLI is a recent effort to create a dataset similar to the English MultiNLI for several languages Conneau et al. (2018b). 2,500 development and 5,000 test sentences have been translated from English into 14 languages by professional translators, making results across different languages directly comparable. Note that no human translated training data is provided; instead, different systems are to use English training data from MultiNLI, and their transfer performance is evaluated on the rest of languages.

We train a classifier on top of our multilingual encoder using the usual combination of the two sentence embeddings: , where and

are the premise and hypothesis. For that purpose, we use a feed-forward neural network with two hidden layers of size 512 and 384, trained with Adam. All hyperparameters were optimized on the English XNLI development corpus, and then, the same classifier was applied to all languages of the XNLI test set. As such, we did not use any training or development data in any of the foreign languages. Note, moreover, that the multilingual sentence embeddings are fixed and not fine-tuned on the task or the language.

We report our results in Table 9, along with several baselines from Conneau et al. (2018b) and the recently released multilingual BERT model (Devlin et al., 2018).999Note that the multilingual variant of BERT is not discussed in its paper (Devlin et al., 2018). Instead, the reported results were extracted from the README of the official GitHub project at on 12/19/2018. As it can be seen, our proposed method establishes a new state-of-the-art in zero-shot cross-lingual transfer (i.e. training a classifier on English data and applying it to all other languages) for all languages but Spanish. Our transfer results are strong and homogeneous across all languages: for 11 of them, the zero-short performance is (at most) 5% lower than the one on English, including distant languages like Arabic, Chinese and Vietnamese, and we also achieve remarkable good results on low-resource languages like Swahili. In contrast, BERT achieves excellent results on English, outperforming our system by 7.5 points, but its zero-shot cross-lingual transfer performance is much weaker. For instance, the loss in accuracy for both Arabic and Chinese is 2.5 points for our system, compared to 19.3 and 17.6 points for BERT. Finally, we also outperform all baselines of Conneau et al. (2018b) by a substantial margin, with the additional advantage that we use a single pre-trained encoder, whereas X-BiLSTM learns a separate encoder for each language by aligning it to the English one.

For completeness, we also provide results that include the use of Machine Translation (MT). This can be done in two ways: 1) translate the test data into English and apply the English NLI classifier, or 2) translate the English training data and train a language specific NLI classifier for each language. It should be stressed that we are not evaluating multilingual sentence embeddings anymore, but rather the quality of the MT system and a monolingual model. Moreover, the use of MT incurs in an important overhead with either strategy: translating test makes inference substantially more expensive, whereas translating train results in a separate model for each language. As shown in Table 9, our approach outperforms all translation baselines of Conneau et al. (2018b) with the exception of Urdu. We also outperform MT BERT for Arabic and Thai, and are very close for Urdu.

Finally, it is worth mentioning that, thanks to its multilingual nature, our system can also handle premises and hypothesis in different languages. As reported in Appendix B, the proposed method obtains very strong results in these settings, even for distant language combinations like French-Chinese.

4.2 MLDoc: cross-lingual classification

de es fr it ja ru zh
Schwenk and Li (2018a) MultiCCA + CNN 92.20 81.20 72.50 72.38 69.38 67.63 60.80 74.73
BiLSTM (Europarl) 88.40 71.83 66.65 72.83 60.73 - - -
BiLSTM (UN) 88.83 - 69.50 74.52 - - 61.42 71.97
Proposed method 89.93 84.78 77.33 77.95 69.43 60.30 67.78 71.93
Table 4: Accuracies on the MLDoc zero-shot cross-lingual document classification task (test set).
de-en fr-en ru-en zh-en de-en fr-en ru-en zh-en
Azpeitia et al. (2017) 83.33 78.83 - - 83.74 79.46 - -
Grégoire and Langlais (2017) - 20.67 - - - 20 - -
Zhang and Zweigenbaum (2017) - - - 43.48 - - - 45.13
Azpeitia et al. (2018) 84.27 80.63 80.89 76.45 85.52 81.47 81.30 77.45
Bouamor and Sajjad (2018) - 75.2 - - - 76.0 - -
Chongman Leong and Chao (2018) - - - 58.54 - - - 56
Schwenk (2018b) 76.1 74.9 73.3 71.6 76.9 75.8 73.8 71.6
Artetxe and Schwenk (2018) 94.84 91.85 90.92 91.04 95.58 92.89 92.03 92.57
Proposed method 95.43 92.40 92.29 91.20 96.19 93.91 93.30 92.27
Table 5: F1 scores on the BUCC mining task.

Cross-lingual document classification is a typical application of multilingual representations. In order to evaluate our sentence embeddings in this task, we use the MLDoc dataset of Schwenk and Li (2018b), which is an improved version of the Reuters benchmark (Lewis et al., 2004; Klementiev et al., 2012) with uniform class priors and a wider language coverage. There are 1,000 training and development documents and 4,000 test documents for each language, divided in 4 different genders. Just as with the XNLI evaluation, we consider the zero-shot transfer scenario: we train a classifier on top of our multilingual encoder using the English training data, optimizing hyper-parameters on the English development set, and evaluating the resulting system in the remaining languages. We use a feed-forward neural network with one hidden layer of 10 units.

As shown in Table 4, our system obtains the best published results for 5 of the 7 transfer languages. We believe that our weaker performance on Japanese can be attributed to the domain and sentence length mismatch between MLDoc and the parallel corpus we use for this language (OpenSubtitles).

4.3 BUCC: bitext mining

Bitext mining is another natural application for multilingual sentence embeddings. Given two comparable corpora in different languages, the task consists in identifying sentence pairs that are translations of each other. For that purpose, one would commonly score sentence pairs by taking the cosine similarity of their respective embeddings, so parallel sentences can be extracted through nearest neighbor retrieval and filtered by setting a fixed threshold over this cosine score

(Schwenk, 2018a). However, it was recently shown that this approach suffers from scale inconsistency issues (Guo et al., 2018), and Artetxe and Schwenk (2018) proposed the following alternative score addressing it:

where and are the source and target sentences, and denotes the nearest neighbors of in the other language. The paper explores different margin functions, with ratio () yielding the best results. This notion of margin is related to CSLS as proposed in Conneau et al. (2018a). The reader is referred to Artetxe and Schwenk (2018) for a detailed discussion.

We use this method to evaluate our sentence embeddings on the BUCC mining task (Zweigenbaum et al., 2017, 2018), using exact same hyper-parameters as Artetxe and Schwenk (2018). The goal is to extract parallel sentences from a comparable corpus between English and four foreign languages: German, French, Russian and Chinese. The dataset consists of 150K to 1.2M sentences for each language, split into a sample, training and test set, with about 2–3% of the sentences being parallel.

As shown in our results in Table 5, our sentence embeddings establish a new state-of-the-art for all language pairs with the exception of English-Chinese test. Quite remarkably, we also outperform Artetxe and Schwenk (2018) themselves, who use two separate models covering 4 languages each (English/French/Spanish/German and English/French/Russian/Chinese). The average performance over the four languages increased from 93.27 to 93.92. Not only are our results better, but our model also covers many more languages, so it can potentially be used to mine bitext for any combination of the 93 languages supported.

4.4 Tatoeba: similarity search

While XNLI, MLDoc and BUCC are well established benchmarks with comparative results available, they only cover a small subset of our 93 languages. So as to better assess the performance of our model in all these different languages, we introduce a new test set of similarity search for 122 languages based on the Tatoeba corpus. The dataset consists of up to 1,000 English-aligned sentence pairs for each language. Appendix C describes how the dataset was constructed in more details. Evaluation is done by finding the nearest neighbor for each sentence in the other language according to cosine similarity and computing the error rate.

We report our results in Tables 1 and 2. Contrasting these results with those of XNLI, one would assume that similarity error rates below 5% are indicative of strong downstream performance.101010We consider the average of enxx and xxen This is the case for 37 languages, while there are 48 languages with an error rate below 10% and 55 with less than 20%, covering 22 different families and 15 different scripts. There are only 15 languages with error rates above 50%.

We believe that our competitive results for many low-resource languages are indicative of the benefits of joint training, which is also supported by our ablation results in Section 5.3. In relation to that, Appendix E reports similarity search results for 29 additional languages without any training data, showing that our encoder can also generalize to unseen languages to some extent as long as it was trained in related languages.

5 Ablation experiments

In this section, we explore different variants of our approach and study the impact on the performance for all our evaluation tasks. We report average results across all languages. For XNLI, we also report the accuracy on English.

5.1 Encoder depth

Table 6 reports the performance on the different tasks for encoders with one, three or five layers. We were not able to achieve good convergence with deeper models. It can be seen that all tasks benefit from deeper models, in particular XNLI and Tatoeba, suggesting that a single layer BiLSTM has not enough capacity to encode so many languages.

Depth Tatoeba BUCC MLDoc XNLI-en XNLI-xx
Err [%] F1 Acc [%] Acc [%] Acc [%]
1 37.96 89.95 69.42 70.94 64.54
3 28.95 92.28 71.64 72.83 68.43
5 26.31 92.83 72.79 73.67 69.92
Table 6: Impact of the depth of the BiLSTM encoder.

5.2 Multitask learning

Multitask learning has been shown to be helpful to learn English sentence embeddings Subramanian et al. (2018); Cer et al. (2018). The most important task in this approach is arguably NLI, so we explored adding an additional NLI objective to our system with different weighting schemes. As shown in Table 7, the NLI objective leads to a better performance on the English NLI test set, but this comes at the cost of a worse cross-lingual transfer performance in XNLI and Tatoeba. The effect in BUCC is negligible.

NLI Tatoeba BUCC MLDoc XNLI-en XNLI-xx
obj. Err [%] F1 Acc [%] Acc [%] Acc [%]
- 26.31 92.83 72.79 73.67 69.92
1 26.89 93.01 74.51 73.71 69.10
2 28.52 93.06 71.90 74.65 67.75
3 27.83 92.98 73.11 75.23 61.86
Table 7: Multitask training with an NLI objective and different weightings.

5.3 Number of training languages

#langs WMT BUCC MLDoc XNLI-en XNLI-xx
Err [%] F1 Acc [%] Acc [%] Acc [%]
All (93) 0.54 92.83 72.79 73.67 69.92
Eval (18) 0.59 92.91 75.63 72.99 68.84
Table 8: Comparison between training on 93 languages and training on the 18 evaluation languages only.

So as to better understand how our architecture scales to a large amount of languages, we train a separate model on a subset of 18 evaluation languages, and compare it to our main model trained on 93 languages. We replaced the Tatoeba corpus with the WMT 2014 test set to evaluate the multilingual similarity error rate. This covers English, Czech, French, German and Spanish, so results between both models are directly comparable. As shown in Table 8, the full model equals or outperforms the one covering the evaluation languages only for all tasks but MLDoc. This suggests that the joint training also yields to overall better representations.

6 Conclusions

In this paper, we propose an architecture to learn multilingual sentence embeddings for 93 languages. We use a single language-agnostic BiLSTM encoder for all languages, which is trained on publicly available parallel corpora and applied to different downstream tasks without any fine-tuning. Our model sets a new state-of-the-art for most languages in zero-shot cross-lingual natural language inference (XNLI), cross-lingual document classification (MLDoc), and bitext mining (BUCC). We also introduce a new test set of cross-lingual similarity search in 122 languages, and show that our approach is competitive even for low-resource languages. To the best of our knowledge, this is the first successful exploration of massively multilingual sentence representations.

In the future, we would like to explore alternative architectures for the encoder. In particular, we plan to replace our BiLSTM with the Transformer, which has been shown to work better in different settings (Vaswani et al., 2017; Devlin et al., 2018). Moreover, we would like to explore possible strategies to exploit monolingual training data in addition to parallel corpora, such as using pre-trained word embeddings, backtranslation (Sennrich et al., 2016; Edunov et al., 2018), or other ideas from unsupervised machine translation (Artetxe et al., 2018c; Lample et al., 2018). Finally, we would like to replace our language-specific tokenization and BPE segmentation with a language agnostic approach similar to SentencePiece.111111

The model and code used in this paper will be freely available in the framework of the LASER toolkit.121212


Appendix A Training data

Our training data consists of the combination of the following publicly available parallel corpora:

  • Europarl provides high-quality translations for 21 European languages Koehn (2005). The size varies from 400k to 2M sentence pairs, in function of the date the respective country joined the European Union.

  • United Nations: More than 11 million sentences in the six official languages of the United Nations Ziemski et al. (2016). We only use the first two million sentences in Arabic, Russian and Chinese.

  • OpenSubtitles2018: A collection of translations of movie subtitles in 57 languages Lison and Tiedemann (2016). The corpus size varies from few thousand sentences (e.g. Armenian or Kazakh) to more than 50 million (e.g. Spanish or Romanian). We keep at most 2 million entries for each language pair.

  • Global Voices: A parallel corpus of news stories from the Global Voices website (38 languages). This is a rather small corpus with less than 100k sentence in most of the languages.

  • Tanzil: A collection of Quran translations in 42 languages. The style and vocabulary is very different from news texts. The average size is 135k sentences.

  • Tatoeba: A community supported collection of English sentences and translations into more than 300 languages. We use this corpus to extract a separate test set of up to 1,000 sentences for many languages (see Section 4.4 and C). For languages with more than 1,000 entries, we use the remaining ones for training.

Using all these corpora would provide parallel data for more than hundred languages. However, we finally only kept 93 different languages to train the multilingual sentence embeddings. In particular, we discarded several constructed languages with little practical use (Klingon, Kotava, Lojban, Toki Pona and Volapük).

Appendix B XNLI results for all language combinations

Table 9 reports the accuracies of our system on the XNLI test set when the premises and hypothesis are in a different language (e.g. premise in Russian and hypothesis in Thai). The numbers in the diagonal correspond to the main results reported in Table 9.

We observe that our approach seems to handle the combination of different languages very well. We do not have evidence that very distant languages perform considerably worse. It rather seems that the combined performance is mostly bounded by the accuracy of the language which performs worst when used alone. As an example, Greek-Russian achieves very similar results than Bulgarian-Russian, two Slavic languages. Combing French with Chinese, two totally different languages, is only 1.5 points worse than French/Spanish, two very close languages.

Appendix C Tatoeba dataset

Tatoeba131313 is an open collection of English sentences and high quality translations into more than three hundred languages. The number of available translations is updated every Saturday. We downloaded the snapshot on November 19th 2018 and performed the following processing:

  • Removal of sentences that contain “@” or “http”. This is motivated by the fact that emails and web addresses are not language specific.

  • Removal of sentences with less than three words (before tokenization). These are usually sentences with limited semantic information.

  • Removal of sentences that appear multiple times, either in the source or the target.

After filtering, we created test sets of up to 1,000 aligned sentences with English. This amount of texts is available for 78 languages. Limiting the number of sentences to 500, we increase the coverage to 101 languages, and even 141 languages with 100 parallel sentences. It should be stressed that, in general, the English sentences are not the same for the different languages. This implies that the error rates are not necessarily comparable between the languages.

Appendix D Tatoeba: result analysis

We provide here some analysis on the results given in Tables 1 and 2. We have 48 languages with an error rate below 10% and 55 with less than 20%, respectively (English included). The languages with less than 20% error belong to 20 different families and use 12 different scripts. It is nice to find six languages in this list for which we have only small amounts of bitexts (less than 400k), namely Esperanto, Galician, Hindi, Interlingua, Malayam and Marathi. The two constructed languages probably benefit from their inspiration by other European languages.

Overall, we observe low similarity error rates on the Indo-Aryan languages, namely Hindi, Bengali, Marathi and Urdu. The performance on Berber languages (“ber” and “kab”) is remarkable, although we have less than 100 thousand sentences to train them. This is a typical example of languages which are spoken by several millions of people, but for which the amount of written resources is very limited. It is quite unlikely that we would be able to train a good sentence embedding with language specific corpora only. This clearly shows the benefit of joint training on many languages.

Fifteen languages have similarity error rates of more than 50%. Four of them are low-resource languages with their own script and which are alone in their family: Amharic, Armenian, Khmer and Georgian. This makes it difficult to benefit from joint training. On the other hand, one can also argue that is surprising that a language like Khmer performs much better than random (99.9% error rate) with only 625 training examples. Khmer probably benefits of the fact that he have trained our model on other languages of the region which have influenced Khmer, namely Thai and Vietnamese. There are also several Turkic languages (Kazakh, Tatar, Uighur and Uzbek) and Celtic languages (Breton and Cornish) with high error rates. We hope to improve their performance in the future.

Appendix E Tatoeba: results for unseen languages

We extend our Tatoeba experiments to 29 languages without any training data (see Table 10). Many of them are recognized minority languages spoken in specific regions, e.g. Asturian, Faroese, Frisian, Kashubian, North Moluccan Malay, Piemontese, Swabian or Sorbian. All share some similarities, at various degrees, with other major languages, but also differ by their own grammar or specific vocabulary. This enables our encoder to perform reasonably well. We can probably assume that these are mainly spoken languages with limited resources in written form. The six languages which perform worst are Mongolian, Welsh, Xhosa Pampangan, Yiddish and Gaelic. We include these results here as baseline for future research.

en ar bg de el es fr hi ru sw th tr ur vi zh avg
Premise en 73.8 70.0 71.9 72.8 72.2 72.1 72.2 65.9 71.3 61.4 67.5 69.6 61.0 70.6 70.3 69.5
ar 70.5 71.3 71.0 70.0 70.5 70.5 69.9 64.8 69.9 60.0 67.1 68.2 60.6 69.4 70.0 68.3
bg 72.6 71.1 74.2 72.3 72.6 72.0 72.6 65.5 71.7 60.8 69.0 69.8 61.2 70.5 70.5 69.7
de 71.9 69.6 71.8 72.5 71.4 71.6 71.5 65.2 70.8 60.4 68.1 69.1 60.5 70.0 70.6 69.0
el 72.8 70.0 71.7 72.1 73.0 71.4 71.5 65.1 71.7 60.9 68.1 69.4 60.9 69.8 70.3 69.2
es 73.2 70.3 72.4 72.6 72.3 72.9 72.2 65.0 71.1 61.4 68.0 69.8 60.5 70.4 70.3 69.5
fr 73.2 70.3 72.2 72.5 71.5 72.1 71.8 65.8 71.2 61.4 68.1 70.0 60.9 70.9 70.4 69.5
hi 66.6 65.9 66.6 67.2 66.0 66.1 65.5 65.4 66.4 58.9 63.7 65.8 59.5 65.5 66.0 65.0
ru 71.3 69.9 72.2 71.4 71.3 71.1 71.2 64.4 72.0 60.8 67.9 68.7 60.5 69.9 70.0 68.8
sw 65.7 64.4 65.6 64.9 65.3 65.1 64.5 61.4 64.8 62.2 63.3 64.5 58.2 65.0 65.1 64.0
th 70.5 69.2 71.3 70.1 70.3 70.1 69.6 65.1 70.1 62.0 69.1 67.7 60.9 69.9 69.6 68.4
tr 70.6 69.1 70.3 70.3 70.1 70.6 69.8 64.0 69.1 61.3 67.3 69.7 60.6 69.7 68.9 68.1
ur 65.5 64.7 65.3 65.9 66.0 65.6 64.8 62.0 65.2 58.2 63.2 64.1 61.0 64.3 65.0 64.1
vi 71.6 69.7 72.1 71.0 71.1 71.2 70.5 65.4 70.9 61.2 68.9 69.2 60.5 71.9 70.3 69.0
zh 71.5 69.9 71.7 71.1 70.9 71.2 70.7 64.0 70.8 60.4 68.6 68.9 60.2 69.7 71.4 68.7
avg 70.8 69.0 70.7 70.4 70.3 70.2 69.9 64.6 69.8 60.8 67.2 68.3 60.5 69.2 69.2 68.1
Table 9: Accuracies on the XNLI test set of our approach when the premise and hypothesis are in different language. The results in the diagonal correspond to the accuracies reported in Table 9.
Details Training [-1pt] corpus [-2pt] size Tatoeba Error [%] Tatoeba [-1pt] test set [-2pt] size
ISO3 ISO2 Name Family Script en xx xx en
arq Algerian Arabic Arabic Arabic none 58.62 62.46 911
ast Asturian Romance Ibero Latin none 12.60 14.96 127
awa Awadhi Indo-Aryan Devanagari none 63.20 64.50 231
ceb Cebuano Malayo-Polynesian Latin none 81.67 87.00 600
cha ch Chamorro Malayo-Polynesian (branch) Latin none 64.23 77.37 137
arz Egyptian Arabic Arabic Arabic none 31.24 31.03 477
fao fo Faroese Germanic Latin none 28.24 28.63 262
gla gd Gaelic; Scottish Gaelic Celtic Latin none 95.66 96.98 829
jav jv Javanese Malayo-Polynesian Latin none 73.66 80.49 205
csb Kashubian Slavic Latin none 54.55 58.89 253
mon mn Mongolian Mongolic Cyrillic none 89.55 94.09 440
max North Moluccan Malay Malay Creole Latin none 48.24 50.00 284
nov Novial constructed Latin none 33.07 35.02 257
nno nn Nynorsk Norwegian Germanic Latin none 13.40 10.00 1000
ang Old English Germanic Latin none 58.96 65.67 134
pam Pampangan; Kapampangan Philippine Latin none 93.10 95.00 1000
pms Piemontese Romance Latin none 50.86 49.90 525
orv Russian old Slavic Cyrillic none 68.26 75.45 835
dsb Sorbian Lower Slavic Latin none 48.64 55.32 479
hsb Sorbian Upper Slavic Latin none 42.44 48.65 483
swg Swabian Germanic Latin none 50.00 58.04 112
gsw Swiss German Germanic Latin none 52.99 58.12 117
tzl Talossan constructed Latin none 54.81 55.77 104
tuk tk Turkmen Turkic Latin none 75.37 83.25 203
war Waray Malayo-Polynesian Latin none 84.20 88.60 1000
cym cy Welsh Celtic Latin-Welsch none 89.74 93.04 575
fry fy Western Frisian Germanic Latin none 46.24 50.29 173
xho xh Xhosa Niger-Congo Latin none 90.85 92.25 142
yid yi Yiddish Germanic Hebrew none 93.28 95.40 848
Table 10: Performance on the Tatoeba test set for languages and variants for which we have no training data.