The 2019–2020 coronavirus pandemic has disrupted lives, societies and economies across the globe. Its classification as a pandemic highlights its global impact, touching people of all languages. Digital content of all types (social media, news articles, videos) have focused for many weeks predominantly on the sanitary crisis and its effects on infected people, their families, healthcare workers and the society and economy at large. This calls not only for a large set of tools to help during the pandemic (as evidenced by the submissions to this workshop), but also for tools to help digest and analyze this data after it ends. By analyzing the representation and reaction across countries with different guidelines or global trends, it might be possible to inform policies in prevention of and reaction to future epidemics. Several institutions and groups have already started to take snapshots of the digital content shared during these weeks (Croquet, 2020; Banda et al., 2020).
However, because of its global scale, all this digital content is accessible in a variety of different languages, and most existing NLP tools remain English-centric (Anastasopoulos and Neubig, 2019). In this paper we describe the release of a multilingual neural machine translation model (MNMT) that can be used to translate biomedical text. The model is both multi-domain and multilingual, covering translation from French, German, Spanish, Italian and Korean to English.
Our contributions consist in the release of:
An MNMT model, and benchmark results on standard test sets;
A new biomedical Korean-English test set.
This paper is structured as follows: in Section 2 we overview previous work upon which we build; Section 3 details the model and data settings, and the released test set; and Section 4 compares our model to other public models and to state-of-the-art results in academic competitions.
2 Related Work
In order to serve its purpose, our model should be able to process multilingual input sentences, and generate tailored translations for COVID-19-related sentences. As far as NMT models are concerned, both multilingual and domain-specific sentences are just sequences of plain tokens that should be distinguished internally and handled in a separate manner depending on the multiple languages or domains. Due to this commonality in both fields of MNMT and domain adaptation of NMT models, they can be broadly categorized into two groups: 1) data-centric and 2) model-centric (Chu and Wang, 2018).
The former focuses on the preparation of the training data such as handling and selecting from multi-domain (Kobus et al., 2016; Tars and Fishel, 2018) or multilingual parallel corpora (Aharoni et al., 2019; Tan et al., 2019a); and generating synthetic parallel data from monolingual corpora (Sennrich et al., 2015; Edunov et al., 2018).
The model-centric approaches, on the other hand, center on adjusting the training objectives (Wang et al., 2017; Tan et al., 2019b); modifying the model architectures (Vázquez et al., 2018; Dou et al., 2019a); and tweaking the decoding procedure (Hasler et al., 2018; Dou et al., 2019b).
While the two types of approaches are orthogonal and can be utilized in tandem, our released model is trained using data-centric approaches. One of the frequently used data-centric methods for handling sentences of multiple languages and domains is simply prepending a special token that indicates the target language or domain that the sentence should be translated into (Kobus et al., 2016; Aharoni et al., 2019). By feeding the task-specific meta-information via the reserved tags, we signal the model to treat the following input tokens accordingly. Recent works show that this method is also applicable to generating diverse translations (Shu et al., 2019) and translations in specific styles (Madaan et al., 2020).
In addition, back-translation of target monolingual or domain-specific sentences is often conducted in order to augment the low-resource data (Edunov et al., 2018; Hu et al., 2019). Afterward, the back-translated data (and existing parallel data) can be filtered (Xu et al., 2019) or treated with varying amount of importance (Wang et al., 2019) using data selection methods. Back-translated sentences can be tagged to achieve even better results (Caswell et al., 2019).
While myriads of research works on MNMT and domain adaptation exist, the number of publicly available pre-trained NMT models is still low. For example, Fairseq, a popular sequence-to-sequence toolkit maintained by Facebook AI Research, has released ten uni-directional models for translating English, French, German, and Russian sentences.111https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md For its widespread usage, we trained our model using this toolkit.
A large number of public MT models are available thanks to OPUS-MT,222https://github.com/Helsinki-NLP/OPUS-MT-train created by the Helsinki-NLP group. Utilizing the OPUS corpora (Tiedemann, 2012), more than a thousand MT models are trained and released, including several multilingual models which we use to compare with our model.
To the best of our knowledge, we release the first public MNMT model that is capable of producing tailored translations for the biomedical domain.
The COVID-19 pandemic has shown the need for multilingual access to hygiene and safety guidelines and policies (McCulloch, 2020). As an example of crowd-sourced translation, we point out “The COVID Translate Project”333https://covidtranslate.org/ which allowed the translation of 75 pages of guidelines for public agents and healthcare workers, from Korean into English in a matter of days. Although our model could assist in furthering such initiatives, we do not recommend relying solely on our model for translating such guidelines, where quality is of uttermost importance. However, the huge amount of digital content created in the last months around the pandemic makes such professional translations of all that content not only infeasible, but sometimes unnecessary depending on the objective. For instance, we believe that the release of this model can unlock the possibility of large-scale translation with the aim of conducting data analysis on the reaction of the media and society on the matter.
3 Model Settings and Training Data
The model uses a variant of the Transformer Big architecture (Vaswani et al., 2017) with a shallower decoder: 16 attention heads, 6 encoder layers, 3 decoder layers, an embedding size of 1024, and a feed-forward dimension of 8192 in the encoder and 4096 in the decoder.
As all language pairs have English as their target language, no special token for target language was used (language detection can be performed internally by the model).
As the model performs many-to-English translation, its encoder should be able to hold most of the complexity. Thus, we increase the capacity of the encoder by doubling the default size of the feed-forward layer as in Ng et al. (2019).
On the other hand, previous works (Clinchant et al., 2019; Kasai et al., 2020) have shown that it is possible to reduce the number of decoder layers without sacrificing much performance, allowing both faster inference, and smaller network size.
During training, regularization was done with a dropout of and label smoothing of . For optimization, we used Adam (Kingma and Ba, 2014) with warm-up, and maximum learning rate of
. The model was trained for 10 epochs and the best checkpoint was selected based on perplexity on the validation set.
As training data, we used the standard open-accessible datasets, including biomedical data whenever available, for example, the “Corona Crisis Corpora” (TAUS, 2020). Following our past success in domain adaptation (Berard et al., 2019), we used domain tokens (Kobus et al., 2016) to differentiate between domains, allowing multi-domain translation with a single model. We initially experimented with more tags, and combinations of tags (e.g., medical patent or medical political) to allow for more fine-grained control of the resulting translation. The results however were not very conclusive, and often under-performed. An exception worth noting was the case of transcribed data such as TED talks, and OpenSubtitles, which are not the main targets of this work. Therefore, for simplicity, we used only two tags: medical and back-translation. No tag was used with training data that does not belong to one of these two categories.
In addition to biomedical data, we also used back-translated data, although only for Korean, the language with the smallest amount of training data (13.8M sentences). Like Arivazhagan et al. (2019), we used a temperature parameter of 5, to give more chance to Korean. Additionally, the biomedical data was oversampled by a factor of 2. Table 1 details the amount of training sentences used for each language and each domain tag.
As for pre-processing, we cleaned the available data by conducting white-space normalization and NFKC normalization. We filtered noisy sentence pairs based on length (min. 1 token, max. 200), and automatic language identification with langid.py (Lui and Baldwin, 2012).
We trained a lower-cased shared BPE model using SentencePiece (Kudo and Richardson, 2018) by using 6M random lines for each language (including English). We filtered out single characters with fewer than 20 occurrences from the vocabulary. This results in a shared vocabulary of size 76k.
We reduced the English vocabulary size to speed up training and inference, by setting a BPE frequency threshold of 20, which gives a target vocabulary of size 38k. To get the benefits of a shared vocabulary (i.e., tied source/target embeddings) we sorted the source Fairseq dictionary to put the 38k English tokens at the beginning, which lets us easily share the embedding matrix between the encoder and the decoder.444We modified the released checkpoint for it to work out-of-the box with vanilla Fairseq.
The BPE segmentation is followed by inline-casing (Berard et al., 2019), where each token is lower-cased and directly followed by a special token specifying its case (<T> for title case, <U> for all caps, no token for lower-case). Word-pieces whose original case is undefined (e.g., “MacDonalds”) are split again into word-pieces with defined case (“mac” and “donalds”).
3.1 New Korean-English Test Set
To benchmark the performance on the COVID-19 domain, we built an in-domain test set for Korean-English, as it is the only language pair that is not included in the Corona Crisis Corpora.
The test set contains 758 Korean-English sentence pairs, obtained by having English sentences translated into Korean by four professional Korean translators. We note that any acronym written without its full form in the source sentence is kept the same in the translation unless it is very widely used in general. The English sentences were distributed among the translators with the same guidelines to get consistent tone and manner.
We gathered English sentences from two sources: 1) The official English guidelines and reports from Korea Centers for Disease Control and Prevention (KCDC)555http://ncov.mohw.go.kr/en under Ministry of Health and Welfare of South Korea (258 sentences); and 2) Abstracts of biomedical papers on SARS-CoV-2 and COVID-19 from arXiv,666https://arxiv.org medRxiv777https://www.medrxiv.org and bioRxiv888https://www.biorxiv.org (500 sentences). The sentences were handpicked, focusing on covering diverse aspects of the pandemic, including safety guidelines, government briefings, clinical tests, and biomedical experimentation.
We benchmarked the released multilingual models against: 1) reported numbers in the literature, and 2) other publicly released models. We used OPUS-MT, a large collection (1000+) of pre-trained models released by the NLP group at University of Helsinki. Note that these models were trained with much smaller amounts of training data.
We note that the biomedical test sets (Medline) are very small (around 600 lines). We do not report comparison for Spanish-English newstest2013, as the latest reported numbers are outdated (the best WMT entry achieved ).
Our single model obtains competitive results on “generic” test sets (News and IWSLT), on par with the state of the art. We also obtain strong results on the biomedical test sets. Note the SOTA models were trained to maximize performance in the very specific Medline domain, for which training data is provided. While we included this data in our tagged biomedical data, we did not fine-tune aggressively over it.
We describe the release of a multilingual translation model that supports translation in both the general and biomedical domains. Our model is trained on more than 350M sentences, covering French, Spanish, German, Italian and Korean (into English). Benchmarks on public test sets show its strength across domains. In particular, we evaluated the model in the biomedical domain, where it performs near state-of-the-art, but with the advantage of being a single model that operates on several languages. To address the shortage of Korean-English data, we also release a dataset of 758 sentence pairs covering recent biomedical text about COVID-19.
Our aim is to support research studying the international impact that this crisis is causing, at a societal, economical and healthcare level.
- Massively multilingual neural machine translation. arXiv preprint arXiv:1903.00089. Cited by: §2, §2.
- Should all cross-lingual embeddings speak english?. arXiv preprint arXiv:1911.03058. Cited by: §1.
- Massively multilingual neural machine translation in the wild: findings and challenges. arXiv preprint arXiv:1907.05019. Cited by: §3.
- A large-scale covid-19 twitter chatter dataset for open scientific research–an international collaboration. arXiv preprint arXiv:2004.03688. Cited by: §1.
- Findings of the WMT 2019 biomedical translation shared task: evaluation for MEDLINE abstracts and biomedical terminologies. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), Florence, Italy, pp. 29–53. External Links: Cited by: Table 3, §4.
- Naver labs Europe’s systems for the WMT19 machine translation robustness task. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 526–532. External Links: Cited by: §3, §3, Table 3, §4.
- Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Florence, Italy, pp. 53–63. External Links: Cited by: §2.
- A survey of domain adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, pp. 1304–1319. Cited by: §2.
- On the use of BERT for neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, pp. 108–117. External Links: Cited by: §3.
- Comment les archivistes de la BNF sauvegardent la mémoire du confinement sur internet. Le Monde. Note: accessed June 2020 External Links: Cited by: §1.
Unsupervised domain adaptation for neural machine translation with domain-aware feature embeddings.
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1417–1422. Cited by: §2.
- Domain differential adaptation for neural machine translation. arXiv preprint arXiv:1910.02555. Cited by: §2.
- Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 489–500. External Links: Cited by: §2, §2.
- Neural machine translation decoding with terminology constraints. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, pp. 506–512. External Links: Cited by: §2.
Domain adaptation of neural machine translation by lexicon induction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. External Links: Cited by: §2.
- Deep encoder, shallow decoder: reevaluating the speed-quality tradeoff in machine translation. External Links: Cited by: §3.
- Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §3.
- Domain control for neural machine translation. arXiv preprint arXiv:1612.06140. Cited by: §2, §2, §3.
- Sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. Cited by: §3.
- Langid.py: an off-the-shelf language identification tool. In Proceedings of the ACL 2012 System Demonstrations, Jeju Island, Korea, pp. 25–30. Cited by: §3.
- Politeness transfer: a tag and generate approach. External Links: Cited by: §2.
- Covid-19 is history’s biggest translation challenge. Wired. Cited by: §2.
- Facebook FAIR’s WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 314–319. External Links: Cited by: §3, Table 3, §4.
- Fairseq: a fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038. Cited by: §1.
- Improving neural machine translation models with monolingual data. CoRR abs/1511.06709. External Links: Cited by: §2.
- Generating diverse translations with sentence codes. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 1823–1827. External Links: Cited by: §2.
- Multilingual neural machine translation with language clustering. arXiv preprint arXiv:1908.09324. Cited by: §2.
- Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations, Cited by: §2.
- Multi-domain neural machine translation. arXiv preprint arXiv:1805.02282. Cited by: §2.
- TAUS corona crisis corpus. External Links: Cited by: §3.
- Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, pp. 2214–2218. Cited by: §2.
- Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008. Cited by: §3.
- Multilingual nmt with a language-independent attention bridge. arXiv preprint arXiv:1811.00498. Cited by: §2.
- Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 1482–1488. External Links: Cited by: §2.
Improving back-translation with uncertainty-based confidence estimation. arXiv preprint arXiv:1909.00157. Cited by: §2.
- Improving neural machine translation by filtering synthetic parallel data. Entropy 21 (12), pp. 1213. Cited by: §2.