This paper describes the submission of the EU project ELITR (European Live Translator)111http://elitr.eu to the non-native speech translation task at IWSLT 2020 (iwslt:2020). It is a result of a collaboration of project partners Charles University (CUNI), Karlsruhe Institute of Technology (KIT), and University of Edinburgh (UEDIN), relying on the infrastructure provided to the project by PerVoice company.
The non-native speech translation shared task at IWSLT 2020 complements other IWSLT tasks by new challenges. Source speech is non-native English. It is spontaneous, sometimes disfluent, and some of the recordings come from a particularly noisy environment. The speakers often have a significant non-native accent. In-domain training data are not available. They consist only of native out-domain speech and non-spoken parallel corpora. The validation data are limited to 6 manually transcribed documents, from which only 4 have reference translations. The target languages are Czech and German.
The task objectives are quality and simultaneity, unlike the previous tasks, which focused only on the quality. Despite the complexity, the resulting systems can be potentially appreciated by many users attending an event in a language they do not speak or having difficulties understanding due to unfamiliar non-native accents or unusual vocabulary.
We build on our experience from the past IWSLT and WMT tasks, see e.g. kit-iwslt19; Nguyen2017a; pham2017kit; WeteskoEtAl2019; BawdenEtAl2019; cuni2019wmt. Each of the participating institutions has offered independent ASR and MT systems trained for various purposes and previous shared tasks. We also create some new systems for this task and deployment for the purposes of the ELITR project. Our short-term motivation for this work is to connect the existing systems into a working cascade for SLT and evaluate it empirically, end-to-end. In the long-term, we want to advance state of the art in non-native speech translation.
2 Overview of Our Submissions
This paper is a joint report for two primary submissions, for online and offline sub-track of the non-native simultaneous speech translation task.
First, we collected all ASR systems that were available for us (Section 3.1) and evaluated them on the validation set (Section 3.2). We selected the best candidate for offline ASR to serve as the source for offline SLT. Then, from the ASR systems, which are usable in online mode, we selected the best candidate for online ASR and as a source for online SLT.
In the next step (Section 4), we punctuated and truecased the online ASR outputs of the validation set, segmented them to individual sentences, and translated them by all the MT systems we had available (Section 5.1). We integrated the online ASRs and MTs into our platform for online SLT (Sections 5.3 and 5.2). We compared them using automatic MT quality measures and by simple human decision, to compensate for the very limited and thus unreliable validation set (Section 5.4). We selected the best candidate systems for each target language, for Czech and German.
Both best candidate MT systems are very fast (see Section 5.5). Therefore, we use them both for the online SLT, where the low translation time is critical, and for offline SLT.
In addition to the primary submissions, we included all the other candidate systems and some public services as contrastive submissions.
3 Automatic Speech Recognition
This section describes our automatic speech recognition systems and their selection.
3.1 ASR Systems
We use three groups of ASR systems. They are described in the following sections.
3.1.1 Kit Asr
KIT has provided three hybrid HMM/ANN ASR systems and an end-to-end sequence-to-sequence ASR system.
The hybrid systems, called KIT-h-large-lm1, KIT-h-large-lm2 and KIT-hybrid, were developed to run on the online low-latency condition, and differ in the use of the language models.
The KIT-h-large-lm adopted a 4-gram language model which was trained on a large text corpus (Nguyen2017a), while the KIT-hybrid employed only the manual transcripts of the speech training data. We would refer the readers to the system paper by Nguyen2017a for more information on the training data and the studies by nguyen2020low; niehues2018lowlatency for more information about the online setup.
The end-to-end ASR, so-called KIT-seq2seq, followed the architecture and the optimizations described by nguyen2019improving. It was trained on a large speech corpus, which is the combination of Switchboard, Fisher, LibriSpeech, TED-LIUM, and Mozilla Common Voice datasets. It was used solely without an external language model.
All KIT ASR systems are unconstrained because they use more training data than allowed for the task.
3.1.2 Kaldi ASR Systems
We used three systems trained in the Kaldi ASR toolkit (Povey_ASRU2011). These systems were trained on Mozilla Common Voice, TED-LIUM, and AMI datasets together with additional textual data for language modeling.
For Kaldi-Mozilla, we used the Mozilla Common Voice baseline Kaldi recipe.222https://github.com/kaldi-asr/kaldi/tree/master/egs/commonvoice/s5
The training data consist of 260 hours of audio. The number of unique words in the lexicon is 7996, and the number of sentences used for the baseline language model is 6994, i.e., the corpus is very repetitive. We first train the GMM-HMM part of the model, where the final number of hidden states for the HMM is 2500, and the number of GMM components is 15000. We then train the chain model, which uses the Time delay neural network (TDNN) architecture(peddinti2015time)
together with the Batch normalization regularization and ReLU activation. We use MFCC features to represent audio frames, and we concatenate them with the 100-dimensional I-vector features for the neural network training. We recompile the final chain model with CMU lexicon to increase the model capacity to 127384 words and 4-gram language model trained with SRILM(stolcke2002srilm) on 18M sentences taken from English news articles.
serves as another baseline, trained on 130 hours of TED-LIUM data (rousseau2012ted) collected before the year 2012. The Kaldi-TedLium model was developed by the University of Edinburgh and was fully described by DBLP:journals/corr/abs-1906-11521. This model was primarily developed for discriminative acoustic adaptation to domains distinct from the original training domain. It is achieved by reusing the decoded lattices from the first decoding pass and by finetuning for TED-LIUM development and test set. The setup follows the Kaldi 1f TED-LIUM recipe. The architecture is similar to Kaldi-Mozilla and uses a combination of TDNN layers with batch normalization and ReLU activation. The input features are MFCC and I-vectors.
was trained on the 100 hours of the AMI data, which comprise of staged meeting recordings (ami-corpus). These data were recorded mostly by non-native English speakers with a different microphone and acoustic environment conditions. The model setup used follows the Kaldi 1i ami recipe. Kaldi-AMI cannot be reliably assessed on the AMI part of the development due to the overlap of training and development data. We have decided not to exclude this overlap so that we do not limit the amount of available training data for our model.
3.1.3 Public ASR Services
As part of our baseline models, we have used Google Cloud Speech-to-Text API333https://cloud.google.com/speech-to-text and Microsoft Azure Speech to Text.444https://azure.microsoft.com/en-us/services/cognitive-services/speech-to-text/ Both of these services provide an API for transcription of audio files in WAV format, and they use neural network acoustic models. We kept the default settings of these systems.
The Google Cloud system supports over 100 languages and several types of English dialects (such as Canada, Ireland, Ghana, or the United Kingdom). For decoding of the development and test set, we have used the United Kingdom English dialect option. The system can be run either in real-time or offline mode. We have used the offline option for this experiment.
The Microsoft Azure Bing Speech API supports fewer languages than Google Cloud ASR but adds more customization options of the final model. It can be also run both in real-time or offline mode. For the evaluation, we have used the offline mode and the United Kingdom English (en-GB) dialect.
3.2 Selection of ASR Candidates
|WER weighted average||avg|
We processed the validation set with all the ASR systems, evaluated WER, and summarized them in Table 1. The validation set (Table 2) contains three different domains with various document sizes, and the distribution does not fully correspond to the test set. The AMI domain is not present in the test set at all, but it is a part of Kaldi-AMI training data. Therefore, a simple selection by an average WER on the whole validation set could favor the systems which perform well on the AMI domain, but they could not be good candidates for the other domains.
In Table 3, we present the weighted average of WER in the validation domains. We weight it by the number of gold transcription words in each of the documents. We observe that Kaldi-AMI has a good performance on the AMI domain, but it is worse on the others. We assume it is overfitted for this domain, and therefore we do not use it as the primary system.
For offline ASR, we use KIT-seq2seq as the primary system because it showed the lowest error rate on the averaged domain.
The online ASR systems can exhibit somewhat lower performance than offline systems. We select KIT-h-large-lm1 as the primary online ASR candidate for Auditing, and KIT-hybrid as primary for the other domains.
Our second primary offline ASR is Kaldi-AMI.
4 Punctuation and Segmentation
All our ASR systems output unpunctuated, often all lowercased text. The MT systems are designed mostly for individual sentences with proper casing and punctuation. To overcome this, we first insert punctuation and casing to the ASR output. Then, we split it into individual sentences by the punctuation marks by a rule-based language-dependent Moses sentence splitter (moses).
Depending on the ASR system, we use one of two possible punctuators. Both of them are usable in online mode.
4.1 KIT Punctuator
The KIT ASR systems use an NMT-based model to insert punctuation and capitalization in an otherwise unsegmented lowercase input stream cho2012segmentation; cho2015punctuation
. The system is a monolingual translation system that translates from raw ASR output to well-formed text by converting words to upper case, inserting punctuation marks, and dropping words that belong to disfluency phenomena. It does not use the typical sequence-to-sequence approach of machine translation. However, it considers a sliding window of recent (uncased) words and classifying each one according to the punctuation that should be inserted and whether the word should be dropped for being a part of disfluency. This gives the system a constant input and output size, removing the need for a sequence-to-sequence model.
While inserting punctuation is strictly necessary for MT to function at all, inserting capitalization and removing disfluencies improves MT performance by making the test case more similar to the MT training conditions cho2017nmt.
4.2 BiRNN Punctuator
For other systems, we use a bidirectional recurrent neural network with an attention-based mechanism bytilk2016 to restore punctuation in the raw stream of ASR output. The model was trained on 4M English sentences from CzEng 1.6 (czeng16:2016) data and a vocabulary of 100K most frequently occurring words. We use CzEng because it is a mixture of domains, both originally spoken, which is close to the target domain, and written, which has richer vocabulary, and both original English texts and translations, which we also expect in the target domain. The punctuated transcript is then capitalized using an English tri-gram truecaser by Lita2003. The truecaser was trained on 2M English sentences from CzEng.
5 Machine Translation
This section describes the translation part of SLT.
5.1 MT Systems
|WMT19 Marian||Marian||encs||no||cuni2019wmt, Section 5.1.1|
|WMT18 T2T||T2T||encs||no||cuni2019wmt, Section 5.1.1|
|IWSLT19||Marian||encs||no||WeteskoEtAl2019, Section 5.1.2|
|OPUS-A||Marian||encs,de+5 l.||no||Section 5.1.3|
|OPUS-B||Marian||encs,de+39 l.||no||Section 5.1.3|
|T2T-multi||T2T||encs,de,en+39 l.||no||Section 5.1.4|
|T2T-multi-big||T2T||encs,de,en+39 l.||no||Section 5.1.4|
|de-LSTM||NMTGMinor||ende||no||dessloch-etal-2018-kit, Section 5.1.6|
See Table 4 for the summary of the MT systems. All except de-LSTM are Transformer-based neural models using Marian (mariannmt) or Tensor2Tensor (t2t) back-end. All of them, except de-T2T, are unconstrained because they are trained not only on the data sets allowed in the task description, but all the used data are publicly available.
5.1.1 WMT Models
WMT19 Marian and WMT18 T2T models are Marian and T2T single-sentence models from cuni2019wmt and popel-2018-cuni. WMT18 T2T was originally trained for the English-Czech WMT18 news translation task, and reused in WMT19. WMT19 Marian is its reimplementation in Marian for WMT19. The T2T model has a slightly higher quality on the news text domain than the Marian model. The Marian model translates faster, as we show in Section 5.5.
5.1.2 IWSLT19 Model
The IWSLT19 system is an ensemble of two English-to-Czech Transformer Big models trained using the Marian toolkit. The models were originally trained on WMT19 data and then finetuned on MuST-C TED data. The ensemble was a component of Edinburgh and Samsung’s submission to the IWSLT19 Text Translation task. See Section 4 of WeteskoEtAl2019 for further details of the system.
5.1.3 OPUS Multi-Lingual Models
The OPUS multilingual systems are one-to-many systems developed within the ELITR project. Both were trained on data randomly sampled from the OPUS collection TIEDEMANN12.463, although they use distinct datasets. OPUS-A is a Transformer Base model trained on 1M sentence pairs each for 7 European target languages: Czech, Dutch, French, German, Hungarian, Polish, and Romanian. OPUS-B is a Transformer Big model trained on a total of 231M sentence pairs covering 41 target languages that are of particular interest to the project555The 41 target languages include all EU languages (other than English) and 18 languages that are official languages of EUROSAI member countries. Specifically, these are Albanian, Arabic, Armenian, Azerbaijani, Belorussian, Bosnian, Georgian, Hebrew, Icelandic, Kazakh, Luxembourgish, Macedonian, Montenegrin, Norwegian, Russian, Serbian, Turkish, and Ukrainian. After initial training, OPUS-B was finetuned on an augmented version of the dataset that includes partial sentence pairs, artificially generated by truncating the original sentence pairs (similar to niehues2018lowlatency). We produce up to 10 truncated sentence pairs for every one original pair.
5.1.4 T2T Multi-Lingual Models
T2T-multi and T2T-multi-big are respectively Transformer and Transformer Big models trained on a Cloud TPU based on the default T2T hyperparameters, with the addition of target language tokens as injohnson-etal-2017-googles. The models were trained with a shared vocabulary on a dataset of English-to-many and many-to-English sentence pairs from OPUS-B containing 42 languages in total, making them suitable for pivoting. The models do not use finetuning.
de-T2T translation model is based on a Tensor2Tensor translation model model using training hyper-parameters similar to popel2018trainingtips. The model is trained using all the parallel corpora provided for the English-German WMT19 News Translation Task, without back-translation. We use the last training checkpoint during model inference. To reduce the decoding time, we apply greedy decoding instead of a beam search.
5.1.6 KIT Model
KIT’s translation model is based on an LSTM encoder-decoder framework with attention pham2017kit. As it is developed for our lecture translation framework muller2016lecture, it is finetuned for lecture content. In order to optimize for a low-latency translation task, the model is also trained on partial sentences in order to provide more stable translations niehues2016dynamic.
5.2 ELITR SLT Platform
We use a server called Mediator for the integration of independent ASR and MT systems into a cascade for online SLT. It is a part of the ELITR platform for simultaneous multilingual speech translation (ELITR-ELG). The workers, which can generally be any audio-to-text or text-to-text processors, such as ASR and MT systems, run inside of their specific software and hardware environments located physically in their home labs around Europe. They connect to Mediator and offer a service. A client, often located in another lab, requests Mediator for a cascade of services, and Mediator connects them. This platform simplifies the cross-institutional collaboration when one institution offers ASR, the other MT, and the third tests them as a client. The platform enables using the SLT pipeline easily in real-time.
5.3 MT Wrapper
The simultaneous ASR incrementally produces the recognition hypotheses and gradually improves them. The machine translation system translates one batch of segments from the ASR output at a time. If the translation is not instant, then some ASR hypotheses may be outdated during the translation and can be skipped. We use a program called MT Wrapper for connecting the output of self-updating ASR with non-instant NMT systems.
MT Wrapper has two threads. The receiving thread segments the input for our MTs into individual sentences, saves the input into a buffer, and continuously updates it. The translating thread is a loop that retrieves the new content from the buffer. If a segment has been translated earlier in the current process, it is outputted immediately. Otherwise, the new segments are sent in one batch to the NMT system, stored to a cache and outputted.
For reproducibility, the translation cache is empty at the beginning of a process, but in theory it could be populated by a translation memory. The cache significantly reduces the latency because the punctuator often oscillates between two variants of casing or punctuation marks within a short time.
MT Wrapper has a parameter to control the stability and latency. It can mask the last words of incomplete sentences from the ASR output, as in ma-etal-2019-stacl and Arivazhagan2019ReTranslationSF, considering only the currently completed sentences, or only the “stable” sentences, which are beyond the ASR and punctuator processing window and never change. We do not tune these parameters in the validation. We do not mask any words or segments in our primary submission, but we submit multiple non-primary systems differing in these parameters.
5.4 Quality Validation
For comparing the MT candidates for SLT, we processed the validation set by three online ASR systems, translated them by the candidates, aligned them with reference by mwerSegmenter (segmentation:matusov:2005:iwslt) and evaluated the BLEU score (post-2018-call; papineni-etal-2002-bleu) of the individual documents. However, we were aware that the size of the validation set is extremely limited (see Table 2
) and that the automatic metrics as the BLEU score estimate the human judgment of the MT quality reliably only if there is a sufficient number of sentences or references. It is not the case of this validation set.
Therefore, we examined them by a simple comparison with source and reference. We realized that the high BLEU score in the Autocentrum document is induced by the fact that one of the translated sentences matches exactly matches a reference because it is a single word “thanks”. This sentence increases the average score of the whole document, although the rest is unusable due to mistranslated words. The ASR quality of the two Antrecorp documents is very low, and the documents are short. Therefore we decided to omit them in comparison of the MT candidates.
We examined the differences between the candidate translations on the Auditing document, and we have not seen significant differences, because this document is very short. The AMIa document is longer, but it contains long pauses and many isolated single-word sentences, which are challenging for ASR. The part with a coherent speech is very short.
Finally, we selected the MT candidate, which showed the highest average BLEU score on the three KIT online ASR systems both on Auditing and AMIa document because we believe that averaging the three ASR sources shows robustness against ASR imperfections. See Table 5 and Table 6 for the BLEU scores on Czech and German. The selected candidates are IWSLT19 for Czech and OPUS-B for German. However, we also submit all other candidates as non-primary systems to test them on a significantly larger test set. We use these candidates both for online and offline SLT.
5.5 Translation Time
Average and standard deviation time for translating one batch in validation set, in milliseconds. Bold are the candidate systems for online SLT.
We measured the average time, in which the MT systems process a batch of segments of the validation set (Table 7). If the ASR updates are distributed uniformly in time, than the average batch translation time is also the expected delay of machine translation. The shortest delay is almost zero; in cases when the translation is cached or for very short segments. The longest delay happens when an ASR update arrives while the machine is busy with processing the previous batch. The delay is time for translating two subsequent batches, waiting and translating.
We suppose that the translation time of our primary candidates is sufficient for real-time translation, as we verified in on online SLT test sessions.
We observe differences between the MT systems. The size and the model type of WMT19 Marian and WMT18 T2T are the same (see cuni2019wmt), but they differ in implementation.
WMT19 Marian is slightly faster than IWSLT19 model because the latter is an ensemble of two models. OPUS-B is slower than OPUS-A because the former is bigger. Both are slower than WMT19 Marian due to multi-targeting and different preprocessing. WMT19 Marian uses embedded SentencePiece (kudo-richardson-2018-sentencepiece), while the multi-target models use an external Python process for BPE (Sennrich2016NeuralMT). The timing may be affected also by different hardware.
At the validation time, T2T-multi and T2T-multi-big used suboptimal setup.
We presented ELITR submission for non-native SLT at IWSLT 2020. We observe a significant qualitative difference between the end-to-end offline ASR methods and hybrid online methods. The component that constrains the offline SLT from real-time processing is the ASR, not the MT.
We selected the best candidates from a pool of pre-existing and newly developed components, and submitted our primary submissions, although the size of the development set limits us from a reliable validation. Therefore, we submitted all our unselected candidates for contrastive evaluation on the test set. For the results, we refer to iwslt:2020.
The research was partially supported by the grants 19-26934X (NEUREM3) of the Czech Science Foundation, H2020-ICT-2018-2-825460 (ELITR) of the EU, 398120 of the Grant Agency of Charles University, and by SVV project number 260 575.