is a neural network (NN) architecture for performing sequence classification. Later, it was also adopted to perform speech recognition[4, 5, 6]. The model allows to integrate the main blocks of ASR (acoustic model, alignment model and language model) into a single neural network architecture. The recent ASR advancements in connectionist temporal classification (CTC) [6, 5] and attention [4, 7] based approaches have generated significant interest in speech community to use seq2seq models. However, outperforming conventional hybrid RNN/DNN-HMM models with seq2seq requires a huge amount of data . Intuitively, this is due to the range of roles this model needs to perform: alignment and language modeling along with acoustic to character label mapping.
Multilingual approaches have been used in hybrid RNN/DNN-HMM systems for tackling the problem of low-resource data. These include language adaptive training and shared layer retraining . Parameter sharing investigated in our previous work  seems to be the most beneficial.
Existing multilingual approaches for seq2seq modeling mainly focus on CTC. A multilingual CTC proposed in  uses a universal phone set, FST decoder and language model. The authors also use a linear hidden unit contribution (LHUC)  technique to rescale the hidden unit outputs for each language as a way to adapt to a particular language. Another work 
on multilingual CTC shows the importance of language adaptive vectors as auxiliary input to the encoder in multilingual CTC model. The decoder used here is based on simple greedy search of applyingon every time frame. An extensive analysis of multilingual CTC performance with limited data is performed in . Here, the authors use a word level FST decoder integrated with CTC during decoding.
On a similar front, attention models are explored within a multilingual setup in [15, 16], where an attempt was made to build an attention-based seq2seq model from multiple languages. The data is just pulled together assuming the target languages are seen during the training. Although our prior study  performs a preliminary investigation of transfer learning techniques to address the unseen languages during training, this is not an intensive study of covering various multi-lingual techniques.
In this paper, we explore the multilingual training approaches [18, 19] in hybrid RNN/DNN-HMMs and we incorporate them into the seq2seq models. In our recent work , we showed the multilingual acoustic models (BLSTM particularly) to be superior to multilingual acoustic features in RNN/DNN-HMM systems. Consequently, similar experiments are performed in this paper on a sequence-to-sequence scheme.
The main motivation and contribution behind this work is as follows:
To incorporate the existing multilingual approaches in a joint CTC-attention  framework.
To compare various multilingual approaches: multilingual features, model architectures, and transfer learning.
2 Sequence-to-Sequence Model
In this work, we use the attention based approach  as it provides an effective methodology to perform sequence-to-sequence training. Considering the limitations of attention in performing monotonic alignment [21, 22]
, we choose to use CTC loss function to aid the attention mechanism in both training and decoding.
Let be a -length speech feature sequence and be an -length grapheme sequence. A multi-objective learning framework proposed in  is used in this work to unify attention loss and CTC loss
with a linear interpolation weight, as follows:
The unified model benefits from both effective sequence level training and the monotonic afforded by the CTC loss.
represents the posterior probability of character label sequencew.r.t input sequence
based on the attention approach, which is decomposed with the probabilistic chain rule, as follows:
where denotes the ground truth history. Detailed explanation of the attention mechanism is given later.
Similarly, represents the CTC posterior probability:
where is a CTC state sequence composed of the original grapheme set and the additional blank symbol. is a set of all possible sequences given the character sequence .
The following paragraphs explain the encoder, attention decoder, CTC, and joint decoding used in our approach.
In our approach, both CTC and attention use the same encoder function:
where is an encoder output state at . As , we use bidirectional LSTM (BLSTM).
Location-aware attention mechanism  is used in this work. The output of location-aware attention is:
Here, acts as attention weight, denotes the decoder hidden state, and is the encoder output state defined in (4). The location-attention function is given by a convolution and maps the attention weight of the previous label to a multi channel view for better representation:
Here, (7) provides the unnormalized attention vectors computed with the learnable vector, and affine transformation . Normalized attention weight are obtained in (8) by a standard operation. Finally, the context vector is obtained as a weighted sum of the encoder output states over all frames, with the attention weight:
The decoder function is an LSTM layer which decodes the next character output label from their previous label , hidden state of the decoder and attention output :
This equation is incrementally applied to form in (2).
Connectionist temporal classification (CTC):
Unlike the attention approach, CTC does not use any specific decoder network. Instead, it invokes two important components to perform character level training and decoding: the first one is an RNN-based encoding module . The second component contains a language model and state transition module. The CTC formalism is a special case  of hybrid DNN-HMM framework with the Bayes rule applied to obtain .
Once we have both CTC and attention-based seq2seq models trained, both are jointly used for decoding as below:
Here is a final score used during beam search. controls the weight between attention and CTC models. and multi-task learning weight in (1) are set differently in our experiments.
The experiments are conducted using the BABEL speech corpus collected during the IARPA Babel program. The corpus is mainly composed of conversational telephone speech (CTS) but some scripted recordings and far field recordings are present as well. Table 1 presents the details of the languages used for training and evaluation in this work. We decided to evaluate also on training languages to see effect of multilingual training on training languages. Therefore, Tok Pisin, Georgian from “train” languages and Assamese, Swahili from “target” languages are taken for evaluation.
|# spkrs.||# hours||# spkrs.||# hours||characters|
4 Sequence to sequence model setup
The Bi-RNN  models mentioned above uses an LSTM  cell followed by a projection layer (BLSTMP). In our experiments below, we use only a character-level seq2seq model based on CTC and attention, as mentioned above. Thus, in the following experiments, we will use character error rate (% CER) as a suitable measure to analyze the model performance. All models are trained in ESPnet, end-to-end speech processing toolkit .
5 Multilingual features
Multilingual features are trained separately from seq2seq model according to a setup from our previous RNN/DNN-HMM work 
. It allows us to easily combine traditional DNN techniques with the seq2seq model such as GMM based alignments for NN target estimation, phoneme units and frame-level randomization. Multilingual features incorporate additional knowledge from non-target languages into features which could better guide the seq2seq model.
5.1 Stacked Bottle-Neck feature extraction
The original idea of Stacked Bottle-Neck feature extraction is described in . The scheme consists of two NN stages: The first one is reading short temporal context, its output is stacked, down-sampled, and fed into the second NN reading longer temporal information.
The first stage bottle-neck NN input features are 24 log Mel filter bank outputs concatenated with fundamental frequency features. Conversation-side based mean subtraction is applied and 11 consecutive frames are stacked. Hamming window followed by discrete cosine transform (DCT) retaining 0 to 5 coefficients are applied on the time trajectory of each parameter resulting in 376=222 coefficients at the first-stage NN input.
In this work, the first-stage NN has 4 hidden layers with 1500 units in each except the bottle-neck (BN) one. The BN layer has 80 neurons. The neurons in the BN layer have linear activations as found optimal in. 21 consecutive frames from the first-stage NN are stacked, down-sampled (each 5 frame is taken) and fed into the second-stage NN with an architecture similar to the first-stage NN, except of BN layer with only 30 neurons. Both neural networks were trained jointly as suggested in  in CNTK toolkit  with block-softmax final layer . Context-independent phoneme states are used as the training targets for the feature-extraction NN, otherwise the size of the final layer would be prohibitive.
Finally, BN outputs from the second stage NN are used as features for further experiments and will be noted as “Mult11-SBN”.
Figure 1 presents the performance curve of the seq2seq model with four “train” and “target” languages, as discussed in Section 3, by changing the amount of training data. It shows significant performance drop of baseline, “fbank” based, systems when the amount of training data is lowered.
On the other hand, the multilingual features present: 1) significantly smaller performance degradation than baseline “fbank” features on small amounts of training data. 2) consistent improvement on both train (seen) and target (unseen) languages where we only use train (seen) languages in feature extractor training data. 3) significant improvement even on the full training set, i.e., 1.6%-5.0% absolute (Table 2 summarizes the full training set results).
6 Multilingual models
Next, we focus on the training of multilingual seq2seq models. As our models are character-based, the multilingual training dictionary is created by concatenation of all train languages, and the system is trained in same way as monolingual on concatenated data.
6.1 Direct decoding from multilingual NN
As the multilingual net is trained to convert a sequence of input features into sequence of output characters, any language from training set or an unknown language with compatible set of characters can be directly decoded. Obviously, characters from wrong language can be generated as the system needs to performs also language identification (LID). Adding LID information as an additional feature, similarly to 
, complicates the system. Therefore, we experimented with “fine-tuning” of the system into the target language by running a few epochs only on desired language data. This is in strengthening the target language characters, therefore it makes the system less prone to language- and character-set-mismatch errors.
The first two rows of table 3 present significant performance degradation from monolingual to multilingual seq2seq models caused by wrong decision of output characters in about 20% of test utterances. However, no out-of-language characters are observed after “fine-tuning” and 1.5% and 4.7% improvement over monolingual baseline is reached.
As mentioned above, multilingual NN can be fine-tuned to the target language if character set is compatible with the training set. Figure 2 shows results on Swahili, which is not part of the training set. Similarly to experiments with multilingual features in Figure 1, the multilingual seq2seq systems are effective especially on small amounts of data, but also beat baseline models on full 50h language set.
6.2 Language-Transfer learning
Language-Transfer learning is necessary if target language characters differ from train set ones. The whole process can be described in three steps: 1) randomly initialize output layer parameters, 2) train only new parameters and freeze the remaining ones 3) “fine-tune” the whole NN. Various experiments are conducted on level of output parameters including output softmax (Out), the attention (Att), and CTC parts. Table 4 compares all combinations and clearly shows that retraining of output softmax only is giving the best results.
Finally, we summarize the comparison of the use of multilingual features for the seq2seq model and language transfer learning of multilingual seq2seq model in Figure 3. Interestingly, on contrary to our previous observations on DNN-HMM systems , we found multilingual features superior to language transfer learning in seq2seq model case.
We have presented various multilingual approaches in seq2seq systems including multilingual features and multilingual models by leveraging our multilingual DNN-HMM expertise. Unlike DNN-HMM systems , we obtain the opposite conclusion that multilingual features are more effective in seq2seq systems. It is probably due to efficient fusion of two complementary approaches: explicit GMM-HMM alignment incorporated in BN features and seq2seq models in the final system. With this finding, we will further explore efficient combinations of the DNN-HMM and seq2seq systems as our future work.
-  Ilya Sutskever, Oriol Vinyals, and Quoc V Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104–3112.
-  Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014.
-  Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014.
-  Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio, “Attention-based models for speech recognition,” in Advances in neural information processing systems, 2015, pp. 577–585.
Alex Graves and Navdeep Jaitly,
“Towards end-to-end speech recognition with recurrent neural networks.,”in ICML, 2014, vol. 14, pp. 1764–1772.
-  Alex Graves, “Supervised sequence labelling,” in Supervised sequence labelling with recurrent neural networks, pp. 5–13. Springer, 2012.
-  William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 4960–4964.
-  Andrew Rosenberg, Kartik Audhkhasi, Abhinav Sethy, Bhuvana Ramabhadran, and Michael Picheny, “End-to-end speech recognition and keyword search on low-resource languages,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 5280–5284.
-  Sibo Tong, Philip N Garner, and Hervé Bourlard, “An investigation of deep neural networks for multilingual speech recognition training and adaptation,” Tech. Rep., 2017.
-  Martin Karafiát, Murali Karthick Baskar, Pavel Matějka, Karel Veselỳ, František Grézl, and Jan Černocky, “Multilingual blstm and speaker-specific vector adaptation in 2016 BUT Babel system,” in Spoken Language Technology Workshop (SLT), 2016 IEEE. IEEE, 2016, pp. 637–643.
-  Sibo Tong, Philip N Garner, and Hervé Bourlard, “Multilingual training and cross-lingual adaptation on CTC-based acoustic model,” arXiv preprint arXiv:1711.10025, 2017.
-  Pawel Swietojanski and Steve Renals, “Learning hidden unit contributions for unsupervised speaker adaptation of neural network acoustic models,” in Spoken Language Technology Workshop (SLT), 2014 IEEE. IEEE, 2014, pp. 171–176.
-  Markus Müller, Sebastian Stüker, and Alex Waibel, “Language adaptive multilingual CTC speech recognition,” in International Conference on Speech and Computer. Springer, 2017, pp. 473–482.
-  Siddharth Dalmia, Ramon Sanabria, Florian Metze, and Alan W. Black, “Sequence-based multi-lingual low resource speech recognition,” in ICASSP. 2018, pp. 4909–4913, IEEE.
-  Shinji Watanabe, Takaaki Hori, and John R Hershey, “Language independent end-to-end architecture for joint language identification and speech recognition,” in Automatic Speech Recognition and Understanding Workshop (ASRU), 2017 IEEE. IEEE, 2017, pp. 265–271.
-  Shubham Toshniwal, Tara N Sainath, Ron J Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao, “Towards language-universal end-to-end speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
-  Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji Watanabe, and Takaaki Hori, “Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling,” in IEEE Workshop on Spoken Language Technology (SLT), 2018.
-  Zoltan Tuske, David Nolden, Ralf Schluter, and Hermann Ney, “Multilingual mrasta features for low-resource keyword search and speech recognition systems,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 7854–7858.
-  Martin Karafiát, K. Murali Baskar, Pavel Matějka, Karel Veselý, František Grézl, and Jan Černocký, “Multilingual blstm and speaker-specific vector adaptation in 2016 but babel system,” in Proceedings of SLT 2016. 2016, pp. 637–643, IEEE Signal Processing Society.
-  Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi, “Hybrid CTC/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017.
-  Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, and Alex Waibel, “Self-attentional acoustic models,” in 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018), 2018.
-  Chung-Cheng Chiu and Colin Raffel, “Monotonic chunkwise attention,” CoRR, vol. abs/1712.05382, 2017.
-  Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio, “Attention-based models for speech recognition,” in Advances in Neural Information Processing Systems. 2015, vol. 2015-January, pp. 577–585, Neural information processing systems foundation.
-  Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al., “The Kaldi speech recognition toolkit,” in Automatic Speech Recognition and Understanding, 2011 IEEE Workshop on. IEEE, 2011, pp. 1–4.
-  Mike Schuster and Kuldip K Paliwal, IEEE Transactions on Signal Processing, vol. 45, no. 11, pp. 2673–2681, 1997.
-  Sepp Hochreiter and Jürgen Schmidhuber, Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
-  Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al., “ESPnet: End-to-end speech processing toolkit,” arXiv preprint arXiv:1804.00015, 2018.
-  Martin Karafiát, František Grézl, Mirko Hannemann, Karel Veselý, Igor Szoke, and Jan ”Honza” Černocký, “BUT 2014 Babel system: Analysis of adaptation in NN based systems,” in Proceedings of Interspeech 2014, Singapure, September 2014.
-  Karel Veselý, Martin Karafiát, and František Grézl, “Convolutive bottleneck network features for LVCSR,” in Proceedings of ASRU 2011, 2011, pp. 42–47.
-  Amit Agarwal et al., “An introduction to computational networks and the computational network toolkit,” Tech. Rep. MSR-TR-2014-112, August 2014.
-  Karel Veselý, Martin Karafiát, František Grézl, Miloš Janda, and Ekaterina Egorova, “The language-independent bottleneck features,” in Proceedings of IEEE 2012 Workshop on Spoken Language Technology. 2012, pp. 336–341, IEEE Signal Processing Society.
-  Suyoun Kim and Michael L. Seltzer, “Towards language-universal end-to-end speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.