Neural Machine Translation (NMT) is an end-to-end learning approach to machine translation which has recently shown promising results on multiple language pairs Luong et al. (2015); Shen et al. (2015); Wu et al. (2016); Zhang et al. (2016); Tu et al. (2016); Zhang and Zong (2016); Jean et al. (2015); Meng et al. (2015). Unlike conventional Statistical Machine Translation (SMT) systems Koehn et al. (2003); Chiang (2005); Liu et al. (2006); Xiong et al. (2006); Mi et al. (2008)
which consist of multiple separately tuned components, NMT aims at building upon a single and large neural network to directly map input text to associated output text. Typical NMT models consists of two recurrent neural networks (RNNs), an encoder to read and encode the input text into a distributed representation and a decoder to generate translated text conditioned on the input representationSutskever et al. (2014); Bahdanau et al. (2014).
Driven by the breakthrough achieved in computer visionHe et al. (2015); Srivastava et al. (2015), research in NMT has recently turned towards studying Deep Neural Networks (DNNs). Wu et al. Wu et al. (2016) and Zhou et al. Zhou et al. (2016) found that deep architectures in both the encoder and decoder are essential for capturing subtle irregularities in the source and target languages. However, training a deep neural network is not as simple as stacking layers. Optimization often becomes increasingly difficult with more layers. One reasonable explanation is the notorious problem of vanishing/exploding gradients which was first studied in the context of vanilla RNNs Pascanu et al. (2013b). Most prevalent approaches to solve this problem rely on short-cut connections between adjacent layers such as residual or fast-forward connections He et al. (2015); Srivastava et al. (2015); Zhou et al. (2016)
. Different from previous work, we choose to reduce the gradient path inside the recurrent units and propose a novel Linear Associative Unit (LAU) which creates a fusion of both linear and non-linear transformations of the input. Through this design, information can flow across several steps both in time and in space with little attenuation. The mechanism makes it easy to train deep stack RNNs which can efficiently capture the complex inherent structures of sentences for NMT. Based on LAUs, we also propose a NMT model , calledDeepLAU, with deep architecture in both the encoder and decoder.
Although DeepLAU is fairly simple, it gives remarkable empirical results. On the NIST Chinese-English task, DeepLAU with proper settings yields the best reported result and also a 4.9 BLEU improvement over a strong NMT baseline with most known techniques (e.g, dropout) incorporated. On WMT English-German and English-French tasks, it also achieves performance superior or comparable to the state-of-the-art.
2 Neural machine translation
A typical neural machine translation system is a single and large neural network which directly models the conditional probabilityof translating a source sentence to a target sentence .
Attention-based NMT, with RNNsearch as its most popular representative, generalizes the conventional notion of encoder-decoder in using an array of vectors to represent the source sentence and dynamically addressing the relevant segments of them during decoding. The process can be explicitly split into an encoding part, a decoding part and an attention mechanism. The model first encodes the source sentenceinto a sequence of vectors . In general, is the annotation of from a bi-directional RNN which contains information about the whole sentence with a strong focus on the parts of . Then, the RNNsearch model decodes and generates the target translation based on the context and the partial traslated sequence by maximizing the probability of
. In the attention model,is dynamically obtained according to the contribution of the source annotation made to the word prediction. This is called automatic alignment Bahdanau et al. (2014) or attention mechanism Luong et al. (2015), but it is essentially reading with content-based addressing defined in Graves et al. (2014). With this addressing strategy the decoder can attend to the source representation that is most relevant to the stage of decoding.
Deep neural models have recently achieved a great success in a wide range of problems. In computer vision, models with more than convolutional layers have outperformed shallow ones by a big margin on a series of image tasks He et al. (2015); Srivastava et al. (2015). Following similar ideas of building deep CNNs, some promising improvements have also been achieved on building deep NMT systems. Zhou et al. Zhou et al. (2016) proposed a new type of linear connections between adjacent layers to simplify the training of deeply stacked RNNs. Similarly, Wu et al. Wu et al. (2016)
introduced residual connections to their deep neural machine translation system and achieve great improvements. However the optimization of deep RNNs is still an open problem due to the massive recurrent computation which makes the gradient propagation path extremely tortuous.
3 Model Description
In this section, we discuss Linear Associative Unit (LAU) to ease the training of deep stack of RNNs. Based on this idea, we further propose DeepLAU, a neural machine translation model with a deep encoder and decoder.
3.1 Recurrent Layers
A recurrent neural network Williams and Zipser (1989) is a class of neural network that has recurrent connections and a state (or its more sophisticated memory-like extension). The past information is built up through the recurrent connections. This makes RNN applicable for sequential prediction tasks of arbitrary length. Given a sequence of vectors as input, a standard RNN computes the sequence hidden states by iterating the following equation from to :
is usually a nonlinear function such as composition of a logistic sigmoid with an affine transformation.
3.2 Gated Recurrent Unit
It is difficult to train RNNs to capture long-term dependencies because the gradients tend to either vanish (most of the time) or explode. The effect of long-term dependencies is dropped exponentially with respect to the gradient propagation length. The problem was explored in depth by Hochreiter and Schmidhuber (1997); Pascanu et al. (2013b)
. A successful approach is to design a more sophisticated activation function than a usual activation function consisting of gating functions to control the information flow and reduce the propagation path. There is a long thread of work aiming to solve this problem, with the long short-term memory units (LSTM) being the most salient examples and gated recurrent unit (GRU) being the most recent oneHochreiter and Schmidhuber (1997); Cho et al. (2014). RNNs employing either of these recurrent units have been shown to perform well in tasks that require capturing long-term dependencies.
GRU can be viewed as a slightly more dramatic variation on LSTM with fewer parameters. The activation function is armed with two specifically designed gates called update and reset gates to control the flow of information inside each hidden unit. Each hidden state at time-step is computed as follows
where is an element-wise product, is the update gate, and is the candidate activation.
where is the reset gate. Both reset and update gates are computed as :
This procedure of taking a linear sum between the existing state and the newly computed state is similar to the LSTM unit.
3.3 Linear Associative Unit
GRU can actually be viewed as a non-linear activation function with gating mechanism. Here we propose LAU which extends GRU by having an additional linear transformation of the input in its dynamics. More formally, the state update function becomes
Here the updated has three sources: 1) the direct transfer from previous state , 2) the candidate update , and 3) a direct contribution from the input . More specifically, contains the nonlinear information of the input and the previous hidden state.
where and express how much of the nonlinear abstraction are produced by the input and previous hidden state , respectively. For simplicity, we set in this paper and find that this works well in our experiments. The term is usually an affine linear transformation of the input to mach the dimensions of , where . The associated term (the input gate) decides how much of the linear transformation of the input is carried to the hidden state and then the output. The gating function (reset gate) and (update gate) are computed following Equation (4) and (5) while is computed as
The term therefore offers a direct way for input to go to later hidden layers, which can eventually lead to a path to the output layer when applied recursively. This mechanism is potentially very useful for translation where the input, no matter whether it is the source word or the attentive reading (context), should sometimes be directly carried to the next stage of processing without any substantial composition or nonlinear transformation. To understand this, imagine we want to translate a English sentence containing a relative rare entity name such as “Bahrain” to Chinese: LAU is potentially able to retain the embedding of this word in its hidden state, which will otherwise be prone to serious distortion due to the scarcity of training instances for it.
Graves et al. Graves (2013)
explored the advantages of deep RNNs for handwriting recognition and text generation. There are multiple ways of combining one layer of RNN with another. Pascanu et al.Pascanu et al. (2013a) introduced Deep Transition RNNs with Skip connections (DT(S)-RNNs). Kalchbrenner et al. Kalchbrenner et al. (2015) proposed to make a full connection of all the RNN hidden layers. In this work we employ vertical stacking where only the output of the previous layer of RNN is fed to the current layer as input. The input at recurrent layer (denoted as ) is exactly the output of the same time step at layer (denoted as ). Additionally, in order to learn more temporal dependencies, the sequences can be processed in different directions. More formally, given an input sequence , the output on layer is
gives the output of layer at location .
is a recurrent function and we choose LAUs in this work.
The directions are marked by a direction term . If we fixed to , the input will be processed in forward direction, otherwise backward direction.
The deep architecture of DeepLAU, as shown in Figure 1, consists of three parts: a stacked LAU-based encoder, a stacked LAU-based decoder and an improved attention model.
One shortcoming of conventional RNNs is that they are only able to make use of previous context. In machine translation, where whole source utterances are transcribed at once, there is no reason not to exploit future context as well. Thus bi-directional RNNs are proposed to integrate information from the past and the future. The typical bidirectional approach processes the raw input in backward and forward direction with two separate layers, and then concatenates them together. Following Zhou et al. Zhou et al. (2016), we choose another bidirectional approach to process the sequence in order to learn more temporal dependencies in this work. Specifically, an RNN layer processes the input sequence in forward direction. The output of this layer is taken by an upper RNN layer as input, processed in reverse direction. Formally, following Equation (9), we set . This approach can easily build a deeper network with the same number of parameters compared to the classical approach. The final encoder consists of layers and produces the output to compute the conditional input to the decoder.
The alignment model scores how well the output at position matches the inputs around position based on and where is the top-most layer of the encoder at step and is the first layer of decoder at step . It is intuitively beneficial to exploit the information of when reading from the source sentence representation, which is missing from the implementation of attention-based NMT in Bahdanau et al. (2014). In this work, we build a more effective alignment path by feeding both the previous hidden state and the context word to the attention model, inspired by the recent implementation of attention-based NMT111github.com/nyu-dl/dl4mt-tutorial/tree/master/session2. The conditional input is a weighted sum of attention score and encoder output . Formally, the calculation of is
is a nonlinear function with the information of (its word embedding being ) added. In our preliminary experiments, we found that GRU works slightly better than tanh function, but we chose the latter for simplicity.
The decoder follows Equation (9) with fixed direction term . At the first layer, we use the following input:
where is the target word embedding at time step , is dynamically obtained follows Equation (10). There are layers of RNNs armed with LAUs in the decoder. At inference stage, we only utilize the top-most hidden states
to make the final prediction with a softmax layer:
We mainly evaluated our approaches on the widely used NIST Chinese-English translation task. In order to show the usefulness of our approaches, we also provide results on other two translation tasks: English-French, English-German. The evaluation metric is BLEU222 For Chinese-English task, we apply case-insensitive NIST BLEU. For other tasks, we tokenized the reference and evaluated the performance with multi-bleu.pl. The metrics are exactly the same as in previous work. Papineni et al. (2002).
For Chinese-English, our training data consists of M sentence pairs extracted from LDC corpora333The corpora include LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08 and LDC2005T06., with M Chinese words and M English words respectively. We choose NIST 2002 (MT02) dataset as our development set, and the NIST 2003 (MT03), 2004 (MT04) 2005 (MT05) and 2006 (MT06) datasets as our test sets.
For English-German, to compare with the results reported by previous work Luong et al. (2015); Zhou et al. (2016); Jean et al. (2015), we used the same subset of the WMT 2014 training corpus that contains 4.5M sentence pairs with 91M English words and 87M German words. The concatenation of news-test 2012 and news-test 2013 is used as the validation set and news-test 2014 as the test set.
To evaluate at scale, we also report the results of English-French. To compare with the results reported by previous work on end-to-end NMT Sutskever et al. (2014); Bahdanau et al. (2014); Jean et al. (2015); Luong et al. (2014); Zhou et al. (2016), we used the same subset of the WMT 2014 training corpus that contains 12M sentence pairs with 304M English words and 348M French words. The concatenation of news-test 2012 and news-test 2013 serves as the validation set and news-test 2014 as the test set.
4.2 Training details
|Our deep NMT systems|
|DeepLAU +Ensemble + PosUnk|
Our training procedure and hyper parameter choices are similar to those used by Bahdanau et al. (2014). In more details, we limit the source and target vocabularies to the most frequent words in both Chinese-English and English-French. For English-German, we set the source and target vocabularies size to and , respectively.
For all experiments, the dimensions of word embeddings and recurrent hidden states are both set to . The dimension of is also of size . Note that our network is more narrow than most previous work where hidden states of dimmention
is used. we initialize parameters by sampling each element from the Gaussian distribution with mean
Parameter optimization is performed using stochastic gradient descent. AdadeltaZeiler (2012) is used to automatically adapt the learning rate of each parameter ( and ). To avoid gradient explosion, the gradients of the cost function which had norm larger than a predefined threshold were normalized to the threshold Pascanu et al. (2013a). We set to at the beginning and halve the threshold until the BLEU score does not change much on the development set. Each SGD is a mini-batch of examples. We train our NMT model with the sentences of length up to words in the training data, while for the Moses system we use the full training data. Translations are generated by a beam search and log-likelihood scores are normalized by sentence length. We use a beam width of in all the experiments. Dropout was also applied on the output layer to avoid over-fitting. The dropout rate is set to . Except when otherwise mentioned, NMT systems are have 4 layers encoders and 4 layers decoders.
4.3 Results on Chinese-English Translation
Table 1 shows BLEU scores on Chinese-English datasets. Clearly DeepLAU leads to a remarkable improvement over their competitors. Compared to DeepGRU, DeepLAU is BLEU score higher on average four test sets, showing the modeling power gained from the liner associative connections. We suggest it is because LAUs apply adaptive gate function conditioned on the input which make it able to automatically decide how much linear information should be transferred to the next step.
To show the power of DeepLAU, we also make a comparison with previous work. Our best single model outperforms both a phrased-based MT system (Moses) as well as an open source attention-based NMT system (Groundhog) by and BLEU points respectively on average. The result is also better than some other state-of-the-art variants of attention-based NMT mode with big margins. After PosUnk and ensemble, DeepLAU seizes another notable gain of BLEU and outperform Moses by BLEU.
4.4 Results on English-German Translation
|Buck et al. Buck et al. (2014)||Winning WMT’14 system – phrase-based + large LM||-|
|Jean et al. Jean et al. (2015)||gated RNN with search + LV + PosUnk|
|Luong et al. Luong et al. (2015)||LSTM with layers + dropout + local att. + PosUnk|
|Shen et al. Shen et al. (2015)||gated RNN with search + PosUnk + MRT|
|Zhou et al. Zhou et al. (2016)||LSTM with layers + F-F connections|
|Wu et al. Wu et al. (2016)||LSTM with laysrs + RL-refined Word||23.1|
|Wu et al. Wu et al. (2016)||LSTM with laysrs + RL-refined WPM-32K||-||24.6|
|Wu et al. Wu et al. (2016)||LSTM with laysrs + RL-refined WPM-32K + Ensemble||-||26.3|
|Our deep NMT systems|
|this work||DeepLAU + PosUnk|
|this work||DeepLAU + PosUnk + Ensemble models|
The results on English-German translation are presented in Table 2. We compare our NMT systems with various other systems including the winning system in WMT’14 Buck et al. (2014), a phrase-based system whose language models were trained on a huge monolingual text, the Common Crawl corpus. For end-to-end NMT systems, to the best of our knowledge, Wu et al. Wu et al. (2016) is currently the SOTA system and about BLEU points on top of previously best reported results even though Zhou et al. Zhou et al. (2016) used a much deeper neural network444It is also worth mentioning that the result reported by Zhou et al. Zhou et al. (2016) does not include PosUnk, and this comparison is not fair enough..
Following Wu et al. Wu et al. (2016), the BLEU score represents the averaged score of 8 models we trained. Our approach achieves comparable results with SOTA system. As can be seen from the Table 2, DeepLAU performs better than the word based model and even not much worse than the best wordpiece models achieved by Wu et al. Wu et al. (2016). Note that DeepLAU are simple and easy to implement, as opposed to previous models reported in Wu et al. Wu et al. (2016), which dependends on some external techniques to achieve their best performance, such as their introduction of length normalization, coverage penalty, fine-tuning and the RL-refined model.
4.5 Results on English-French Translation
|Enc-Dec Luong et al. (2014)|
|RNNsearch Bahdanau et al. (2014)|
|RNNsearch-LV Jean et al. (2015)|
|Deep-Att Zhou et al. (2016)|
To evaluate at scale, we also show the results on an English-French task with sentence pairs and vocabulary in Table 3. Luong et al. Luong et al. (2014) achieves BLEU score of with a six layers deep Encoder-Decoder model. The two attention models, RNNSearch and RNNsearch-LV achieve BLEU scores of and respectively. The previous best single NMT Deep-Att model with an layers encoder and layers decoder achieves BLEU score of . For DeepLAU, we obtain the BLEU score of with a layers encoder and layers decoder, which is on par with the SOTA system in terms of BLEU. Note that Zhou et al. Zhou et al. (2016) utilize a much larger depth as well as external alignment model and extensive regularization to achieve their best results.
Then we will study the main factors that influence our results on NIST Chinese-English translation task. We also compare our approach with two SOTA topologies which were used in building deep NMT systems.
Residual Networks (ResNet) are among the pioneering works Szegedy et al. (2016); He et al. (2016) that utilize extra identity connections to enhance information flow such that very deep neural networks can be effectively optimized. Share the similar idea, Wu et al. Wu et al. (2016) introduced to leverage residual connections to train deep RNNs.
Fast Forward (F-F) connections were proposed to reduce the propagation path length which is the pioneer work to simplify the training of deep NMT model Zhou et al. (2016). The work can be viewed as a parametric ResNet with short cut connections between adjacent layers. The procedure takes a linear sum between the input and the newly computed state.
LAU vs. GRU
Table 4 shows the effect of the novel LAU. By comparing row to row , we see that when and are set to , the average BLEU scores achieved by DeepGRU and DeepLAU are and , respectively. LAU can bring an improvement of in terms of BLEU. After increasing the model depth to (row and row ), the improvement is enlarged to . When DeepGRU is trained with larger depth (say, ), the training becomes more difficult and the performance falls behind its shallow partner. While for DeepLAU, as can be see in row , with increasing the depth even to and we can still obtain growth by BLEU score. Compared to previous short-cut connection methods (row and row ), The LAU still achieve meaningful improvements over F-F connections and Residual connections by and BLEU points respectively.
DeepLAU introduces more parameters than DeepGRU. In order to figure out the effect of DeepLAU comparing models with the same parameter size, we increase the hidden size of DeepGRU model. Row shows that, after using a twice larger GRU layer, the BLEU score is , which is still worse than the corresponding DeepLAU model with fewer parameters.
Depth vs. Width
Next we will study the model size. In Table 4, starting from and and gradually increasing the model depth, we can achieve substantial improvements in terms of BLEU. With and , our DeepLAU model yields the best BLEU score. We tried to increase the model depth with the same hidden size but failed to see further improvements.
We then tried to increase the hidden size. By comparing row and row , we find the improvements is relative small with a wider hidden size. It is also worth mentioning that a deep and thin network with fewer parameters can still achieve comparable results with its shallow partner. This suggests that depth plays a more important role in increasing the complexity of neural networks than width and our deliberately designed LAU benefit from the optimizing of such a deep model.
A more detailed comparison between DeepLAU (4 layers encoder and 4 layers decoder), DeepLAU(2 layer encoder and 2 layer decoder) and DeepGRU (4 layers encoder and 4 layers decoder), suggest that with deep architectures are essential to the superior performance of our system. In particular, we test the BLEU scores on sentences longer than on the merged test set. Clearly, in all curves, performance degrades with increased sentence length. However, DeepLAU models yield consistently higher BLEU scores than the DeepGRU model on longer sentences. These observations are consistent with our intuition that very deep RNN model is especially good at modeling the nested latent structures on relatively complicated sentences and LAU plays an important role on optimizing such a complex deep model.
We propose a Linear Associative Unit (LAU) which makes a fusion of both linear and non-linear transformation inside the recurrent unit. On this way, gradients decay much slower compared to the standard deep networks which enable us to build a deep neural network for machine translation. Our empirical study shows that it can significantly improve the performance of NMT.
We sincerely thank the anonymous reviewers for their thorough reviewing and valuable suggestions. Wang’s work is partially supported by National Science Foundation for Deep Semantics Based Uighur to Chinese Machine Translation (ID 61662077). Qun Liu’s work is partially supported by Science Foundation Ireland in the ADAPT Centre for Digital Content Technology (www.adaptcentre.ie) at Dublin City University funded under the SFI Research Centres Programme (Grant 13/RC/2106) co-funded under the European Regional Development Fund.
- Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 .
- Buck et al. (2014) Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram counts and language models from the common crawl. In LREC. Citeseer, volume 2, page 4.
- Chiang (2005) David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 263–270.
- Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 .
- Graves (2013) Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 .
- Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401 .
- He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 .
He et al. (2016)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.
Deep residual learning for image recognition.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 770–778.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780.
Jean et al. (2015)
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015.
On using very large
target vocabulary for neural machine translation.
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, pages 1–10. http://www.aclweb.org/anthology/P15-1001.
- Kalchbrenner et al. (2015) Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. 2015. Grid long short-term memory. arXiv preprint arXiv:1507.01526 .
- Koehn et al. (2003) Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Linguistics, pages 48–54.
- Liu et al. (2006) Yang Liu, Qun Liu, and Shouxun Lin. 2006. Tree-to-string alignment template for statistical machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 609–616.
- Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 .
- Luong et al. (2014) Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014. Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206 .
- Meng et al. (2015) Fandong Meng, Zhengdong Lu, Zhaopeng Tu, Hang Li, and Qun Liu. 2015. Neural transformation machine: A new architecture for sequence-to-sequence learning. CoRR abs/1506.06442. http://arxiv.org/abs/1506.06442.
- Mi et al. (2008) Haitao Mi, Liang Huang, and Qun Liu. 2008. Forest-based translation. In ACL. pages 192–199.
- Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318.
- Pascanu et al. (2013a) Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2013a. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026 .
- Pascanu et al. (2013b) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013b. On the difficulty of training recurrent neural networks. ICML (3) 28:1310–1318.
- Shen et al. (2015) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015. Minimum risk training for neural machine translation. arXiv preprint arXiv:1512.02433 .
- Srivastava et al. (2015) Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in neural information processing systems. pages 2377–2385.
- Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112.
- Szegedy et al. (2016) Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261 .
- Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. ArXiv eprints, January .
- Williams and Zipser (1989) Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation 1(2):270–280.
- Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 .
- Xiong et al. (2006) Deyi Xiong, Qun Liu, and Shouxun Lin. 2006. Maximum entropy based phrase reordering model for statistical machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 521–528.
- Zeiler (2012) Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 .
- Zhang et al. (2016) Biao Zhang, Deyi Xiong, and Jinsong Su. 2016. Variational neural machine translation. arXiv preprint arXiv:1605.07869 .
- Zhang and Zong (2016) Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of EMNLP.
- Zhou et al. (2016) Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199 .