1 Introduction
Sequencetosequence models for machine translation (Seq2Seq) [Sutskever et al.2014, Bahdanau et al.2014, Cho et al.2014, Kalchbrenner and Blunsom2013, Sennrich et al.2015a, Sennrich et al.2015b, Gulcehre et al.2015] are of growing interest for their capacity to learn semantic and syntactic relations between sequence pairs, capturing contextual dependencies in a more continuous way than phrasebased SMT approaches. Seq2Seq models require minimal domain knowledge, can be trained endtoend, have a much smaller memory footprint than the large phrase tables needed for phrasebased SMT, and achieve stateoftheart performance in largescale tasks like English to French [Luong et al.2015b] and English to German [Luong et al.2015a, Jean et al.2014] translation.
Seq2Seq models are implemented as an encoderdecoder network, in which a source sequence input
is mapped (encoded) to a continuous vector representation from which a target output
will be generated (decoded). The framework is optimized through maximizing the loglikelihood of observing the paired output given :(1) 
While standard Seq2Seq models thus capture the unidirectional dependency from source to target, i.e., , they ignore , the dependency from the target to the source, which has long been an important feature in phrasebased translation [Och and Ney2002, Shen et al.2010]. Phrase based systems that combine , and other features like sentence length yield significant performance boost.
We propose to incorporate this bidirectional dependency and model the maximum mutual information (MMI) between source and target into Seq2Seq models. As li2015diversity recently showed in the context of conversational response generation, the MMI based objective function is equivalent to linearly combining and . With a tuning weight
, such a loss function can be written as :
(2)  
But as also discussed in li2015diversity, direct decoding from (2) is infeasible because computing cannot be done until the target has been computed^{1}^{1}1 As demonstrated in [Li et al.2015]
To avoid this enormous search space, we propose to use a reranking approach to approximate the mutual information between source and target in neural machine translation models. We separately trained two Seq2Seq models, one for and one for . The model is used to generate Nbest lists from the source sentence . The lists are followed by a reranking process using the second term of the objective function, .
Because reranking approaches are dependent on having a diverse Nbest list to rerank, we also propose a diversitypromoting decoding model tailored to neural MT systems. We tested the mutual information objective function and the diversitypromoting decoding model on EnglishFrench, EnglishGerman and German
English translation tasks, using both standard LSTM settings and the more advanced attentionmodel based settings that have recently shown to result in higher performance.
The next section presents related work, followed by a background section 3 introducing LSTM/Attention machine translation models. Our proposed model will be described in detail in Sections 4, with datasets and experimental results in Section 6 followed by conclusions.
2 Related Work
This paper draws on three prior lines of research: Seq2Seq models, modeling mutual information, and promoting translation diversity.
Seq2Seq Models
Seq2Seq models map source sequences to vector space representations, from which a target sequence is then generated. They yield good performance in a variety of NLP generation tasks including conversational response generation [Vinyals and Le2015, Serban et al.2015a, Li et al.2015], and parsing [Vinyals et al.2014, luong2015multi].
A neural machine translation system uses distributed representations to model the conditional probability of targets given sources, using two components, an encoder and a decoder. Kalchbrenner and Blunsom kalchbrenner2013recurrent used an encoding model akin to convolutional networks for encoding and standard hidden unit recurrent nets for decoding. Similar convolutional networks are used in
[Meng et al.2015] for encoding. sutskever2014sequence,luong2015effective employed a stacking LSTM model for both encoding and decoding. bahdanau2014neural, jean2014using adopted bidirectional recurrent nets for the encoder.Maximum Mutual Information
Maximum Mutual Information (MMI) was introduced in speech recognition [Bahl et al.1986] as a way of measuring the mutual dependence between inputs (acoustic feature vectors) and outputs (words) and improving discriminative training [Woodland and Povey2002]. li2015diversity show that MMI could solve an important problem in Seq2Seq conversational response generation. Prior Seq2Seq models tended to generate highly generic, dull responses (e.g., I don’t know) regardless of the inputs [Sordoni et al.2015, Vinyals and Le2015, Serban et al.2015b]. Li et al. li2015diversity show that modeling the mutual dependency between messages and response promotes the diversity of response outputs.
Our goal, distinct from these previous uses of MMI, is to see whether the mutual information objective improves translation by bidirectionally modeling sourcetarget dependencies. In that sense, our work is designed to incorporate into Seq2Seq models features that have proved useful in phrasebased MT, like the reverse translation probability or sentence length [Och and Ney2002, Shen et al.2010, Devlin et al.2014].
Generating Diverse Translations
Various algorithms have been proposed for generated diverse translations in phrasebased MT, including compact representations like lattices and hypergraphs [Macherey et al.2008, Tromble et al.2008, Kumar and Byrne2004], “traits” like translation length [Devlin and Matsoukas2012], bagging/boosting [Xiao et al.2013], or multiple systems [Cer et al.2013]
. gimpel2013systematic,batra2012diverse, produce diverse Nbest lists by adding a dissimilarity function based on Ngram overlaps, distancing the current translation from alreadygenerated ones by choosing translations that have higher scores but distinct from previous ones. While we draw on these intuitions, these existing diversity promoting algorithms are tailored to phrasebased translation frameworks and not easily transplanted to neural MT decoding which requires batched computation.
3 Background: Neural Machine Translation
Neural machine translation models map source to a continuous vector representation, from which target output is to be generated.
3.1 LSTM Models
A longshort term memory model
[Hochreiter and Schmidhuber1997] associates each time step with an input gate, a memory gate and an output gate, denoted respectively as , and . Let denote the vector for the current word , the vector computed by the LSTM model at time by combining and ., the cell state vector at time , andthe sigmoid function. The vector representation
for each time step is given by:(4)  
(5)  
(6)  
(7)  
(8)  
(9) 
where , , , . The LSTM defines a distribution over outputs and sequentially predicts tokens using a softmax function:
where
denotes the activation function between
and , where is the representation output from the LSTM at time . Each sentence concludes with a special endofsentence symbol EOS. Commonly, the input and output each use different LSTMs with separate sets of compositional parameters to capture different compositional patterns. During decoding, the algorithm terminates when an EOS token is predicted.3.2 Attention Models
Attention models adopt a lookback strategy that links the current decoding stage with input time steps to represent which portions of the input are most responsible for the current decoding state [Xu et al.2015, Luong et al.2015b, Bahdanau et al.2014].
Let be the collection of hidden vectors outputted from LSTMs during encoding. Each element in contains information about the input sequences, focusing on the parts surrounding each specific token. Let be the LSTM outputs for decoding at time . Attention models link the currentstep decoding information, i.e., with each of the representations at decoding step using a weight variable . can be constructed from different scoring functions such as the dot product between the two vectors, i.e., , a general
model akin to tensor operation i.e.,
, and the concatenation model by concatenating the two vectors i.e., tanh). The behavior of different attention scoring functions have been extensively studied in luong2015effective. For all experiments in this paper, we adopt the general strategy where the relevance score between the current step of the decoding representation and the encoding representation is given by:(10)  
The attention vector is created by averaging weights over all input timesteps:
(11) 
Attention models predict subsequent tokens based on the combination of the last step outputted LSTM vectors and attention vectors :
(12)  
where , with V denoting vocabulary size. luong2015effective reported a significant performance boost by integrating into the next step LSTM hidden state computation (referred to as the inputfeeding model), making LSTM compositions in decoding as follows:
(13)  
where , , , . For the attention models implemented in this work, we adopt the inputfeeding strategy.
3.3 Unknown Word Replacements
One of the major issues in neural MT models is the computational complexity of the softmax function for target word prediction, which requires summing over all tokens in the vocabulary. Neural models tend to keep a shortlist of 50,0080,000 most frequent words and use an unknown (UNK) token to represent all infrequent tokens, which significantly impairs BLEU scores. Recent work has proposed to deal with this issue: [Luong et al.2015b] adopt a postprocessing strategy based on aligner from IBM models, while [Jean et al.2014] approximates softmax functions by selecting a small subset of target vocabulary.
In this paper, we use a strategy similar to that of jean2014using, thus avoiding the reliance on external IBM model word aligner. From the attention models, we obtain word alignments from the training dataset, from which a bilingual dictionary is extracted. At test time, we first generate target sequences. Once a translation is generated, we link the generated UNK tokens back to positions in the source inputs, and replace each UNK token with the translation word of its correspondent source token using the preconstructed dictionary.
As the unknown word replacement mechanism relies on automatic word alignment extraction which is not explicitly modeled in vanilla Seq2Seq models, it can not be immediately applied to vanilla Seq2Seq models. However, since unknown word replacement can be viewed as a postprocessing technique, we can apply a pretrained attentionmodel to any given translation. For Seq2Seq models, we first generate translations and replace UNK tokens within the translations using the pretrained attention models to postprocess the translations.
4 Mutual Information via Reranking
As discussed in li2015diversity, direct decoding from (2) is infeasible since the second part, , requires completely generating the target before it can be computed. We therefore use an approximation approach:

Train and separately using vanilla Seq2Seq models or Attention models.

Generate Nbest lists from .

Rerank the Nbest list by linearly adding .
4.1 Standard Beam Search for Nbest lists
Nbest lists are generated using a beam search decoder with beam size set to from models. As illustrated in Figure 1, at time step in decoding, we keep record of hypotheses based on score . As we move on to time step , we expand each of the K hypotheses (denoted as , ), by selecting top of the translations, denoted as , , leading to the construction of new hypotheses:
The score for each of the hypotheses is computed as follows:
(14) 
In a standard beam search model, the top hypotheses are selected (from the hypotheses computed in the last step) based on the score . The remaining hypotheses are ignored as we proceed to the next time step.
We set the minimum length and maximum length to 0.75 and 1.5 times the length of sources. Beam size N is set to 200. To be specific, at each time step of decoding, we are presented with word candidates. We first add all hypotheses with an EOS token being generated at current time step to the Nbest list. Next we preserve the top K unfinished hypotheses and move to next time step. We therefore maintain batch size of 200 constant when some hypotheses are completed and taken down by adding in more unfinished hypotheses. This will lead the size of final Nbest list for each input much larger than the beam size^{2}^{2}2For example, for the development set of the EnglishGerman WMT14 task, each input has an average of 2,500 candidates in the Nbest list. .
4.2 Generating a Diverse Nbest List
Unfortunately, the Nbest lists outputted from standard beam search are a poor surrogate for the entire search space [Finkel et al.2006, Huang2008]. The beam search algorithm can only keep a small proportion of candidates in the search space and most of the generated translations in Nbest list are similar, differing only by punctuation or minor morphological variations, with most of the words overlapping. Because this lack of diversity in the Nbest list will significantly decrease the impact of our reranking process, it is important to find a way to generate a more diverse Nbest list.
We propose to change the way is computed in an attempt to promote diversity, as shown in Figure 1. For each of the hypotheses ( and ), we generate the top translations, , as in the standard beam search model. Next we rank the translated tokens generated from the same parental hypothesis based on in descending order: he is ranks the first among he is and he has, and he has ranks second; similarly for it is and it has.
Next we rewrite the score for by adding an additional part , where denotes the ranking of the current hypothesis among its siblings, which is first for he is and it is, second for he has and it has.
(15) 
The top hypothesis are selected based on as we move on to the next time step. By adding the additional term , the model punishes bottom ranked hypotheses among siblings (hypotheses descended from the same parent). When we compare newly generated hypotheses descended from different ancestors, the model gives more credit to top hypotheses from each of different ancestors. For instance, even though the original score for it is is lower than he has, the model favors the former as the latter is more severely punished by the intrasibling ranking part . The model thus generally favors choosing hypotheses from diverse parents, leading to a more diverse Nbest list.
The proposed model is straightforwardly implemented with minor adjustment to the standard beam search model^{3}^{3}3Decoding for neural based MT model using large batchsize can be expensive resulted from softmax word prediction function. The proposed model supports batched decoding using GPU, significantly speed up decoding process than other diversity fostering models tailored to phrase based MT systems. .
We employ the diversity evaluation metrics in
[Li et al.2015] to evaluate the degree of diversity of the Nbest lists: calculating the average number of distinct unigrams distinct1 and bigrams distinct2 in the Nbest list given each source sentence, scaled by the total number of tokens. By employing the diversity promoting model with tuned from the development set based on BLEU score, the value of distinct1 increases from to , and distinct2 increases from to for EnglishGerman translation. Similar phenomenon are observed from EnglishFrench translation tasks and details are omitted for brevity.4.3 Reranking
The generated Nbest list is then reranked by linearly combining with . The score of the source given each generated translation can be immediately computed from the previously trained .
Other than , we also consider , which denotes the average language model probability trained from monolingual data. It is worth nothing that integrating and into reranking is not a new one and has long been employed by in noisy channel models in standard MT. In neural MT literature, recent progress has demonstrated the effectiveness of modeling reranking with language model [Gulcehre et al.2015].
We also consider an additional term that takes into account the length of targets (denotes as ) in decoding. We thus linearly combine the three parts, making the final ranking score for a given target candidate as follows:
(16)  
We optimize , and using MERT [Och2003] BLEU score [Papineni et al.2002] on the development set.
5 Experiments
Our models are trained on the WMT’14 training dataset containing 4.5 million pairs for EnglishGerman and GermanEnglish translation, and 12 million pairs for EnglishFrench translation. For EnglishGerman translation, we limit our vocabularies to the top 50K most frequent words for both languages. For EnglishFrench translation, we keep the top 200K most frequent words for the source language and 80K for the target language. Words that are not in the vocabulary list are noted as the universal unknown token.
For the EnglishGerman and EnglishGerman translation, we use newstest2013 (3000 sentence pairs) as the development set and translation performances are reported in BLEU [Papineni et al.2002] on newstest2014 (2737) sentences. For EnglishFrench translation, we concatenate newstest2012 and newstest2013 to make a development set (6,003 pairs in total) and evaluate the models on newstest2014 with 3,003 pairs^{4}^{4}4As in [Luong et al.2015a]. All texts are tokenized with tokenizer.perl and BLEU scores are computed with multibleu.perl.
5.1 Training Details for and
We trained neural models on Standard Seq2Seq Models and Attention Models. We trained following the standard training protocols described in [Sutskever et al.2014]. is trained identically but with sources and targets swapped.
We adopt a deep structure with four LSTM layers for encoding and four LSTM layers for decoding, each of which consists of a different set of parameters. We followed the detailed protocols from luong2015effective: each LSTM layer consists of 1,000 hidden neurons, and the dimensionality of word embeddings is set to 1,000. Other training details include: LSTM parameters and word embeddings are initialized from a uniform distribution between [0.1,0.1]; For EnglishGerman translation, we run 12 epochs in total. After 8 epochs, we start halving the learning rate after each epoch; for EnglishFrench translation, the total number of epochs is set to 8, and we start halving the learning rate after 5 iterations. Batch size is set to 128; gradient clipping is adopted by scaling gradients when the norm exceeded a threshold of 5. Inputs are reversed.
Our implementation on a single GPU^{5}^{5}5Tesla K40m, 1 Kepler GK110B, 2880 Cuda cores. processes approximately 8001200 tokens per second. Training for the EnglishGerman dataset (4.5 million pairs) takes roughly 1215 days. For the FrenchEnglish dataset, comprised of 12 million pairs, training takes roughly 46 weeks.
Model  Features  BLEU scores 
Standard  p(yx)  13.2 
Standard  p(yx)+Length  13.6 (+0.4) 
Standard  p(yx)+p(xy)+Length  15.0 (+1.4) 
Standard  p(yx)+p(xy)+p(y)+Length  15.4 (+0.4) 
Standard  p(yx)+p(xy)+p(y)+Length+Diver decoding  15.8 (+0.4) 
+2.6 in total  
Standard+UnkRep  p(yx)  14.7 
Standard+UnkRep  p(yx)+Length  15.2 (+0.7) 
Standard+UnkRep  p(yx)+p(xy)+Length  16.3 (+1.1) 
Standard+UnkRep  p(yx)+p(xy)+p(y)+Length  16.7 (+0.4) 
Standard+UnkRep  p(yx)+p(xy)+p(y)+Length+Diver decoding  17.3 (+0.3) 
+2.6 in total  
Attention+UnkRep  p(yx)  20.5 
Attention+UnkRep  p(yx)+Length  20.9 (+0.4) 
Attention+UnkRep  p(yx)+p(xy)+Length  21.8 (+0.9) 
Attention+UnkRep  p(yx)+p(xy)+p(y)+Length  22.1 (+0.3) 
Attention+UnkRep  p(yx)+p(xy)+p(y)+Length+Diver decoding  22.6 (+0.3) 
+2.1 in total  
Jean et al., 2015 (without ensemble)  19.4  
Jean et al., 2015 (with ensemble)  21.6  
luong2015effective (with UnkRep, without ensemble)  20.9  
luong2015effective (with UnkRep, with ensemble)  23.0 
Model  Features  BLEU scores 
Standard  p(yx)  29.0 
Standard  p(yx)+Length  29.7 (+0.7) 
Standard  p(yx)+p(xy)+Length  31.2 (+1.5) 
Standard  p(yx)+p(xy)+p(y)+Length  31.7 (+0.5) 
Standard  p(yx)+p(xy)+p(y)+Length+Diver decoding  32.2 (+0.5) 
+3.2 in total  
Standard+UnkRep  p(yx)  31.0 
Standard+UnkRep  p(yx)+Length  31.5 (+0.5) 
Standard+UnkRep  p(yx)+p(xy)+Length  32.9 (+1.4) 
Standard+UnkRep  p(yx)+p(xy)+p(y)+Length  33.3 (+0.4) 
Standard+UnkRep  p(yx)+p(xy)+p(y)+Length+Diver decoding  33.6 (+0.3) 
+2.6 in total  
Attention+UnkRep  p(yx)  33.4 
Attention+UnkRep  p(yx)+Length  34.3 (+0.9) 
Attention+UnkRep  p(yx)+p(xy)+Length  35.2 (+0.9) 
Attention+UnkRep  p(yx)+p(xy)+p(y)+Length  35.7 (+0.5) 
Attention+UnkRep  p(yx)+p(xy)+p(y)+Length+Diver decoding  36.3 (+0.4) 
+2.7 in total  
LSTM (Google) (without ensemble))  30.6  
LSTM (Google) (with ensemble)  33.0  
luong2015addressing, UnkRep (without ensemble)  32.7  
luong2015addressing, UnkRep (with ensemble)  37.5 
5.2 Training p(y) from Monolingual Data
We respectively train singlelayer LSTM recurrent models with 500 units for German and French using monolingual data. We News Crawl corpora from WMT13^{6}^{6}6http://www.statmt.org/wmt13/translationtask.html as additional training data to train monolingual language models. We used a subset of the original dataset which roughly contains 5060 millions sentences. Following [Gulcehre et al.2015, Sennrich et al.2015a], we remove sentences with more than Unknown words based on the vocabulary constructed using parallel datasets. We adopted similar protocols as we trained Seq2Seq models, such as gradient clipping and mini batch.
5.3 EnglishGerman Results
We reported progressive performances as we add in more features for reranking. Results for different models on WMT2014 EnglishGerman translation task are shown in Figure 1. Among all the features, reverse probability from mutual information (i.e., p(xy)) yields the most significant performance boost, +1.4 and +1.1 for standard Seq2Seq models without and with unknown word replacement, +0.9 for attention models^{7}^{7}7Target length has long proved to be one of the most important features in phrase based MT due to the BLEU score’s significant sensitiveness to target lengths. However, here we do not observe as large performance boost here as in phrase based MT. This is due to the fact that during decoding, target length has already been strictly constrained. As described in 4.1, we only consider candidates of lengths between 0.75 and 1.5 times that of the source.. In line with [Gulcehre et al.2015, Sennrich et al.2015a], we observe consistent performance boost introduced by language model.
We see the benefit from our diverse Nbest list by comparing mutual+diversity models with diversity models. On top of the improvements from standard beam search due to reranking, the diversity models introduce additional gains of +0.4, +0.3 and +0.3, leading the total gains roughly up to +2.6, +2.6, +2.1 for different models. The unknown token replacement technique yields significant gains, in line with observations from jean2014using,luong2015effective.
We compare our EnglishGerman system with various others: (1) The endtoend neural MT system from jean2014using using a large vocabulary size. (2) Models from luong2015effective that combines different attention models. For the models described in [Jean et al.2014] and [Luong et al.2015a], we reprint their results from both the single model setting and the ensemble setting, which a set of (usually 8) neural models that differ in random initializations and the order of minibatches are trained, the combination of which jointly contributes in the decoding process. The ensemble procedure is known to result in improved performance [Luong et al.2015a, Jean et al.2014, Sutskever et al.2014].
Note that the reported results from the standard Seq2Seq models and attention models in Table 1 (those without considering mutual information) are from models identical in structure to the corresponding models described in [Luong et al.2015a], and achieve similar performances (13.2 vs 14.0 for standard Seq2Seq models and 20.5 vs 20.7 for attention models). Due to time and computational constraints, we did not implement an ensemble mechanism, making our results incomparable to the ensemble mechanisms in these papers.
5.4 FrenchEnglish Results
Results from the WMT’14 FrenchEnglish datasets are shown in Table 2, along with results reprinted from sutskever2014sequence,luong2015addressing. We again observe that applying mutual information yields better performance than the corresponding standard neural MT models.
Relative to the EnglishGerman dataset, the EnglishFrench translation task shows a larger gap between our new model and vanilla models where reranking information is not considered; our models respectively yield up to +3.2, +2.6, +2.7 boost in BLEU compared to standard neural models without and with unknown word replacement, and Attention models.
6 Discussion
In this paper, we introduce a new objective for neural MT based on the mutual dependency between the source and target sentences, inspired by recent work in neural conversation generation [Li et al.2015]. We build an approximate implementation of our model using reranking, and then to make reranking more powerful we introduce a new decoding method that promotes diversity in the firstpass Nbest list. On EnglishFrench and EnglishGerman translation tasks, we show that the neural machine translation models trained using the proposed method perform better than corresponding standard models, and that both the mutual information objective and the diversityincreasing decoding methods contribute to the performance boost..
The new models come with the advantages of easy implementation with sources and targets interchanged, and of offering a general solution that can be integrated into any neural generation models with minor adjustments. Indeed, our diversityenhancing decoder can be applied to generate more diverse Nbest lists for any NLP reranking task. Finding a way to introduce mutual information based decoding directly into a firstpass decoder without reranking naturally constitutes our future work.
References
 [Bahdanau et al.2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

[Bahl et al.1986]
LR Bahl, Peter F Brown, Peter V De Souza, and Robert L Mercer.
1986.
Maximum mutual information estimation of hidden Markov model parameters for speech recognition.
In proc. icassp, volume 86, pages 49–52.  [Batra et al.2012] Dhruv Batra, Payman Yadollahpour, Abner GuzmanRivera, and Gregory Shakhnarovich. 2012. Diverse mbest solutions in Markov random fields. In Computer Vision–ECCV 2012, pages 1–16. Springer.
 [Cer et al.2013] Daniel Cer, Christopher D Manning, and Daniel Jurafsky. 2013. Positive diversity tuning for machine translation system combination. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 320–328.
 [Cho et al.2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
 [Devlin and Matsoukas2012] Jacob Devlin and Spyros Matsoukas. 2012. Traitbased hypothesis selection for machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 528–532. Association for Computational Linguistics.

[Devlin et al.2014]
Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M Schwartz,
and John Makhoul.
2014.
Fast and robust neural network joint models for statistical machine translation.
In ACL (1), pages 1370–1380. Citeseer. 
[Finkel et al.2006]
Jenny Rose Finkel, Christopher D Manning, and Andrew Y Ng.
2006.
Solving the problem of cascading errors: Approximate Bayesian inference for linguistic annotation pipelines.
InProceedings of the 2006 Conference on Empirical Methods in Natural Language Processing
, pages 618–626. Association for Computational Linguistics.  [Gimpel et al.2013] Kevin Gimpel, Dhruv Batra, Chris Dyer, Gregory Shakhnarovich, and Virginia Tech. 2013. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, October.
 [Gulcehre et al.2015] Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, HueiChi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535.
 [Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long shortterm memory. Neural computation, 9(8):1735–1780.
 [Huang2008] Liang Huang. 2008. Forest reranking: Discriminative parsing with nonlocal features. In ACL, pages 586–594.
 [Jean et al.2014] Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007.
 [Kalchbrenner and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709.
 [Kumar and Byrne2004] Shankar Kumar and William Byrne. 2004. Minimum Bayesrisk decoding for statistical machine translation. Technical report, DTIC Document.
 [Li et al.2015] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversitypromoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055.
 [Luong et al.2015a] MinhThang Luong, Hieu Pham, and Christopher D Manning. 2015a. Effective approaches to attentionbased neural machine translation. EMNLP.
 [Luong et al.2015b] MinhThang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In Proceedings of ACL.
 [Macherey et al.2008] Wolfgang Macherey, Franz Josef Och, Ignacio Thayer, and Jakob Uszkoreit. 2008. Latticebased minimum error rate training for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 725–734. Association for Computational Linguistics.
 [Meng et al.2015] Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang, and Qun Liu. 2015. Encoding source language with convolutional neural network for machine translation. arXiv preprint arXiv:1503.01838.
 [Och and Ney2002] Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of ACL 2002, pages 295–302.
 [Och2003] Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational LinguisticsVolume 1, pages 160–167. Association for Computational Linguistics.
 [Papineni et al.2002] Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
 [Sennrich et al.2015a] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
 [Sennrich et al.2015b] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
 [Serban et al.2015a] Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015a. Building endtoend dialogue systems using generative hierarchical neural network models. arXiv preprint arXiv:1507.04808.
 [Serban et al.2015b] Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2015b. A survey of available corpora for building datadriven dialogue systems. arXiv preprint arXiv:1512.05742.
 [Shen et al.2010] Libin Shen, Jinxi Xu, and Ralph Weischedel. 2010. Stringtodependency statistical machine translation. Computational Linguistics, 36(4):649–671.
 [Sordoni et al.2015] Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, JianYun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to contextsensitive generation of conversational responses. arXiv preprint arXiv:1506.06714.
 [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112.
 [Tromble et al.2008] Roy W Tromble, Shankar Kumar, Franz Och, and Wolfgang Macherey. 2008. Lattice minimum bayesrisk decoding for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 620–629. Association for Computational Linguistics.
 [Vinyals and Le2015] Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
 [Vinyals et al.2014] Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2014. Grammar as a foreign language. arXiv preprint arXiv:1412.7449.
 [Woodland and Povey2002] P. C. Woodland and D. Povey. 2002. Large scale discriminative training of hidden Markov models for speech recognition. Computer Speech and Language, 16:25–47.
 [Xiao et al.2013] Tong Xiao, Jingbo Zhu, and Tongran Liu. 2013. Bagging and boosting statistical machine translation systems. Artificial Intelligence, 195:496–527.
 [Xu et al.2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044.