1 Introduction
In recent years, neural machine translation (NMT) has surpassed traditional phrasebased or syntaxbased machine translation, becoming the new state of the art in MT Kalchbrenner and Blunsom (2013); Sutskever et al. (2014); Bahdanau et al. (2014). While NMT training is typically done in a “local” fashion which does not employ any search (bar notable exceptions such as ranzato+:2016, shen+:2016, and wiseman+rush:2016), the decoding phase of all NMT systems universally adopts beam search, a widely used heuristic, to improve translation quality.
Unlike phrasebased MT systems which enjoy the benefits of very large beam sizes (in the order of 100–500) Koehn et al. (2007) , most NMT systems choose tiny beam sizes up to 5; for example, Google’s GNMT Wu et al. (2016) and Facebook’s ConvS2S Gehring et al. (2017) use beam sizes 3 and 5, respectively. Intuitively, the larger the beam size is, the more candidates it explores, and the better the translation quality should be. While this definitely holds for phrasebased MT systems, surprisingly, it is not the case for NMT: many researchers observe that translation quality degrades with beam sizes beyond 5 or 10 Tu et al. (2017); Koehn and Knowles (2017). We call this phenomenon the “beam search curse”, which is listed as one of the six biggest challenges for NMT Koehn and Knowles (2017).
However, there has not been enough attention on this problem. huang+:2017 hint that length ratio is the problem, but do not explain why larger beam sizes cause shorter lengths and worse BLEU. Ott+:2018 attribute it to two kinds of “uncertainties” in the training data, namely the copying of source sentence and the nonliteral translations. However, the first problem is only found in European language datasets and the second problem occurs in all datasets but does not seem to bother preneural MT systems. Therefore, their explanations are not satisfactory.
On the other hand, previous work adopts several heuristics to address this problem, but with various limitations. For example, RNNSearch Bahdanau et al. (2014) and ConvS2S use length normalization, which (we will show in Sec. 6) seems to somewhat alleviate the problem, but far from being perfect. Meanwhile, he+:2016 and huang+:2017 use wordreward, but their reward is a hyperparameter to be tuned on dev set.
Our contributions are as follows:
After we finish our paper, we became aware of a parallel work Murray and Chiang (2018) that also reveals the same root cause we found for the beam search curse: the length ratio problem.
2 Preliminaries: NMT and Beam Search
We briefly review the encoderdecoder architecture with attention mechanism Bahdanau et al. (2014). An RNN encoder takes an input sequence
, and produces a sequence of hidden states. For each time step, the RNN decoder will predict the probability of next output word given the source sequence and the previously generated prefix. Therefore, when doing greedy search, at time step
, the decoder will choose the word with highest probability as . The decoder will continue generating until it emits </eos>. In the end, the generated hypothesis is with , with model score(1) 
As greedy search only explores a single path, we always use beam search to improve search quality. Let denote the beam size, then at step the beam is an ordered list of size :
In the most naive case, after reaching the maximum length (a hard limit), we get possible candidate sequences . The default strategy chooses the one with highest model score. We will discuss more sophistcated ways of stopping and choosing candidates in later sections.
3 Beam Search Curse
The most popular translation quality metric, BLEU Papineni et al. (2002), is defined as:
(2)  
(3)  
(4) 
Here are the gram precisions, and and denote the hypothesis and reference lengths, while is the brevity penalty (penalizing short translations) and is the length ratio Shi et al. (2016); Koehn and Knowles (2017), respectively.
With beam size increasing, decrases, which causes the length ratio to drop, as shown in Fig. 1. Then the brevity penalty term, as a function of the length ratio, decreases even more severely. Since is a key factor in BLEU, this explains why the beam search curse happens.^{1}^{1}1The length ratio is not just about BLEU: if the hypothesis length is only 75% of reference length, something that should have been translated must be missing; i.e., bad adequacy. Indeed, murray+chiang:2018 confirm the same phenomenon with METEOR.
The reason why decreases as beam size increases is actually twofold:

As beam size increases, the more candidates it could explore. Therefore, it becomes easier for the search algorithm to find the </eos> symbol. Fig. 2 shows that the </eos> indices decrease steadily with larger beams.^{2}^{2}2Preneural SMT models, being probabilistic, also favor short translations (and derivations), which is addressed by word (and phrase) reward. The crucial difference between SMT and NMT is that the former stops when covering the whole input, while the latter stops on emitting </eos>.

Then, as shown in Fig. 3, shorter candidates have clear advantages w.r.t. model score.
Hence, as beam size increases, the search algorithm will generate shorter candidates, and then prefer even shorter ones among them.^{3}^{3}3murray+chiang:2018 attribute the fact that beam search prefers shorter candidates to the label bias problem Lafferty et al. (2001) due to NMT’s local normalization.
4 Rescoring Methods
We first review existing methods to counter the length problem and then propose new ones to address their limitations. In particular, we propose to predict the target length from the source sentence, in order to choose a hypothesis with proper length.
4.1 Previous Rescoring Methods
RNNSearch Bahdanau et al. (2014) first introduces the length normalization method, whose score is simply the average model score:
(5) 
This is the most widely used rescoring method since it is hyperparameterfree.
GNMT Wu et al. (2016) incorporates length and coverage penalty into the length normalization method, while also adding two hyperparameters to adjust their influences. (please check out their paper for exact formulas).
Baidu NMT He et al. (2016) borrows the Word Reward method from preneural MT, which gives a reward to every word generated, where is a hyperparameter tuned on the dev set:
(6) 
Based on the above, huang+:2017 propose a variant called Bounded WordReward which only rewards up to an “optimal” length. This length is calculated using a fixed “generation ratio” , which is the ratio between target and source sequence length, namely the average number of target words generated for each source word. It gives reward to each word up to a bounded length :
(7) 
4.2 Rescoring with Length Prediction
To remove the fixed generation ratio from Bounded WordReward, we use a 2layer MLP, which takes the mean of source hidden states as input, to predict the generation ratio . Then we replace the fixed ratio with it, and get our predicted length .
4.2.1 Bounded WordReward
With predicted length, the new predicted bound and final score would be:
(8)  
(9) 
While the predicted length is more accurate, there is still a hyperparameter (word reward), so we design two methods below to remove it.
4.2.2 Bounded AdaptiveReward
We propose Bounded AdaptiveReward to automatically calculate proper rewards based on the current beam. With beam size , the reward for time step is the average negative logprobability of the words in the current beam.
4.2.3 BPNorm
Inspired by the BLEU score definition, we propose BPNorm method as follows:
(12) 
is the same brevity penalty term as in (3). Here, we regard our predicted length as the reference length. The beauty of this method appears when we drop the logarithmic symbol in (12):
which is in the same form of BLEU score (2).
5 Stopping Criteria
Besides rescoring methods, the stopping criteria (when to stop beam search) is also important, for both efficiency and accuracy.
5.1 Conventional Stopping Criteria
By default, OpenNMTpy Klein et al. (2017) stops when the topmost beam candidate stops, because there will not be any future candidates with higher model scores. However, this is not the case for other rescoring methods; e.g., the score of length normalization (5) could still increase.
Another popular stopping criteria, used by RNNSearch Bahdanau et al. (2014), stops the beam search when exactly finished candidates have been found. Neither method is optimal.
5.2 Optimal Stopping Criteria
For Bounded WordReward, huang+:2017 introduces a provablyoptimal stopping criterion that could stop both early and optimally. We also introduce an optimal stopping criterion for BPNorm. Each time we generate a finished candidate, we update our best score . Then, for the topmost beam candidate of time step , we have:
(13) 
where is the maximum generation length. Since will drop after time step , if , we reach optimality. This stopping criterion could also be applied to length normalization (5).
Meawhile, for Bounded AdaptiveReward, we can have a similar optimal stopping criterion: If the score of topmost beam candidate at time step is lower than , we reach optimality.
Proof.
The first part of in (11) will decrease after time step , while the second part stays the same when . So the score in the future will monotonically decrease. ∎
6 Experiments
Our experiments are on ChinesetoEnglish translation task, based on the OpenNMTpy codebase.^{4}^{4}4https://github.com/OpenNMT/OpenNMTpy We train our model on 2M sentences, and apply BPE Sennrich et al. (2015) on both sides, which reduces Chinese and English vocabulary sizes down to 18k and 10k respectively. We then exclude pairs with more than 50 source or target tokens. We validate on NIST 06 and test on NIST 08 (newswire portions only for both). We report caseinsensitive, 4 reference BLEU scores.
We use 2layers bidirectional LSTMs for the encoder. We train the model for 15 epochs, and choose the one with lowest perplexity on the dev set. Batch size is 64; both word embedding and hidden state sizes 500; and dropout 0.3. The total parameter size is 28.5M.
6.1 Parameter Tuning and Results
Small beam ()  dev  test  
BLEU  ratio  BLEU  ratio  
Moses (=70)  30.14    29.41   
Default (=5)  36.45  0.87  32.88  0.87 
Length Norm.  37.73  0.89  34.07  0.89 
+ optimal stopping  38.69  0.92  35.00  0.92 
wu+:2016 ==0.3  38.12  0.89  34.26  0.89 
Bounded wordr. =1.3  39.22  0.98  35.76  0.98 
with predicted length  
Bounded wordr. =1.4  39.53  0.97  35.81  0.97 
Bounded adaptivereward  39.44  0.98  35.75  0.98 
BPNorm  39.35  0.98  35.84  0.99 
Large beam ()  dev  test  
BLEU  ratio  BLEU  ratio  
Moses (=70)  30.14    29.41   
Default (=5)  36.45  0.87  32.88  0.87 
Length Norm.  38.15  0.88  34.26  0.88 
+ optimal stopping  39.07  0.91  35.14  0.91 
wu+:2016 ==0.3  38.40  0.89  34.41  0.88 
Bounded wordr. =1.3  39.60  0.98  35.98  0.98 
with predicted length  
Bounded wordr. =1.4  40.11  0.98  36.13  0.97 
Bounded adaptivereward  40.14  0.98  36.23  0.98 
BPNorm  39.97  0.99  36.22  0.99 
We compare all rescoring methods mentioned above. For the length normalization method, we also show its results with optimal stopping.
For Bounded WordReward method with and without our predicted length, we choose the best on the dev set seperately. The length normalization used by wu+:2016 has two hyperparameters, namely for length penalty and for coverage penalty. We jointly tune them on the dev set, and choose the best config. (=0.3, =0.3).
Figure 4 show our results on the dev set. We see that our proposed methods get the best performance on the dev set, and continue growing as beam size increases. We also observe that optimal stopping boosts the performance of length normalization method by around +0.9 BLEU. In our experiments, we regard our predicted length as the maximum generation length in (13). We further observe from Fig. 5 that our methods keep the length ratio close to 1, and greatly improve the quality on longer input sentences, which are notoriously hard for NMT Shen et al. (2016).
Table 1 collects our results on both dev and test sets. Without loss of generality, we show results with both small and large beam sizes, which average over =14,15,16 and =39,40,41, respectively.
6.2 Discussion
From Table 1, we could observe that with our length prediction model, Bounded wordreward method gains consistent improvement. On the other hand, results from length normalization method show that optimal stopping technique gains significant improvement by around +0.9 BLEU. While with both, our proposed methods beat all previous methods, and gain improvement over hyperparameterfree baseline (i.e. length normalization) by +2.0 BLEU.
Among our proposed methods, Bounded wordreward has the reward as an hyperparameter, while the other two methods get rid of that. Among them, we recommend the BPNorm method, because it is the simplest method, and yet works equally well with others.
7 Conclusions
We first explain why the beam search curse exists and then formalize all previous rescoring methods. Beyond that, we also propose several new methods to address this problem. Results from the ChineseEnglish task show that our hyperparameterfree methods beat the hyperparameterfree baseline (length normalization) by +2.0 BLEU.
Acknowledgements
Kenton Lee suggested the length prediction idea. This work was partially supported by DARPA N660011724030, and NSF IIS1817231 and IIS1656051. We thank the anonymous reviewers for suggestions and Juneki Hong for proofreading.
References
 Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
 Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proc. of ICML.

He et al. (2016)
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016.
Improved neural machine translation with smt features.
In
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence
, pages 151–157. AAAI Press. 
Huang et al. (2017)
Liang Huang, Kai Zhao, and Mingbo Ma. 2017.
When to finish? optimal beam search for neural text generation (modulo beam size).
In EMNLP.  Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, volume 3, page 413.
 Klein et al. (2017) G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. OpenNMT: OpenSource Toolkit for Neural Machine Translation. ArXiv eprints.
 Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris CallisonBurch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177–180. Association for Computational Linguistics.
 Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. CoRR, abs/1706.03872.
 Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML.
 Murray and Chiang (2018) Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of WMT 2018.
 Ott et al. (2018) Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. arXiv preprint arXiv:1803.00047.
 Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL, pages 311–318, Philadephia, USA.

Ranzato et al. (2016)
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.
Sequence level training with recurrent neural networks.
ICLR.  Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
 Shen et al. (2016) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of ACL.

Shi et al. (2016)
Xing Shi, Kevin Knight, and Deniz Yuret. 2016.
Why neural translations are the right length.
In
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
, pages 2278–2282.  Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112.
 Tu et al. (2017) Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In AAAI, pages 3097–3103.
 Wiseman and Rush (2016) Sam Wiseman and Alexander M Rush. 2016. Sequencetosequence learning as beamsearch optimization. In Proceedings of EMNLP.
 Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.