Statistical language models estimating the distribution of various natural language phenomena are crucial for many applications. In machine translation, it measures the fluency and well-formness of a translation, and therefore is important for the translation quality, see[Och2002] and [Koehn, Och and Marcu2003] etc.
Common applications of LMs include estimating the distribution based on N-gram coverage of words, to predict word and word orders, as in[Stolcke2002] and [Lafferty et. al.2001]. The independence assumption for each word is one of the simplifying method widely adopted. However, it does not hold in textual data, and underlying content structures need to be investigated as discussed in [Gao et. al.2004].
We model the prediction of phrase and phrase orders. By considering all word sequences as phrases, the dependency inside a phrase is preserved, and the phrase level structure of a sentence can be learned from observations. This can be considered as an n-gram model on the n-gram of words, therefore word based LM is a special case of phrase based LM if only single-word phrases are considered. Intuitively our approach has the following advantages:
1) Long distance dependency: The phrase based LM can capture the long distance relationship easily. To capture the sentence level dependency, e.g. between the first and last word of the sentence in Table 1, we need a 7-gram word based LM, but only a 3-gram phrase based LM, if we take “played the basketball” and “the day before yesterday” as phrases.
2) Consistent translation unit with phrase based MT: Some words may acquire meaning only in context, such as “day”, or “the” in “the day before yesterday” in Table 1. Considering the frequent phrases as single units will reduce the entropy of the language model. More importantly, current MT is performed on phrases, which is taken as the translation unit. The translation task is to predict the next phrase, which corresponds to the phrased based LM.
3) Fewer independence assumptions in statistical models
: The sentence probability is computed as the product of the single word probabilities in the word based n-gram LM and the product of the phrase probabilities in the phrase based n-gram LM, given their histories. The less words/phrases in a sentence, the fewer mistakes the LM may contain due to less independence assumption on words/phrases. Once the phrase segmentation is fixed, the number of elements via phrase based LM is much less than that via the word based LM. Therefore, our approach is less likely to obtain errors due to assumptions.
4) Phrase boundaries as additional information: We consider different segmentation of phrases in one sentence as a hidden variable, which provides additional constraints to align phrases in translation. Therefore, the constraint alignment in the blocks of words can provide more information than the word based LM.
Comparison to Previous Work
In the dependency or structured LM, phrases corresponding to the grammars are considered, and dependencies are extracted, such as in [Gao et. al.2004] and in [Shen et. al.2008]. However, in the phrase based SMT, even phrases violating the grammar structure may help as a translation unit. For instance, the partial phrase “the day before” may appear both in “the day before yesterday” and “the day before Spring”. Most importantly, the phrase candidates in our phrase based LM are same as that in the phrase based translation, therefore are more consistent in the whole translation process, as mentioned in item 2 in Section 1.
, new phrases are added to the lexicon with different measure function. In[Heeman and Damnati1997], a different LM was proposed which derived the phrase probabilities from a language model built at the lexical level. Nonetheless, these methods do not consider the dependency between phrases and the re-ordering problem, and therefore are not suitable for the MT application.
2 Phrase Based LM
We are given a sentence as a sequence of words , where is the sentence length.
In the word based LM [Stolcke2002], the probability of a sentence 111The notational convention will be as follows: we use the symbol to denote general probability distributions with (almost) no specific assumptions. In contrast, for model-based probability distributions, we use the generic symbol
to denote general probability distributions with (almost) no specific assumptions. In contrast, for model-based probability distributions, we use the generic symbol.is defined as the product of the probabilities of each word given its previous words:
The positions of phrase boundaries on a word sequence is indicated by and , where , and is the number of phrases in the sentence. We use to indicate that the -th phrase segmentation is placed after the word and in front of word , where . is a boundary on the left side of the first word , which is defined as , and is always placed after the last word and therefore equals .
An example is illustrated in Table 1. The English sentence () contains seven words (), where denotes “John”, etc. The first phrase segmentation boundary is placed after the first word, and the second boundary is after the third word () and so on. The phrase sequence in this sentence have a different order than that in its translation, on the phrase level. Hence, the phrase based LM advances the word based LM in learning the phrase re-ordering.
(1) Model description
Given a sequence of words and its phrase segmentation boundaries , a sentence can also be represented in the form of a sequence of phrases , and each individual phrase is defined as
In phrase based LM, we consider the phrase segmentation as hidden variable and the Equation 1 can be extended as follows:
(2) Sentence probability
Sum Model (Baum-Welch)
We consider all segmentation candidates. Equation 2 is defined as
Max Model (Viterbi)
The sentence probability formula of the second model is defined as
In practice we select the segmentation that maximizes the perplexity of the sentence instead of the probability to consider the length normalization.
Sentence perplexity and text perplexity in the sum model use the same definition as that in the word based LM. Sentence perplexity in the max model is defined as
(4) Parameter estimation
We apply maximum likelihood to estimate probabilities in both sum model and max model :
where is the frequency of a phrase. The uni-gram phrase probability is , and is the frequency of all single phrases, in the training text. Since we generate exponential number of phrases to the sentence length, the number of parameters is huge. Therefore, we set the maximum n-gram length on the phrase level (note not the phrase length) as in experiments.
For the unseen events, we perform Good-Turing smoothing as commonly done in word based LMs. Moreover, we interpolate between the phrase probability and the product of single word probabilities in a phrase using a convex optimization:
where phrase is made up of words . The idea of this interpolation is to make the probability of a phrase consisting of of words smooth with a -word unigram probability after normalization. In our experiments, we set for convenience.
(6) Algorithm of calculating phrase n-gram counts
The training task is to calculate n-gram counts on the phrase level in Equation 3.
Given a training corpus , where there are sentences (), our goal is to to compute , for all phrase n-grams that the number of phrases is no greater than . Therefore, for each sentence , we should find out every -gram phrases that .
We do Dynamic Programming to collect the phrase n-grams in one sentence :
where is the auxiliary function denoting the multiset of all phrase n-grams or unigram ending at position (). denotes the starting word position of the last phrase in the multiset. The is a multiset, and means to append the element to each element in the multiset. denotes the union of multisets. After appending , we consider all that is no less than and no greater than .
The phrase counts is the sum of all phrase n-grams from all sentences , with each sentence , and is the number of elements in a multiset:
This is an ongoing work, and we performed preliminary experiments on the IWSLT [IWSLT2011] task, then evaluated the LM performance by measuring the LM perplexity and the MT translation performance. Because of the computational requirement, we only employed sentences which contain no more than 15 words in the training corpus and no more than 10 words in the test corpora (Dev2010, on Tst2010 and on Tst2011), as shown in Table 2.
We took word based LM in Equation 1 as the baseline method (Base). We calculated the perplexities of Tst2011 with different n-gram orders using both sum model and max model, with and without smoothing (S.) as in Section 2. Table 3 shows that perplexities in our approaches are all lower than those in the baseline.
For MT, we selected the single best translation output based on the LM perplexity of the 100-best translation candiates, using different LMs as shown in Table 4. Max model along with smoothing outperforms the baseline method under all three test sets with the BLEU score [Papineni et. al.2002] increase of 0.3% on Dev2010, 0.45% on Tst2010, and 0.22% on Tst2011, respectively.
Table 5 shows two examples from the Tst2010, where we can see that our max model generates better selection results than the baseline method in these cases.
We showed the preliminary results that a phrase based LM can improve the performance of MT systems and the LM perplexity. We presented two phrase based models which consider phrases as the basic components of a sentence and perform exhaustive search. Our future work will focus on the efficiency for a larger data track as well as the improvements on the smoothing methods.
- [Aho and Ullman1972] Alfred V. Aho and Jeffrey D. Ullman. 1972. The Theory of Parsing, Translation and Compiling, volume 1. Prentice-Hall, Englewood Cliffs, NJ.
- [American Psychological Association1983] American Psychological Association. 1983. Publications Manual. American Psychological Association, Washington, DC.
- [Association for Computing Machinery1983] Association for Computing Machinery. 1983. Computing Reviews, 24(11):503–512.
- [Chandra et al.1981] Ashok K. Chandra, Dexter C. Kozen, and Larry J. Stockmeyer. 1981. Alternation. Journal of the Association for Computing Machinery, 28(1):114–133.
- [Gusfield1997] Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences. Cambridge University Press, Cambridge, UK.
- [Kuo and Reichl1999] Hong-Kwang Jeff Kuo and Wolfgang Reichl. 1999. Phrase-Based Language Models for Speech Recognition. In EUROSPEECH.
- [Tang2002] Haijiang Tang. 2002. Building Phrase Based Language Model from Large Corpus. Master thesis, The Hong Kong University of Science and Technology, Hong Kong.
- [Och2002] F. Och. 2002. Statistical Machine Translation: From Single Word Models to Alignment Templates. Ph.D. thesis, RWTH Aachen, Germany.
- [Papineni et. al.2002] K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL, pp. 311–318.
- [Koehn, Och and Marcu2003] P. Koehn, F. Och, and D. Marcu. 2003. Statistical phrasebased translation. In Proc. of HLT-NAACL, pp. 48–54.
- [Heeman and Damnati1997] Peter A. Heeman and Geraldine Damnati. 1997. Deriving Phrase-based Language Models. In Automatic Speech Recognition and Understanding, 1997. Proceedings., 1997 IEEE Workshop on pp. 41–48. IEEE.
- [Brown, S. Della Pietra, V. Della Pietra, and R. Mercer1993] P. Brown, S. Della Pietra, V. Della Pietra, and R. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2), 263-311.
- [Gao et. al.2004] J. Gao, J. Y. Nie, G. Wu, and G. Cao. 2004. Dependence language model for information retrieval. In Proc. of ACM, pp. 170-177. ACM.
- [Shen et. al.2008] L. Shen, J. Xu, and R. M. Weischedel. 2008. A New String-to-Dependency Machine Translation Algorithm with a Target Dependency Language Model. In ACL, pp. 577-585.
- [Rosenfeld2000] R. Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here?. Proc. of IEEE, 88(8), 1270-1278.
- [Stolcke2002] A. Stolcke. 2002. SRILM-an extensible language modeling toolkit. In INTERSPEECH.
[Lafferty et. al.2001]
Lafferty, John, Andrew McCallum, and Fernando CN Pereira.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Intl. Conf. on Machine Learning,
[Roark, Saraclar, Collins, and Johnson2004]
B. Roark, M. Saraclar, M. Collins, and M. Johnson.
Discriminative language modeling with conditional random fields and the perceptron algorithm.In Proc. of ACL, pp. 47. ACL.
- [Chelba1997] C. Chelba. 1997. A structured language model. In Proc. of ACL, pp. 498-500. ACL.
- [IWSLT2011] IWSLT. 2011. http://iwslt2011.org/. Homepage.