The performance of a SMT system is dependent on the quantity and quality of the available training data. Typically, SMT systems are trained with all available data, assuming that the more data used to train the system, the better. This assumption is backed by the evidence that scaling to ever larger data shows continued improvements in quality, even when one trains models over billions of n-grams[Brants et al.2007]. In the SMT context, n-grams refers to sequences of consecutive words. However, growing the amount of data available is only feasible to a certain extent. Moreover so, whenever it is critical that such data is related to the task at hand. In fact, translation quality is negatively affected when there is insufficient training data for the specific domain to be tackled in production conditions [Callison-Burch et al.2007, Koehn2010].
Data selection (DS) has the aim of selecting for training, the best subset of sentence pairs from an available pool, so that the translation quality achieved in the target domain is improved.
State-of-the-art DS approaches rely on the idea of choosing those sentence pairs in an out-of-domain training corpus that are in some way similar to an in-domain training corpus in terms of some different metrics. Cross-entropy difference is a typical ranking function [Moore and Lewis2010, Axelrod et al.2011, Rousseau2013, Schwenk et al.2012, Mansour et al.2011].
On the other hand, distributed representations of words have proliferated during the last years in the research community. Neural networks provide powerful tools for processing text, achieving success in text classification[Kim2014] or in machine translation [Sutskever et al.2014, Bahdanau et al.2015]. Furthermore, [Duh et al.2013] leveraged neural language models to perform DS, reporting substantial gains over conventional n-gram language model-based DS.
Recently, convolutional neural networks (CNN) have also been used in data selection[Chen and Huang2016, Chen et al.2016]. In these works, the authors used a similar strategy to the one proposed in Section 2.3, but in a different scenario: they have no in-domain training corpus; only a large out-of-domain pool and small sets of translation instances. Their goal was to select, from the out-of-domain corpus, the more suitable samples for translating their in-domain corpora.
This paper tackles DS by taking advantage of neural network as sentence classifiers with the ultimate goal of obtaining corpus subsets that minimize the training corpus size, while improving translation quality.
and a bidirectional long short-term memory (BLSTM) network[Hochreiter and Schmidhuber1997, Schuster and Paliwal1997]. In Section 3, the experimental design and results are detailed. Finally, the main results of the work and future work are discussed in Section 4.
2 Data Selection
The goal of DS methods consists in to select a subset of source language sentences from an out-of-domain pool of sentences based on a source in-domain corpus .
2.1 Data Selection using Cross-entropy
As mentioned in Section 1, one well-established DS method consists in scoring the sentences in the out-of-domain corpus by their perplexity. Moore and Lewis Moore2010 used cross-entropy rather than perplexity, even though they are both monotonically related. For selecting , this technique relates the cross-entropy given by a language model trained over the together with the out-of-domain () language model, computing a score for a sentence :
where and are the in-domain and out-of-domain cross-entropy of sentence , respectively.
2.2 Data Selection using Neural Networks
In this work, we tackle the DS problem as a classification task. Let us consider a classifier model
that assigns a probabilityto a sentence , depending whether belongs to the in-domain corpus or not.
In this case, to obtain the selection , one could just apply to each sentence from the out-of-domain pool and select the most probable ones.
We explored the use of CNN and BLSTM networks as sentence encoders. As shown in Fig. 1, the input sentence is fed to our system following a one-hot codification scheme and is projected to a continuous space by means of a word-embedding matrix. Next, the input sentence is processed either by a CNN or a BLSTM network. After this, we add one or more fully-connected (FC) layers. Finally, we apply a softmax function, in order to obtain normalized probabilities. All elements from the neural classifier can be jointly trained by maximum likelihood.
Convolutional Neural Networks.
CNNs have proven their representation capacity, not only in computer vision tasks[Szegedy et al.2015], but also representing text [Kalchbrenner and Blunsom2013, Kim2014]. In this work, we used the non-static CNN proposed by Kim Kim14.
Recurrent Neural Networks.
In recurrent neural networks, connections form a directed cycle. This allows the network to maintain an internal state and be powerful sequence modelers. Moreover, bidirectional networks have two independent recurrent layers, one processing the input sequence in a forward manner and other processing it a backward manner. Therefore, they allow to exploit the full context at each time-step. Gated units, such as LSTM[Gers et al.2000]
, mitigate the vanishing gradient problem and hence, they are able tp properly model long sequences.
BLSTM networks can be used for effectively encoding a sentence by concatenating the last hidden state of the forward and backward LSTM layers. This provides a compact representation of the sentence, which takes into account relationships in both time directions.
2.3 Semi-supervised Selection
Properly training these neural classifiers may be a challenging task, since the in-domain data is scarce. Hence, for training them, we follow a semi-supervised iterative protocol [Yarowsky1995].
Since the data selection is a binary classification problem, we need a set of positive and negative training samples. We start from an initial set of positive samples and a set of negative samples and train an initial model . At each iteration , we classify all sentences belonging to the out-of-domain pool (). We extract a number of top-scoring sentences and include them into the set of positive samples, producing a new set of positive samples . Analogously, the bottom-scoring sentences are included into a new negative samples set . Hence, at each iteration, we remove samples from the out-of-domain set, producing the pool . Then, we start a new iteration, training a new model with . This is repeated until there are no more sentences in the out-of-domain pool.
We set our in-domain corpus as . We randomly extract sentences from for constructing . The initial out-of-domain pool is defined as .
3 Experiments in SMT
In this section, we empirically evaluate the DS strategy proposed in Section 2. We conducted experiments with different language pairs with the purpose of evaluating whether the conclusions drawn from one single language pair hold in further scenarios.
Two corpora are involved within the DS task: an out-of-domain corpus and an in-domain corpus . DS selects only a portion of the out-of-domain corpus, and leverages that subset together with the in-domain data to train a, hopefully improved, SMT system. As out-of-domain corpus, we used the Europarl corpus [Koehn2005]. As in-domain data, we used the EMEA corpus [Tiedemann2009]. The Medical-Test and Medical-Mert corpora are partitions established in the 2014 Workshop on SMT111www.statmt.org/wmt14/ [Bojar et al.2014]. We focused on the English-French (En-Fr), French-English (Fr-En), German-English (De-En) and English-German (En-De) language pairs. The main figures of the corpora used are shown in Table 1.
3.2 Experimental Setup
All neural models were initialized using the word-embedding matrices from word2vec222code.google.com/archive/p/word2vec. These matrices were fine-tuned during training. Following Kim Kim14, we used filter windows of lengths 3, 4, 5 with 100 features maps each for the CNN classifier. For training it, we used Adadelta [Zeiler2012] with its default parameters. The BLSTM network was trained with Adam [Kingma and Ba2014], with a learning rate of . In order to have a similar number of parameters than in the CNN (48 million approximately), we used 300 units in each LSTM layer. 2 fully-connected layers of size 200 and 100 were introduced after the CNN and BLSTM (see Section 1). All neural models333Source code available at https://github.com/lvapeab/sentence-selectioNN.
were implemented using Theano[Theano Development Team2016]. The number of sentences selected at each iteration () should be empirically determined. In our experiments, we set .
|Strategy||BLEU||# Sentences||BLEU||# Sentences|
|Strategy||BLEU||# Sentences||BLEU||# Sentences|
All SMT experiments were carried out using the open-source phrase-based SMT toolkit Moses [Koehn et al.2007]. The language model used was a 5-gram, standard in SMT research, with modified Kneser-Ney smoothing [Kneser and Ney1995], built with the SRILM toolkit [Stolcke2002]. The phrase table was generated by means of symmetrised word alignments obtained with GIZA++ [Och and Ney2003]. The log-lineal combination weights were optimized using MERT (minimum error rate training) [Och2003]. Since MERT requires a random initialisation of that often leads to different local optima being reached, every result of this paper constitutes the average of 10 repetitions, with the purpose of providing robustness to the results. In the tables, confidence intervals of these repetitions are shown. SMT output was evaluated by means of BLEU [Papineni et al.2002].
We compared the selection methods with two baseline systems. The first one was obtained by training the SMT system with in-domain training data (EMEA-Domain data). We will refer to this setup with the name of baseline-emea. A second baseline experiment has been carried out with the concatenation of the Europarl corpus and EMEA training data (i.e., all the data available). We will refer to this setup as bsln-emea-euro. In addition, we also included results for a purely random sentence selection without replacement.
3.3 Experimental Results
Table 2 shows the best results obtained with our data selection using the two neural network architectures proposed (CNN and BLSTM) and cross-entropy method for each language pair (EN-FR, DE-EN, EN-DE).
In EN-FR and EN-DE, FR-EN, translation quality using DS improves over bsln-emea-euro, but using significantly less data (, and of the total amount of out-of-domain data, respectively). In the case of DE-EN, translation quality results are similar, but using only a of the data. According to these results, we can state that our DS strategy is able to deliver similar quality than using all the data, but only with a rough quarter of the data.
All proposed DS methods are mostly able to improve over random selection but in some cases differences are not significant. It should be noted that beating random is very hard, since all DS methods, including random, will eventually converge to the same point: adding all the data available. The key difference is the amount of data needed for achieving the same translation quality.
Results obtained in terms of BLEU with our DS method are slightly better than the ones obtained with cross-entropy. However, cross-entropy requires significantly more sentences to reach comparable translation quality.
Lastly, CNN and BLSTM networks seem to perform similarly. Therefore, we conclude that both architectures are good options for this task.
4 Conclusion and Future Work
We developed a DS method, based on sentence classification techniques. The classifier is based on CNNs or BLSTM neural networks. We thoroughly evaluated it over four language pairs. Our method yielded better translation performance than the cross-entropy DS technique, requiring a minor amount of data. Additionally, we found that both CNN and BLSTM networks performed similarly, thus being both suitable sentence encoders.
As future work, we aim to delve into the usage of semi-supervised training strategies for the classifier. Ladder networks [Rasmus et al.2015] seem a promising tool. We should investigate how to include them in our pipeline. We should also investigate the application of one-shot learning strategies to a similar scenario, where only the text to translate is available.
- [Axelrod et al.2011] Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proc. of EMNLP, pages 355–362.
- [Bahdanau et al.2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.
- [Bojar et al.2014] Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia, editors. 2014. In Proc. of WMT. ACL.
- [Brants et al.2007] Thorsten Brants, Ashok C Popat, Peng Xu, Franz J Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proc. of EMNLP/CoNLL, pages 858–867.
- [Callison-Burch et al.2007] Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proc. of WMT, pages 136–158.
- [Chen and Huang2016] Boxing Chen and Fei Huang. 2016. Semi-supervised convolutional networks for translation adaptation with tiny amount of in-domain data. CoNLL 2016, page 314.
- [Chen et al.2016] Boxing Chen, Roland Kuhn, George Foster, Colin Cherry, and Fei Huang. 2016. Bilingual methods for adaptive training data selection for machine translation. AMTA 2016, Vol., page 93.
- [Duh et al.2013] Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013. Adaptation data selection using neural language models: Experiments in machine translation. In Proc. of ACL, pages 678–683.
- [Gers et al.2000] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10):2451–2471.
- [Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780.
- [Kalchbrenner and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proc. of EMNLP, pages 1700–1709.
- [Kim2014] Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. of EMNLP, pages 1746–1751.
- [Kingma and Ba2014] Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980.
- [Kneser and Ney1995] Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proc. of ICASSP, pages 181–184.
- [Koehn et al.2007] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. of ACL, pages 177–180.
- [Koehn2005] Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proc. of MT Summit, pages 79–86.
- [Koehn2010] Philipp Koehn. 2010. Statistical machine translation. Cambridge University Press.
- [LeCun et al.1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. IEEE, 86(11):2278–2324, November.
- [Mansour et al.2011] Saab Mansour, Joern Wuebker, and Hermann Ney. 2011. Combining translation and language model scoring for domain-specific data filtering. In Proc. of IWSLT, pages 222–229.
- [Moore and Lewis2010] Robert C Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proc. of ACL, pages 220–224.
- [Och and Ney2003] Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51.
- [Och2003] Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL, pages 160–167.
- [Papineni et al.2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL, pages 311–318.
- [Rasmus et al.2015] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. 2015. Semi-supervised learning with ladder networks. In Proc. of NIPS, pages 3546–3554.
Xenc: An open-source tool for data selection in natural language processing.The Prague Bulletin of Mathematical Linguistics, 100:73–82.
- [Schuster and Paliwal1997] Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681.
- [Schwenk et al.2012] Holger Schwenk, Anthony Rousseau, and Mohammed Attik. 2012. Large, pruned or continuous space language models on a gpu for statistical machine translation. In Proc. of NAACL-HLT, pages 11–19.
- [Stolcke2002] Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proc. of ICSLP, pages 901–904.
- [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS, pages 3104–3112.
- [Szegedy et al.2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proc. of CVRP, pages 1–9.
- [Theano Development Team2016] Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv:1605.02688.
- [Tiedemann2009] Jörg Tiedemann. 2009. News from opus - a collection of multilingual parallel corpora with tools and interfaces. In Proc. of RANLP, pages 237–248.
- [Yarowsky1995] David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proc. of ACL, pages 189–196.
- [Zeiler2012] Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv:1212.5701.