Simple Fusion: Return of the Language Model

09/01/2018
by   Felix Stahlberg, et al.
0

Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alternative simple method to use monolingual data for NMT training: We combine the scores of a pre-trained and fixed language model (LM) with the scores of a translation model (TM) while the TM is trained from scratch. To achieve that, we train the translation model to predict the residual probability of the training data added to the prediction of the LM. This enables the TM to focus its capacity on modeling the source sentence since it can rely on the LM for fluency. We show that our method outperforms previous approaches to integrate LMs into NMT while the architecture is simpler as it does not require gating networks to balance TM and LM. We observe gains of between +0.24 and +2.36 BLEU on all four test sets (English-Turkish, Turkish-English, Estonian-English, Xhosa-English) on top of ensembles without LM. We compare our method with alternative ways to utilize monolingual data such as backtranslation, shallow fusion, and cold fusion.

READ FULL TEXT
research
11/20/2015

Improving Neural Machine Translation Models with Monolingual Data

Neural Machine Translation (NMT) has obtained state-of-the art performan...
research
09/11/2019

Dynamic Fusion: Attentional Language Model for Neural Machine Translation

Neural Machine Translation (NMT) can be used to generate fluent output. ...
research
01/11/2019

ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation

We present ParaBank, a large-scale English paraphrase dataset that surpa...
research
11/10/2019

Language Model-Driven Unsupervised Neural Machine Translation

Unsupervised neural machine translation(NMT) is associated with noise an...
research
11/01/2016

Dual Learning for Machine Translation

While neural machine translation (NMT) is making good progress in the pa...
research
02/17/2022

End-to-End Training of Both Translation Models in the Back-Translation Framework

Semi-supervised learning algorithms in neural machine translation (NMT) ...
research
04/29/2023

Synthetic Cross-language Information Retrieval Training Data

A key stumbling block for neural cross-language information retrieval (C...

Please sign up or login with your details

Forgot password? Click here to reset