Norm-Based Curriculum Learning for Neural Machine Translation

06/03/2020
by   Xuebo Liu, et al.
0

A neural machine translation (NMT) system is expensive to train, especially with high-resource settings. As the NMT architectures become deeper and wider, this issue gets worse and worse. In this paper, we aim to improve the efficiency of training an NMT by introducing a novel norm-based curriculum learning method. We use the norm (aka length or module) of a word embedding as a measure of 1) the difficulty of the sentence, 2) the competence of the model, and 3) the weight of the sentence. The norm-based sentence difficulty takes the advantages of both linguistically motivated and model-based sentence difficulties. It is easy to determine and contains learning-dependent features. The norm-based model competence makes NMT learn the curriculum in a fully automated way, while the norm-based sentence weight further enhances the learning of the vector representation of the NMT. Experimental results for the WMT'14 English-German and WMT'17 Chinese-English translation tasks demonstrate that the proposed method outperforms strong baselines in terms of BLEU score (+1.17/+1.56) and training speedup (2.22x/3.33x).

READ FULL TEXT

page 1

page 2

page 3

page 4

07/29/2017

Curriculum Learning and Minibatch Bucketing in Neural Machine Translation

We examine the effects of particular orderings of sentence pairs on the ...
05/10/2021

Self-Guided Curriculum Learning for Neural Machine Translation

In the field of machine learning, the well-trained model is assumed to b...
10/09/2020

Self-Paced Learning for Neural Machine Translation

Recent studies have proven that the training of neural machine translati...
02/28/2019

Reinforcement Learning based Curriculum Optimization for Neural Machine Translation

We consider the problem of making efficient use of heterogeneous trainin...
09/23/2021

Exploiting Curriculum Learning in Unsupervised Neural Machine Translation

Back-translation (BT) has become one of the de facto components in unsup...
11/02/2018

An Empirical Exploration of Curriculum Learning for Neural Machine Translation

Machine translation systems based on deep neural networks are expensive ...
09/14/2022

Toward Improving Health Literacy in Patient Education Materials with Neural Machine Translation Models

Health literacy is the central focus of Healthy People 2030, the fifth i...