Dynamic Curriculum Learning for Low-Resource Neural Machine Translation

11/30/2020
by   Bojie Hu, et al.
0

Large amounts of data has made neural machine translation (NMT) a big success in recent years. But it is still a challenge if we train these models on small-scale corpora. In this case, the way of using data appears to be more important. Here, we investigate the effective use of training data for low-resource NMT. In particular, we propose a dynamic curriculum learning (DCL) method to reorder training samples in training. Unlike previous work, we do not use a static scoring function for reordering. Instead, the order of training samples is dynamically determined in two ways - loss decline and model competence. This eases training by highlighting easy samples that the current model has enough competence to learn. We test our DCL method in a Transformer-based system. Experimental results show that DCL outperforms several strong baselines on three low-resource machine translation benchmarks and different sized data of WMT' 16 En-De.

READ FULL TEXT

page 1

page 2

page 3

page 4

08/18/2017

Neural machine translation for low-resource languages

Neural machine translation (NMT) approaches have improved the state of t...
03/20/2021

Token-wise Curriculum Learning for Neural Machine Translation

Existing curriculum learning approaches to Neural Machine Translation (N...
05/20/2019

Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation

To improve low-resource Neural Machine Translation (NMT) with multilingu...
03/23/2019

Competence-based Curriculum Learning for Neural Machine Translation

Current state-of-the-art NMT systems use large neural networks that are ...
03/03/2021

Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation

Meta-learning has been sufficiently validated to be beneficial for low-r...
01/14/2022

Cost-Effective Training in Low-Resource Neural Machine Translation

While Active Learning (AL) techniques are explored in Neural Machine Tra...
04/07/2020

Self-Induced Curriculum Learning in Neural Machine Translation

Self-supervised neural machine translation (SS-NMT) learns how to extrac...