Adaptive Scheduling for Multi-Task Learning

09/13/2019
by   Sébastien Jean, et al.
29

To train neural machine translation models simultaneously on multiple tasks (languages), it is common to sample each task uniformly or in proportion to dataset sizes. As these methods offer little control over performance trade-offs, we explore different task scheduling approaches. We first consider existing non-adaptive techniques, then move on to adaptive schedules that over-sample tasks with poorer results compared to their respective baseline. As explicit schedules can be inefficient, especially if one task is highly over-sampled, we also consider implicit schedules, learning to scale learning rates or gradients of individual tasks instead. These techniques allow training multilingual models that perform better for low-resource language pairs (tasks with small amount of data), while minimizing negative effects on high-resource tasks.

READ FULL TEXT

page 7

page 8

page 9

page 10

research
04/08/2020

Transfer learning and subword sampling for asymmetric-resource one-to-many neural translation

There are several approaches for improving neural machine translation fo...
research
10/06/2020

Multi-task Learning for Multilingual Neural Machine Translation

While monolingual data has been shown to be useful in improving bilingua...
research
12/15/2022

Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation

Sparsely gated Mixture of Experts (MoE) models have been shown to be a c...
research
03/11/2021

Learning Policies for Multilingual Training of Neural Machine Translation Systems

Low-resource Multilingual Neural Machine Translation (MNMT) is typically...
research
01/10/2020

Learning to Multi-Task Learn for Better Neural Machine Translation

Scarcity of parallel sentence pairs is a major challenge for training hi...
research
10/06/2021

The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation

A "bigger is better" explosion in the number of parameters in deep neura...
research
12/17/2021

Mitigating inefficient task mappings with an Adaptive Resource-Moldable Scheduler (ARMS)

Efficient runtime task scheduling on complex memory hierarchy becomes in...

Please sign up or login with your details

Forgot password? Click here to reset