Training Flexible Depth Model by Multi-Task Learning for Neural Machine Translation

by   Qiang Wang, et al.

The standard neural machine translation model can only decode with the same depth configuration as training. Restricted by this feature, we have to deploy models of various sizes to maintain the same translation latency, because the hardware conditions on different terminal devices (e.g., mobile phones) may vary greatly. Such individual training leads to increased model maintenance costs and slower model iterations, especially for the industry. In this work, we propose to use multi-task learning to train a flexible depth model that can adapt to different depth configurations during inference. Experimental results show that our approach can simultaneously support decoding in 24 depth configurations and is superior to the individual training and another flexible depth model training method – LayerDrop.


Neural Machine Translation for Bilingually Scarce Scenarios: A Deep Multi-task Learning Approach

Neural machine translation requires large amounts of parallel training t...

Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning

Linguistic resources such as part-of-speech (POS) tags have been extensi...

NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems

In this paper, we present nmtpy, a flexible Python toolkit based on Thea...

A Multi-Task Architecture on Relevance-based Neural Query Translation

We describe a multi-task learning approach to train a Neural Machine Tra...

Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation

Non-Autoregressive machine Translation (NAT) models have demonstrated si...

Masked Adversarial Generation for Neural Machine Translation

Attacking Neural Machine Translation models is an inherently combinatori...

Learning Flexible Translation between Robot Actions and Language Descriptions

Handling various robot action-language translation tasks flexibly is an ...