DeepAI
Log In Sign Up

Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation

10/01/2019
by   Kenton Murray, et al.
0

Neural sequence-to-sequence models, particularly the Transformer, are the state of the art in machine translation. Yet these neural networks are very sensitive to architecture and hyperparameter settings. Optimizing these settings by grid or random search is computationally expensive because it requires many training runs. In this paper, we incorporate architecture search into a single training run through auto-sizing, which uses regularization to delete neurons in a network over the course of training. On very low-resource language pairs, we show that auto-sizing can improve BLEU scores by up to 3.9 points while removing one-third of the parameters from the model.

READ FULL TEXT
11/04/2020

Optimizing Transformer for Low-Resource Neural Machine Translation

Language pairs with limited amounts of parallel data, also known as low-...
08/17/2019

Language Graph Distillation for Low-Resource Machine Translation

Neural machine translation on low-resource language is challenging due t...
09/01/2021

Survey of Low-Resource Machine Translation

We present a survey covering the state of the art in low-resource machin...
03/11/2021

Learning Policies for Multilingual Training of Neural Machine Translation Systems

Low-resource Multilingual Neural Machine Translation (MNMT) is typically...
06/07/2021

Lexicon Learning for Few-Shot Neural Sequence Modeling

Sequence-to-sequence transduction is the core problem in language proces...
01/30/2019

The Evolved Transformer

Recent works have highlighted the strengths of the Transformer architect...
10/08/2020

Query-Key Normalization for Transformers

Low-resource language translation is a challenging but socially valuable...