On Compositionality in Neural Machine Translation

11/04/2019
by   Vikas Raunak, et al.
0

We investigate two specific manifestations of compositionality in Neural Machine Translation (NMT) : (1) Productivity - the ability of the model to extend its predictions beyond the observed length in training data and (2) Systematicity - the ability of the model to systematically recombine known parts and rules. We evaluate a standard Sequence to Sequence model on tests designed to assess these two properties in NMT. We quantitatively demonstrate that inadequate temporal processing, in the form of poor encoder representations is a bottleneck for both Productivity and Systematicity. We propose a simple pre-training mechanism which alleviates model performance on the two properties and leads to a significant improvement in BLEU scores.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/26/2017

English-Japanese Neural Machine Translation with Encoder-Decoder-Reconstructor

Neural machine translation (NMT) has recently become popular in the fiel...
research
05/19/2023

Pseudo-Label Training and Model Inertia in Neural Machine Translation

Like many other machine learning applications, neural machine translatio...
research
02/27/2020

Echo State Neural Machine Translation

We present neural machine translation (NMT) models inspired by echo stat...
research
10/06/2020

On the Sparsity of Neural Machine Translation Models

Modern neural machine translation (NMT) models employ a large number of ...
research
08/22/2019

The compositionality of neural networks: integrating symbolism and connectionism

Despite a multitude of empirical studies, little consensus exists on whe...
research
09/23/2019

Data Ordering Patterns for Neural Machine Translation: An Empirical Study

Recent works show that ordering of the training data affects the model p...
research
10/07/2019

On Leveraging the Visual Modality for Neural Machine Translation

Leveraging the visual modality effectively for Neural Machine Translatio...

Please sign up or login with your details

Forgot password? Click here to reset