Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models

07/18/2019
by   Gongbo Tang, et al.
0

In this paper, we try to understand neural machine translation (NMT) via simplifying NMT architectures and training encoder-free NMT models. In an encoder-free model, the sums of word embeddings and positional embeddings represent the source. The decoder is a standard Transformer or recurrent neural network that directly attends to embeddings via attention mechanisms. Experimental results show (1) that the attention mechanism in encoder-free models acts as a strong feature extractor, (2) that the word embeddings in encoder-free models are competitive to those in conventional models, (3) that non-contextualized source representations lead to a big performance drop, and (4) that encoder-free models have different effects on alignment quality for German-English and Chinese-English.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/16/2018

Towards Robust Neural Machine Translation

Small perturbations in the input can severely distort intermediate repre...
research
04/18/2020

SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings

Word alignments are useful for tasks like statistical and neural machine...
research
06/07/2019

Shared-Private Bilingual Word Embeddings for Neural Machine Translation

Word embedding is central to neural machine translation (NMT), which has...
research
05/31/2019

Examining Structure of Word Embeddings with PCA

In this paper we compare structure of Czech word embeddings for English-...
research
11/25/2019

Learning to Reuse Translations: Guiding Neural Machine Translation with Examples

In this paper, we study the problem of enabling neural machine translati...
research
09/13/2021

Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions

This work proposes an extensive analysis of the Transformer architecture...
research
04/28/2019

Neural Machine Translation with Recurrent Highway Networks

Recurrent Neural Networks have lately gained a lot of popularity in lang...

Please sign up or login with your details

Forgot password? Click here to reset