Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

06/14/2016
by   Jie Zhou, et al.
0

Neural machine translation (NMT) aims at solving machine translation (MT) problems using neural networks and has exhibited promising results in recent years. However, most of the existing NMT models are shallow and there is still a performance gap between a single NMT model and the best conventional MT system. In this work, we introduce a new type of linear connections, named fast-forward connections, based on deep Long Short-Term Memory (LSTM) networks, and an interleaved bi-directional architecture for stacking the LSTM layers. Fast-forward connections play an essential role in propagating the gradients and building a deep topology of depth 16. On the WMT'14 English-to-French task, we achieve BLEU=37.7 with a single attention model, which outperforms the corresponding single shallow model by 6.2 BLEU points. This is the first time that a single NMT model achieves state-of-the-art performance and outperforms the best conventional model by 0.7 BLEU points. We can still achieve BLEU=36.3 even without using an attention mechanism. After special handling of unknown words and model ensembling, we obtain the best score reported to date on this task with BLEU=40.4. Our models are also validated on the more difficult WMT'14 English-to-German task.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/17/2015

Effective Approaches to Attention-based Neural Machine Translation

An attentional mechanism has lately been used to improve neural machine ...
research
05/02/2017

Deep Neural Machine Translation with Linear Associative Unit

Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art ...
research
10/10/2016

Neural Paraphrase Generation with Stacked Residual LSTM Networks

In this paper, we propose a novel neural approach for paraphrase generat...
research
12/19/2018

DTMT: A Novel Deep Transition Architecture for Neural Machine Translation

Past years have witnessed rapid developments in Neural Machine Translati...
research
12/26/2019

Amharic-Arabic Neural Machine Translation

Many automatic translation works have been addressed between major Europ...
research
04/29/2020

Multiscale Collaborative Deep Models for Neural Machine Translation

Recent evidence reveals that Neural Machine Translation (NMT) models wit...
research
11/05/2019

Training Neural Machine Translation (NMT) Models using Tensor Train Decomposition on TensorFlow (T3F)

We implement a Tensor Train layer in the TensorFlow Neural Machine Trans...

Please sign up or login with your details

Forgot password? Click here to reset