Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

by   Yonghui Wu, et al.

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60 compared to Google's phrase-based production system.


page 1

page 2

page 3

page 4


Six Challenges for Neural Machine Translation

We explore six challenges for neural machine translation: domain mismatc...

Neural Machine Translation of Logographic Languages Using Sub-character Level Information

Recent neural machine translation (NMT) systems have been greatly improv...

Neural Machine Translation with Characters and Hierarchical Encoding

Most existing Neural Machine Translation models use groups of characters...

Towards one-shot learning for rare-word translation with external experts

Neural machine translation (NMT) has significantly improved the quality ...

Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU

Attentional sequence-to-sequence models have become the new standard for...

Return of the RNN: Residual Recurrent Networks for Invertible Sentence Embeddings

This study presents a novel model for invertible sentence embeddings usi...

Deep Neural Machine Translation with Linear Associative Unit

Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art ...

Code Repositories


Neural Machine Translation of WordPress Strings

view repo


A sequence-to-sequence model with attention for Machine Translation

view repo

Please sign up or login with your details

Forgot password? Click here to reset