Towards Robust Neural Machine Translation

05/16/2018 ∙ by Yong Cheng, et al. ∙ Tsinghua University Tencent 0

Small perturbations in the input can severely distort intermediate representations and thus impact translation quality of neural machine translation (NMT) models. In this paper, we propose to improve the robustness of NMT models with adversarial stability training. The basic idea is to make both the encoder and decoder in NMT models robust against input perturbations by enabling them to behave similarly for the original input and its perturbed counterpart. Experimental results on Chinese-English, English-German and English-French translation tasks show that our approaches can not only achieve significant improvements over strong NMT systems but also improve the robustness of NMT models.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Neural machine translation (NMT) models have advanced the state of the art by building a single neural network that can better learn representations 

Cho et al. (2014); Sutskever et al. (2014)

. The neural network consists of two components: an encoder network that encodes the input sentence into a sequence of distributed representations, based on which a decoder network generates the translation with an attention model 

Bahdanau et al. (2015); Luong et al. (2015). A variety of NMT models derived from this encoder-decoder framework have further improved the performance of machine translation systems  Gehring et al. (2017); Vaswani et al. (2017). NMT is capable of generalizing better to unseen text by exploiting word similarities in embeddings and capturing long-distance reordering by conditioning on larger contexts in a continuous way.

Input tamen bupa kunnan zuochu weiqi AI.
Output They are not afraid of difficulties to make Go AI.
Input tamen bu wei kunnan zuochu weiqi AI.
Output They are not afraid to make Go AI.
Table 1: The non-robustness problem of neural machine translation. Replacing a Chinese word with its synonym (i.e., “bupabuwei”) leads to significant erroneous changes in the English translation. Both “bupa” and “buwei” can be translated to the English phrase “be not afraid of.”

However, studies reveal that very small changes to the input can fool state-of-the-art neural networks with high probability 

Goodfellow et al. (2015); Szegedy et al. (2014). Belinkov:17 confirm this finding by pointing out that NMT models are very brittle and easily falter when presented with noisy input. In NMT, due to the introduction of RNN and attention, each contextual word can influence the model prediction in a global context, which is analogous to the “butterfly effect.” As shown in Table 1, although we only replace a source word with its synonym, the generated translation has been completely distorted. We investigate severe variations of translations caused by small input perturbations by replacing one word in each sentence of a test set with its synonym. We observe that % of translations have changed and the BLEU score is only between the translations of the original inputs and the translations of the perturbed inputs, suggesting that NMT models are very sensitive to small perturbations in the input. The vulnerability and instability of NMT models limit their applicability to a broader range of tasks, which require robust performance on noisy inputs. For example, simultaneous translation systems use automatic speech recognition (ASR) to transcribe input speech into a sequence of hypothesized words, which are subsequently fed to a translation system. In this pipeline, ASR errors are presented as sentences with noisy perturbations (the same pronunciation but incorrect words), which is a significant challenge for current NMT models. Moreover, instability makes NMT models sensitive to misspellings and typos in text translation.

In this paper, we address this challenge with adversarial stability training for neural machine translation. The basic idea is to improve the robustness of two important components in NMT: the encoder and decoder. To this end, we propose two approaches to constructing noisy inputs with small perturbations to make NMT models resist them. As important intermediate representations encoded by the encoder, they directly determine the accuracy of final translations. We introduce adversarial learning to make behaviors of the encoder consistent for both an input and its perturbed counterpart. To improve the stability of the decoder, our method jointly maximizes the likelihoods of original and perturbed data. Adversarial stability training has the following advantages:

  1. Improving both the robustness and translation performance: Our adversarial stability training is capable of not only improving the robustness of NMT models but also achieving better translation performance.

  2. Applicable to arbitrary noisy perturbations: In this paper, we propose two approaches to constructing noisy perturbations for inputs. However, our training framework can be easily extended to arbitrary noisy perturbations. Especially, we can design task-specific perturbation methods.

  3. Transparent to network architectures: Our adversarial stability training does not depend on specific NMT architectures. It can be applied to arbitrary NMT systems.

Experiments on Chinese-English, English-French and English-German translation tasks show that adversarial stability training achieves significant improvements across different languages pairs. Our NMT system outperforms the state-of-the-art RNN-based NMT system (GNMT) Wu et al. (2016) and obtains comparable performance with the CNN-based NMT system Gehring et al. (2017). Related experimental analyses validate that our training approach can improve the robustness of NMT models.

2 Background

NMT is an end-to-end framework which directly optimizes the translation probability of a target sentence given its corresponding source sentence :


where is a set of model parameters and is a partial translation. is defined on a holistic neural network which mainly includes two core components: an encoder encodes a source sentence

into a sequence of hidden representations

, and a decoder generates the -th target word based on the sequence of hidden representations:


where is the -th hidden state on target side. Thus the model parameters of NMT include the parameter sets of the encoder and the decoder : . The standard training objective is to minimize the negative log-likelihood of the training corpus :


Due to the vulnerability and instability of deep neural networks, NMT models usually suffer from a drawback: small perturbations in the input can dramatically deteriorate its translation results. Belinkov and Bisk (2018) point out that character-based NMT models are very brittle and easily falter when presented with noisy input. We find that word-based and subword-based NMT models also confront with this shortcoming, as shown in Table 1. We argue that the distributed representations should fulfill the stability expectation, which is the underlying concept of the proposed approach. Recent work has shown that adversarially trained models can be made robust to such perturbations Zheng et al. (2016); Madry et al. (2018). Inspired by this, in this work, we improve the robustness of encoder representations against noisy perturbations with adversarial learning Goodfellow et al. (2014).

3 Approach

3.1 Overview

Figure 1: The architecture of NMT with adversarial stability training. The dark solid arrow lines represent the forward-pass information flow for the input sentence , while the red dashed arrow lines for the noisy input sentence , which is transformed from by adding small perturbations.

The goal of this work is to propose a general approach to make NMT models learned to be more robust to input perturbations. Our basic idea is to maintain the consistency of behaviors through the NMT model for the source sentence and its perturbed counterpart . As aforementioned, the NMT model contains two procedures for projecting a source sentence to its target sentence : the encoder is responsible for encoding as a sequence of representations , while the decoder outputs with as input. We aim at learning the perturbation-invariant encoder and decoder.

Figure 1 illustrates the architecture of our approach. Given a source sentence , we construct a set of perturbed sentences , in which each sentence is constructed by adding small perturbations to . We require that is a subtle variation from and they have similar semantics. Given the input pair (, ), we have two expectations: (1) the encoded representation should be close to ; and (2) given , the decoder is able to generate the robust output . To this end, we introduce two additional objectives to improve the robustness of the encoder and decoder:

  • to encourage the encoder to output similar intermediate representations and for and to achieve an invariant encoder, which benefits outputting the same translations. We cast this objective in the adversarial learning framework.

  • to guide the decoder to generate output given the noisy input , which is modeled as . It can also be defined as KL divergence between and that indicates using to teach .

As seen, the two introduced objectives aim to improve the robustness of the NMT model which can be free of high variances in target outputs caused by small perturbations in inputs. It is also natural to introduce the original training objective

on and , which can guarantee good translation performance while keeping the stability of the NMT model.

Formally, given a training corpus , the adversarial stability training objective is


where and are calculated using Equation 3, and is the adversarial loss to be described in Section 3.3. and control the balance between the original translation task and the stability of the NMT model. are trainable parameters of the encoder, decoder, and the newly introduced discriminator used in adversarial learning. As seen, the parameters of encoder and decoder are trained to minimize both the translation loss and the stability losses ( and ).

Since evaluates the translation loss on the perturbed neighbour and its corresponding target sentence , it means that we augment the training data by adding perturbed neighbours, which can potentially improve the translation performance. In this way, our approach not only makes the output of NMT models more robust, but also improves the performance on the original translation task.

In the following sections, we will first describe how to construct perturbed inputs with different strategies to fulfill different goals (Section 3.2), followed by the proposed adversarial learning mechanism for the perturbation-invariant encoder (Section 3.3). We conclude this section with the training strategy (Section 3.4).

3.2 Constructing Perturbed Inputs

At each training step, we need to generate a perturbed neighbour set for each source sentence for adversarial stability training. In this paper, we propose two strategies to construct the perturbed inputs at multiple levels of representations.

The first approach generates perturbed neighbours at the lexical level. Given an input sentence , we randomly sample some word positions to be modified. Then we replace words at these positions with other words in the vocabulary according to the following distribution:


where is the word embedding for word , is the source vocabulary set excluding the word , and measures the similarity between word and . Thus we can change the word to another word with similar semantics.

One potential problem of the above strategy is that it is hard to enumerate all possible positions and possible types to generate perturbed neighbours. Therefore, we propose a more general approach to modifying the sentence at the feature level. Given a sentence, we can obtain the word embedding for each word. We add the Gaussian noise to a word embedding to simulate possible types of perturbations. That is


where the vector

is sampled from a Gaussian distribution with variance

. is a hyper-parameter. We simply introduce Gaussian noise to all of word embeddings in .

The proposed scheme is a general framework where one can freely define the strategies to construct perturbed inputs. We just present two possible examples here. The first strategy is potentially useful when the training data contains noisy words, while the latter is a more general strategy to improve the robustness of common NMT models. In practice, one can design specific strategies for particular tasks. For example, we can replace correct words with their homonyms (same pronunciation but different meanings) to improve NMT models for simultaneous translation systems.

3.3 Adversarial Learning for the Perturbation-invariant Encoder

The goal of the perturbation-invariant encoder is to make the representations produced by the encoder indistinguishable when fed with a correct sentence and its perturbed counterpart , which is directly beneficial to the output robustness of the decoder. We cast the problem in the adversarial learning framework Goodfellow et al. (2014). The encoder serves as the generator , which defines the policy that generates a sequence of hidden representations given an input sentence . We introduce an additional discriminator to distinguish the representation of perturbed input from that of the original input . The goal of the generator (i.e., encoder) is to produce similar representations for and which could fool the discriminator, while the discriminator tries to correctly distinguish the two representations.

Formally, the adversarial learning objective is


The discriminator outputs a classification score given an input representation, and tries to maximize to 1 and minimize to 0. The objective encourages the encoder to output similar representations for and , so that the discriminator fails to distinguish them.

The training procedure can be regarded as a min-max two-player game. The encoder parameters

are trained to maximize the loss function to fool the discriminator. The discriminator parameters

are optimized to minimize this loss for improving the discriminating ability. For efficiency, we update both the encoder and the discriminator simultaneously at each iteration, rather than the periodical training strategy that is commonly used in adversarial learning. Lamb et al. (2016) also propose a similar idea to use Professor Forcing to make the behaviors of RNNs be indistinguishable when training and sampling the networks.

3.4 Training

As shown in Figure 1

, our training objective includes three sets of model parameters for three modules. We use mini-batch stochastic gradient descent to optimize our model. In the forward pass, besides a mini-batch of

and , we also construct a mini-batch consisting of the perturbed neighbour and . We propagate the information to calculate these three loss functions according to arrows. Then, gradients are collected to update three sets of model parameters. Except for the gradients of with respect to are multiplying by , other gradients are normally back-propagated. Note that we update and simultaneously for training efficiency.

System Training MT06 MT02 MT03 MT04 MT05 MT08
Shen et al. (2016) MRT 37.34 40.36 40.93 41.37 38.81 29.23
Wang et al. (2017) MLE 37.29 39.35 41.15 38.07
Zhang et al. (2018) MLE 38.38 40.02 42.32 38.84
this work MLE 41.38 43.52 41.50 43.64 41.58 31.60
AST 43.57 44.82 42.95 45.05 43.45 34.85
AST 44.44 46.10 44.07 45.61 44.06 34.94
Table 2: Case-insensitive BLEU scores on Chinese-English translation.
System Architecture Training BLEU
Shen et al. (2016) Gated RNN with 1 layer MRT 20.45
Luong et al. (2015) LSTM with 4 layers MLE 20.90
Kalchbrenner et al. (2017) ByteNet with 30 layers MLE 23.75
Wang et al. (2017) DeepLAU with 4 layers MLE 23.80
Wu et al. (2016) LSTM with 8 layers RL 24.60
Gehring et al. (2017) CNN with 15 layers MLE 25.16
Vaswani et al. (2017) Self-attention with 6 layers MLE 28.40
this work Gated RNN with 2 layers MLE 24.06
AST 25.17
AST 25.26
Table 3: Case-sensitive BLEU scores on WMT 14 English-German translation.
Training tst2014 tst2015
MLE 36.92 36.90
AST 37.35 37.03
AST 38.03 37.64
Table 4: Case-sensitive BLEU scores on IWSLT English-French translation.

4 Experiments

4.1 Setup

We evaluated our adversarial stability training on translation tasks of several language pairs, and reported the 4-gram BLEU Papineni et al. (2002) score as calculated by the multi-bleu.perl script.

Chinese-English We used the LDC corpus consisting of 1.25M sentence pairs with 27.9M Chinese words and 34.5M English words respectively. We selected the best model using the NIST 2006 set as the validation set (hyper-parameter optimization and model selection). The NIST 2002, 2003, 2004, 2005, and 2008 datasets are used as test sets.

English-German We used the WMT 14 corpus containing 4.5M sentence pairs with 118M English words and 111M German words. The validation set is newstest2013, and the test set is newstest2014.

English-French We used the IWSLT corpus which contains 0.22M sentence pairs with 4.03M English words and 4.12M French words. The IWLST corpus is very dissimilar from the NIST and WMT corpora. As they are collected from TED talks and inclined to spoken language, we want to verify our approaches on the non-normative text. The IWSLT 14 test set is taken as the validation set and 15 test set is used as the test set.

For English-German and English-French, we tokenize both English, German and French words using tokenize.perl script. We follow Sennrich et al. (2016b) to split words into subword units. The numbers of merge operations in byte pair encoding (BPE) are set to 30K, 40K and 30K respectively for Chinese-English, English-German, and English-French. We report the case-sensitive tokenized BLEU score for English-German and English-French and the case-insensitive tokenized BLEU score for Chinese-English.

Our baseline system is an in-house NMT system. Following Bahdanau et al. (2015)

, we implement an RNN-based NMT in which both the encoder and decoder are two-layer RNNs with residual connections between layers

He et al. (2016b)

. The gating mechanism of RNNs is gated recurrent unit (GRUs)

Cho et al. (2014). We apply layer normalization Ba et al. (2016) and dropout Hinton et al. (2012)

to the hidden states of GRUs. Dropout is also added to the source and target word embeddings. We share the same matrix between the target word embeedings and the pre-softmax linear transformation

Vaswani et al. (2017). We update the set of model parameters using Adam SGD Kingma and Ba (2015). Its learning rate is initially set to and varies according to the formula in Vaswani et al. (2017).

Our adversarial stability training initializes the model based on the parameters trained by maximum likelihood estimation (MLE). We denote adversarial stability training based on lexical-level perturbations and feature-level perturbations respectively as AST

and AST. We only sample one perturbed neighbour for training efficiency. For the discriminator used in , we adopt the CNN discriminator proposed by Kim (2014)

to address the variable-length problem of the sequence generated by the encoder. In the CNN discriminator, the filter windows are set to 3, 4, 5 and rectified linear units are applied after convolution operations. We tune the hyper-parameters on the validation set through a grid search. We find that both the optimal values of

and are set to . The standard variance in Gaussian noise used in the formula (6) is set to . The number of words that are replaced in the sentence during lexical-level perturbations is taken as in which is the length of . The default beam size for decoding is .

4.2 Translation Results

4.2.1 NIST Chinese-English Translation

Table 2 shows the results on Chinese-English translation. Our strong baseline system significantly outperforms previously reported results on Chinese-English NIST datasets trained on RNN-based NMT. Shen et al. (2016) propose minimum risk training (MRT) for NMT, which directly optimizes model parameters with respect to BLEU scores. Wang et al. (2017) address the issue of severe gradient diffusion with linear associative units (LAU). Their system is deep with an encoder of 4 layers and a decoder of 4 layers. Zhang et al. (2018) propose to exploit both left-to-right and right-to-left decoding strategies for NMT to capture bidirectional dependencies. Compared with them, our NMT system trained by MLE outperforms their best models by around 3 BLEU points. We hope that the strong baseline systems used in this work make the evaluation convincing.

We find that introducing adversarial stability training into NMT can bring substantial improvements over previous work (up to BLEU points over Shen et al. (2016), up to BLEU points over Wang et al. (2017) and up to BLEU points over Zhang et al. (2018)) and our system trained with MLE across all the datasets. Compared with our baseline system, AST achieves BLEU improvement on average. AST performs better, which can obtain BLEU points on average and up to BLEU points on NIST08.

Synthetic Type Training 0 Op. 1 Op. 2 Op. 3 Op. 4 Op. 5 Op.
Swap MLE 41.38 38.86 37.23 35.97 34.61 32.96
AST 43.57 41.18 39.88 37.95 37.02 36.16
AST 44.44 42.08 40.20 38.67 36.89 35.81
Replacement MLE 41.38 37.21 31.40 27.43 23.94 21.03
AST 43.57 40.53 37.59 35.19 32.56 30.42
AST 44.44 40.04 35.00 30.54 27.42 24.57
Deletion MLE 41.38 38.45 36.15 33.28 31.17 28.65
AST 43.57 41.89 38.56 36.14 34.09 31.77
AST 44.44 41.75 39.06 36.16 33.49 30.90
Table 5: Translation results of synthetic perturbations on the validation set in Chinese-English translation. “1 Op.” denotes that we conduct one operation (swap, replacement or deletion) on the original sentence.
Source zhongguo dianzi yinhang yewu guanli xingui jiangyu sanyue yiri qi shixing
Reference china’s new management rules for e-banking operations to take effect on march 1
MLE china’s electronic bank rules to be implemented on march 1
AST new rules for business administration of china ’s electronic banking industry
will come into effect on march 1 .
AST new rules for business management of china ’s electronic banking industry to
come into effect on march 1
Perturbed Source zhong fang dianzi yinhang yewu guanli xingui jiangyu sanyue yiri qi shixing
MLE china to implement new regulations on business management
AST the new regulations for the business administrations of the chinese electronics
bank will come into effect on march 1 .
AST new rules for business management of china’s electronic banking industry to
come into effect on march 1
Table 6: Example translations of a source sentence and its perturbed counterpart by replacing a Chinese word “zhongguo” with its synonym “zhongfang.”

4.2.2 WMT 14 English-German Translation

In Table 3, we list existing NMT systems as comparisons. All these systems use the same WMT 14 English-German corpus. Except that Shen et al. (2016) and Wu et al. (2016)

respectively adopt MRT and reinforcement learning (RL), other systems all use MLE as training criterion. All the systems except for

Shen et al. (2016) are deep NMT models with no less than four layers. Google’s neural machine translation (GNMT) Wu et al. (2016) represents a strong RNN-based NMT system. Compared with other RNN-based NMT systems except for GNMT, our baseline system with two layers can achieve better performance than theirs.

When training our NMT system with AST, significant improvement ( BLEU points) can be observed. AST can obtain slightly better performance. Our NMT system outperforms the state-of-the-art RNN-based NMT system, GNMT, with BLEU point and performs comparably with Gehring et al. (2017) which is based on CNN with 15 layers. Given that our approach can be applied to any NMT systems, we expect that the adversarial stability training mechanism can further improve performance upon the advanced NMT architectures. We leave this for future work.

4.2.3 IWSLT English-French Translation

Table 4 shows the results on IWSLT English-French Translation. Compared with our strong baseline system trained by MLE, we observe that our models consistently improve translation performance in all datasets. AST can achieve significant improvements on the tst2015 although AST obtains comparable results. These demonstrate that our approach maintains good performance on the non-normative text.

4.3 Results on Synthetic Perturbed Data

In order to investigate the ability of our training approaches to deal with perturbations, we experiment with three types of synthetic perturbations:

  • Swap: We randomly choose positions from a sentence and then swap the chosen words with their right neighbours.

  • Replacement: We randomly replace sampled words in the sentence with other words.

  • Deletion: We randomly delete words from each sentence in the dataset.

As shown in Table 5, we can find that our training approaches, AST and AST, consistently outperform MLE against perturbations on all the numbers of operations. This means that our approaches have the capability of resisting perturbations. Along with the number of operations increasing, the performance on MLE drops quickly. Although the performance of our approaches also drops, we can see that our approaches consistently surpass MLE. In AST, with 0 operation, the difference is +2.19 (43.57 Vs. 41.38) for all synthetic types, but the differences are enlarged to +3.20, +9.39, and +3.12 respectively for the three types with 5 operations.

In the Swap and Deletion types, AST and AST perform comparably after more than four operations. Interestingly, AST performs significantly better than both of MLE and AST after more than one operation in the Replacement type. This is because AST trains the model specifically on perturbation data that is constructed by replacing words, which agrees with the Replacement Type. Overall, AST performs better than AST against perturbations after multiple operations. We speculate that the perturbation method for AST and synthetic type are both discrete and they keep more consistent. Table 6 shows example translations of a Chinese sentence and its perturbed counterpart.

These findings indicate that we can construct specific perturbations for a particular task. For example, in simultaneous translation, an automatic speech recognition system usually generates wrong words with the same pronunciation of correct words, which dramatically affects the quality of machine translation system. Therefore, we can design specific perturbations aiming for this task.

4.4 Analysis

Table 7: Ablation study of adversarial stability training AST on Chinese-English translation. “” means the loss function is included in the training objective while “” means it is not.
Figure 2: BLEU scores of AST over iterations on Chinese-English validation set.
Figure 3: Learning curves of three loss functions, , and over iterations on Chinese-English validation set.

4.4.1 Ablation Study

Our training objective function Eq. (4) contains three loss functions. We perform an ablation study on the Chinese-English translation to understand the importance of these loss functions by choosing AST as an example. As Table 7 shows, if we remove , the translation performance decreases by BLEU point. However, when is excluded from the training objective function, it results in a significant drop of BLEU point. Surprisingly, only using is able to lead to an increase of BLEU point.

4.4.2 BLEU Scores over Iterations

Figure 2 shows the changes of BLEU scores over iterations respectively for AST and AST. They behave nearly consistently. Initialized by the model trained by MLE, their performance drops rapidly. Then it starts to go up quickly. Compared with the starting point, the maximal dropping points reach up to about BLEU points. Basically, the curves present the state of oscillation. We think that introducing random perturbations and adversarial learning can make the training not very stable like MLE.

4.4.3 Learning Curves of Loss Functions

Figure 3 shows the learning curves of three loss functions, , and . We can find that their costs of loss functions decrease not steadily. Similar to the Figure 2, there still exist oscillations in the learning curves although they do not change much sharply. We find that converges to around after about iterations, which indicates that discriminator outputs probability for both positive and negative samples and it cannot distinguish them. Thus the behaviors of the encoder for and its perturbed neighbour perform nearly consistently.

5 Related Work

Our work is inspired by two lines of research: (1) adversarial learning and (2) data augmentation.

Adversarial Learning

Generative Adversarial Network (GAN) Goodfellow et al. (2014)

and its related derivative have been widely applied in computer vision 

Radford et al. (2015); Salimans et al. (2016)

and natural language processing 

Li et al. (2017); Yang et al. (2018). Previous work has constructed adversarial examples to attack trained networks and make networks resist them, which has proved to improve the robustness of networks Goodfellow et al. (2015); Miyato et al. (2016); Zheng et al. (2016). Belinkov and Bisk (2018) introduce adversarial examples to training data for character-based NMT models. In contrast to theirs, adversarial stability training aims to stabilize both the encoder and decoder in NMT models. We adopt adversarial learning to learn the perturbation-invariant encoder.

Data Augmentation

Data augmentation has the capability to improve the robustness of NMT models. In NMT, there is a number of work that augments the training data with monolingual corpora Sennrich et al. (2016a); Cheng et al. (2016); He et al. (2016a); Zhang and Zong (2016). They all leverage complex models such as inverse NMT models to generate translation equivalents for monolingual corpora. Then they augment the parallel corpora with these pseudo corpora to improve NMT models. Some authors have recently endeavored to achieve zero-shot NMT through transferring knowledge from bilingual corpora of other language pairs Chen et al. (2017); Zheng et al. (2017); Cheng et al. (2017) or monolingual corpora Lample et al. (2018); Artetxe et al. (2018). Our work significantly differs from these work. We do not resort to any complicated models to generate perturbed data and do not depend on extra monolingual or bilingual corpora. The way we exploit is more convenient and easy to implement. We focus more on improving the robustness of NMT models.

6 Conclusion

We have proposed adversarial stability training to improve the robustness of NMT models. The basic idea is to train both the encoder and decoder robust to input perturbations by enabling them to behave similarly for the original input and its perturbed counterpart. We propose two approaches to construct perturbed data to adversarially train the encoder and stabilize the decoder. Experiments on Chinese-English, English-German and English-French translation tasks show that the proposed approach can improve both the robustness and translation performance.

As our training framework is not limited to specific perturbation types, it is interesting to evaluate our approach in natural noise existing in practical applications, such as homonym in the simultaneous translation system. It is also necessary to further validate our approach on more advanced NMT architectures, such as CNN-based NMT Gehring et al. (2017) and Transformer Vaswani et al. (2017).


We thank the anonymous reviewers for their insightful comments and suggestions. We also thank Xiaoling Li for analyzing experimental results and providing valuable examples. Yang Liu is supported by the National Key R&D Program of China (No. 2017YFB0202204), National Natural Science Foundation of China (No. 61761166008, No. 61522204), Beijing Advanced Innovation Center for Language Resources, and the NExT++ project supported by the National Research Foundation, Prime Minister’s Office, Singapore under its IRC@Singapore Funding Initiative.


  • Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of ICLR.
  • Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
  • Bahdanau et al. (2015) Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR.
  • Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR.
  • Chen et al. (2017) Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. In Proceedings of ACL.
  • Cheng et al. (2016) Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Proceedings of ACL.
  • Cheng et al. (2017) Yong Cheng, Qian Yang, Yang Liu, Maosong Sun, and Wei Xu. 2017. Joint training for pivot-based neural machine translation. In Proceedings of IJCAI.
  • Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of EMNLP.
  • Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of ICML.
  • Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of NIPS.
  • Goodfellow et al. (2015) Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of ICLR.
  • He et al. (2016a) Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016a. Dual learning for machine translation. In Proceedings of NIPS.
  • He et al. (2016b) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Deep residual learning for image recognition. In Proceedings of CVPR.
  • Hinton et al. (2012) Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580.
  • Kalchbrenner et al. (2017) Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2017. Neural machine translation in linear time. In Proceedings of ICML.
  • Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP.
  • Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR.
  • Lamb et al. (2016) Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Proceedings of NIPS.
  • Lample et al. (2018) Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In Proceedings of ICLR.
  • Li et al. (2017) Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Proceedings of EMNLP.
  • Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP.
  • Madry et al. (2018) Aleksander Madry, Makelov Aleksandar, Schmidt Ludwig, Dimitris Tsipras, and Adrian Vladu. 2018.

    Towards deep learning models resistant to adversarial attacks.

    In Proceedings of ICLR.
  • Miyato et al. (2016) Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2016. Distributional smoothing with virtual adversarial training. In Proceedings of ICLR.
  • Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a methof for automatic evaluation of machine translation. In Proceedings of ACL.
  • Radford et al. (2015) Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
  • Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Proceedings of NIPS.
  • Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving nerual machine translation models with monolingual data. In Proceedings of ACL.
  • Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of ACL.
  • Shen et al. (2016) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of ACL.
  • Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceddings of NIPS.
  • Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Sutskever Ilya, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of ICML.
  • Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
  • Wang et al. (2017) Mingxuan Wang, Zhengdong Lu, Jie Zhou, and Qun Liu. 2017. Deep neural machine translation with linear associative unit. In Proceedings of ACL.
  • Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
  • Yang et al. (2018) Z. Yang, W. Chen, F. Wang, and B. Xu. 2018. Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets. In Proceedings of NAACL.
  • Zhang and Zong (2016) Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of EMNLP.
  • Zhang et al. (2018) Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. 2018. Asynchronous Bidirectional Decoding for Neural Machine Translation. In Proeedings of AAAI.
  • Zheng et al. (2017) Hao Zheng, Yong Cheng, and Yang Liu. 2017. Maximum expected likelihood estimation for zero-resource neural machine translation. In Proceedings of IJCAI.
  • Zheng et al. (2016) Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. 2016. Improving the robustness of deep neural networks via stability training. In Proceedings of CVPR.