Log In Sign Up

D-PAGE: Diverse Paraphrase Generation

In this paper, we investigate the diversity aspect of paraphrase generation. Prior deep learning models employ either decoding methods or add random input noise for varying outputs. We propose a simple method Diverse Paraphrase Generation (D-PAGE), which extends neural machine translation (NMT) models to support the generation of diverse paraphrases with implicit rewriting patterns. Our experimental results on two real-world benchmark datasets demonstrate that our model generates at least one order of magnitude more diverse outputs than the baselines in terms of a new evaluation metric Jeffrey's Divergence. We have also conducted extensive experiments to understand various properties of our model with a focus on diversity.


page 1

page 2

page 3

page 4


A Simple, Fast Diverse Decoding Algorithm for Neural Generation

In this paper, we propose a simple, fast decoding algorithm that fosters...

Future-Prediction-Based Model for Neural Machine Translation

We propose a novel model for Neural Machine Translation (NMT). Different...

Generating Diverse Translation from Model Distribution with Dropout

Despite the improvement of translation quality, neural machine translati...

Reducing Hallucinations in Neural Machine Translation with Feature Attribution

Neural conditional language generation models achieve the state-of-the-a...

Generating Diverse Numbers of Diverse Keyphrases

Existing keyphrase generation studies suffer from the problems of genera...

DeepGen: Diverse Search Ad Generation and Real-Time Customization

We present DeepGen, a system deployed at web scale for automatically cre...

Sentence-Permuted Paragraph Generation

Generating paragraphs of diverse contents is important in many applicati...

1 Introduction

Diversity is an essential characteristic of human language, as the meaning of a text or passage can often be restated. This work promotes the diversity of language by focusing on paraphrase generation, which aims to rephrase a text in multiple ways, while preserving its semantic meaning. Therefore, a full-fledged paraphrase generation system features two desired properties, The first is fidelity, preserving semantic meanings while paraphrasing. The second is diversity, capturing a wide range of linguistic variations.

In the past few years, deep learning models Prakash et al. (2016); Li et al. (2017) achieve superior performance than the shallow models Zhao et al. (2010); Wubben et al. (2010) on paraphrase generation, as well as a related task text simplification Nisioi et al. (2017); Zhang and Lapata (2017)

. Those deep models employ the encoder-decoder architecture to formulate paraphrasing as monolingual machine translation problem, where the encoder encodes the input texts into hidden representations, and then the decoder takes the representations as input and generates output word sequences in a word-by-word fashion.

In recent years, there are growing interests in generating lexically and syntactically diverse texts Narayan and Gardent (2014); Li et al. (2016a); Dai et al. (2017). For deep models, the techniques capable of generating diverse paraphrases fall into two categories: i) applying decoding methods such as sampling or top- beam search; ii) including random noise Dai et al. (2017); Jain et al. (2017) as model input. The former methods can always be applied to a trained deep model. As the model utilizes the same parameters, serving as a deterministic mapping from an input to one optimal output, the other outputs selected by the decoding methods are suboptimal. Moreover, empirically, the fidelity of outputs decreases as the number of outputs grows, as observed in Section 5.1. Alternatively, although adding random noise to model inputs can sometimes make slight changes to model outputs, we observe in our experiments that in most times the corresponding models generate identical outputs.

In this paper, we propose a simple but effective method, called Diverse PAraphrase GEneration (D-PAGE), extending neural machine translation (NMT) models to support multiple optimal outputs. We assume that different outputs of the same input can be explained by different rewriting patterns. The rewriting patterns are encoded in different subsets of model parameters, called pattern embeddings. In particular, we keep the encoder intact, and augment the decoder with pattern embeddings. Given an input text, the model selects appropriate pattern embeddings for different outputs so that the model uses different parameters for different input-output mappings. Different pattern embeddings may also be used to customize models for specific scenarios. For example, a child education system may prefer a model with relatively small vocabulary while a journalist assistant system prefers richer vocabulary. Our models equipped with pattern embeddings provide more flexibility than the models with fixed parameters.

To measure the diversity between models, caused by varying patterns, we introduce an evaluation metric called Jeffrey’s Divergence. According to this metric, our results on two real-world datasets show that our model is at least one order of magnitude better than the baselines. Furthermore, our model with at least one pattern embedding is able to achieve competitive fidelity as the baselines. Moreover, since it is difficult to explain the underlying complex rewriting patterns in real-world datasets, we construct synthetic datasets with atomic and interpretable patterns. On those datasets, our model is the only one among all models in comparison being able to capture the predefined patterns.

To sum up, our contributions are three-folds:

  • To the best of our knowledge, D-PAGE is the first attempt to extend NMT models for diverse outputs from model’s perspective.

  • We propose an evaluation metric and synthetic datasets for measuring diversity caused by various rewriting patterns.

  • Our model achieves substantial improvement in terms of the new evaluation metric, without compromising fidelity.

2 Related Work

Our work is closely related to the following three fields, paraphrase generation, creative language generation, and stylistic language generation.

The mainstream approaches on paraphrase generation are based on monolingual machine translation Barzilay and McKeown (2001); Ibrahim et al. (2003); Zhao et al. (2010); Wubben et al. (2012); Xu et al. (2012, 2016), where the original sequences and paraphrased sequences are treated as source and target languages, respectively. Adopting contemporary Neural Machine Translation Sutskever et al. (2014) technology, paraphrase generation systems manage to improve quality of paraphrases with better grammaticality and meaning preservation Nisioi et al. (2017); Prakash et al. (2016); Li et al. (2017); Zhang and Lapata (2017). Although previous paraphrasing systems provide multiple outputs, they focus on lexical diversity Wubben et al. (2012); Dong et al. (2017) and syntactic diversity Narayan and Gardent (2014); Narayan et al. (2016). The diversity caused by varying rewriting patterns has been largely neglected.

Recently, there has been increasing attention to the diversity of language generation systems, such as conversation systems Li et al. (2016a, b) and image caption systems Mathews et al. (2016); Gan et al. (2017)

. NAACL2016_DiversityObjFunc introduce a novel loss function to increase the usage of more interesting words.

Nema et al. (2017)

proposes to use a novel attention mechanism to reduce repeating phrases, and thus improve diversity. These two approaches treat the diverse text generation problem as improving creativity of the generator, by introducing lexical diversity. Another line of works is capable of generating multiple outputs, by introducing random noise as input of the decoder 

Dai et al. (2017) or adding noise to the encoder’s final hidden representation Jain et al. (2017).

Our work is also relevant to stylistic language generation that generates sentences with particular writing style. Xu et al. (2012) paraphrases sentences with the style of a writer, e.g. Shakespeare. SentiCap Mathews et al. (2016) investigates generating image descriptions with positive or negative sentiments. Later on, StyleNet Gan et al. (2017) proposes to generate romantic and humorous captions to increase the attractiveness of captions. These approaches can be used to generate outputs with diverse styles, while they rely on the definitions of the styles by human. To train the stylish language generation model, one should collect additional corpora with designated styles, which is expensive in many scenarios. In contrast, our model do not rely on such copora.

3 Diverse Paraphrase Generation

To generate paraphrases with patterns, D-PAGE extends the decoder of NMT models by adding multiple rewriting pattern embeddings as inputs. We apply this method to the widely used sequence-to-sequence (Seq2Seq) Sutskever et al. (2014) so that the resulted model, coined D-PAGE-K, is able to generate varying paraphrases according to different rewriting patterns.

Formally, a paraphrase generation model takes a word sequence of length as input, where denotes a word from a vocabulary . The model learns a function to map the input to a paraphrase, , where is a word also from .

Figure 1: The main structure of Diverse Paraphrase Generation method, D-PAGE.

3.1 Seq2Seq Model

Seq2Seq architecture consists of two components: (i) an encoder projects an input sentence into a representation and (ii) a decoder

generates a target word at a time to compose a word sequence. Both of the encoder and the decoder utilize Recurrent Neural Networks (RNN) to construct hidden representations, using Long Short-Term Memory (LSTM) 

Hochreiter and Schmidhuber (1997). At the th position, LSTM computes current hidden state and memory cell based on an input word embedding , its previous state and memory cell:


The mapping between the input and the output is defined as a set of the following equations:


Conventionally, , , and are referred to as input gate, forget gate and output gate, respectively.

In this work, the encoder is a (stacked) LSTM which reads a sequence of word embeddings recursively and uses the last hidden state as the final representation of the input .

The decoder consists of another (stacked) LSTM, an attention layer and a softmax classifier. The LSTM of the decoder takes the input representation from the encoder as the initial hidden state. At the

th step, the hidden state is computed by


denotes the word embeddings of words . Global attention Luong et al. (2015)

is added before the softmax layer.


The softmax classifier computes the probability of each target word



3.2 D-PAGE-K Model

There are various ways to rephrase a sentence and preserve its semantic meaning. Different rewriting patterns lead to varying actions. Since those rewriting patterns are not directly observable, D-PAGE represents them as latent embeddings. Assuming there are rewriting patterns , we denote their corresponding pattern embeddings by . A decoder is supposed to have different word choices conditioned on different rewriting patterns, thus we feed those pattern embeddings as input of the decoder. We let the pattern embeddings influence the word preference by changing the hidden representations of the decoder through LSTM.


At each time step, and are concatenated to jointly change the hidden states. From another perspective, serves as a pattern condition in addition to previously generated words and input sentences. In this way, a decoder with different rewriting patterns can be viewed as different decoders, as illustrated in Figure 1.

The LSTM parameters are shared among different decoders in order to capture the common language modeling. Otherwise it can easily lead to the explosion of parameter space. It is also beneficial to learn the pattern embeddings as model parameters instead of using randomly generated ones, as the learned embeddings can better encode the differences between rewriting patterns than the constant noisy ones.

3.3 Training and Decoding

The special challenge of training is the unobserved alignment between input sentences and rewriting patterns. We propose a customized loss, which automatically finds the hard assignment of rewriting patterns to input sentences.

For each ground truth paraphrase pair , the cross entropy loss associated with a rewriting pattern is defined as:


The above loss measures the compatibility between input and output sequences using . For the same paraphrase pair, if the is smaller than , is more likely to explain the pair than . Thus, we select the smallest loss among all rewriting patterns for each pair, which is given as:


Then we apply SGD Robbins and Monro (1951) to update the selected pattern embedding and the model parameters of the backbone Seq2Seq model associated with the smallest loss.

In the decoding phase, we apply first the encoder to construct the embedding of an input sequence. Taking as the initial hidden state, each decoder generates the most likely word sequence using beam search, conditioned on its own rewriting pattern. Thus, D-PAGE-K guarantees to generate sequences.

4 Experimental Setup

4.1 Datasets

Real-world Datasets

The same as Prakash et al. (2016), we use PPDB Pavlick et al. (2015) and Paralex Fader et al. (2013) for evaluation.

PPDB is a widely used automatically extracted multilingual paraphrase dataset. We use the most recent Phrasal English PPDB 2.0111 from the XXXL size. We randomly sample 4,500K and 500K paraphrase pairs for training and testing respectively.

Paralex is a large-scale question paraphrase corpus crawled from WikiAnswers222 Semantically similar questions are annotated by web users. As a result, the dataset contains 5,326,492 question pairs, where 4,826,492 pairs are used for training and the remaining 500K are for testing. 333We use the same dataset as Prakash et al. (2016)

Synthetic Datasets

Paraphrase generation can be viewed as translating a word sequence to another by applying certain edit operations, such as substitution and insertion. Different choices of edit operation indicate different rewriting patterns. For real-world sentences, the underlying rewriting patterns are often ambiguous or comprise a complex combination of operations. It is difficult to evaluate which patterns a model fails to capture. Therefore, we build two synthetic datasets with increasing complexities to evaluate which rewriting patterns a model is able to capture. Each pattern is associated with one or two types of edit operation. The edit operations are aligned with those used in previous works Narayan and Gardent (2014); Woodsend and Lapata (2011).

Syn-Sub, one common operation of paraphrasing is to substitute words with their synonyms. As a word could have more than one synonyms, we construct different dictionaries for the same input words. We randomly generate input sequences of length . Then we apply each dictionary to replace all words in an input sequence with their synonyms, and end up with output sequences. In this way, different dictionaries correspond to different patterns.

Syn-Scale, another common edit operation is measurement conversion. Herein, each input sequence consists of an integer sampled from and a length unit , such as ‘2357 ’. We convert each input expression to another unit , , , , and respectively, such as ‘’ and ‘’. To avoid a large vocabulary, we treat each digit, each unit of measure, as well as the decimal point as a word. Each conversion corresponds to a rewriting pattern, which comprises two types of operation: i) insertions of digits or the decimal point, and ii) one substitution of unit.

For both synthetic datasets, we randomly generated 5,000 and 1,000 input sequences for training and testing, separately. Then we transform each of them into five paraphrase sequences, and obtain 25,000 and 5,000 target sequences for training and testing, respectively.

4.2 Evaluation

We evaluate system performance under two perspectives, fidelity and diversity.

Fidelity We use multi-reference BLEU Papineni et al. (2002)

, a modified n-gram precision score, as the main metric. We also evaluate the results using SARI 

Xu et al. (2016), which emphasizes the changes in outputs against the inputs444As SARI provides similar results as BLEU, we provide the results in Appendix C..

Diversity Diversity caused by different rewriting patterns often appears as different choices of words. Distinct-N Li et al. (2016a) is the only existing evaluation metric for lexical diversity. In particular, let denote a corpus consisting of all paraphrases generated by a model , Distinct-N is the number of distinct ngrams of order N in divided by the total number of ngrams of the same order in . This corpus-level measure encourages the generation of a large number of infrequent ngrams. However, a large body of infrequent ngrams often accompany with grammatical errors. Thus, as a common practice in statistics, it is better off measuring differences of word distributions instead of distinct words.

Moreover, it is also desirable to understand the differences between rewriting patterns. Since their differences lead to different word distributions, we apply Kullback-Leibler divergence (KL)

Kullback and Leibler (1951) to characterize the distance between the corresponding word distributions. Let and denote the word distributions of a pattern and a pattern respectively, we have


where denotes the vocabulary. Since there are more than two rewriting patterns, we average KL between all pairs of word distributions as the final measure, which is equivalent to Jeffrey’s Divergence (JD) Jeffreys (1998).


A high diversity between rewriting patterns corresponds to a large JD, which is also a theoretically well grounded measure in information theory and information geometry Nock et al. (2017). Such a diversity caused by rewriting patterns is referred to as pattern diversity.

4.3 Baselines

We compare D-PAGE-K with three baselines in terms of both fidelity and diversity:

  • Beam-K Tillmann and Ney (2003) outputs top paraphrases using beam search.

  • Noise-K Dai et al. (2017) generates outputs by adding random noise , as input to the decoder.

  • VAE-K Jain et al. (2017) provides outputs by introducing noise as input right after , the hidden representations generated by the encoder.

Although Noise-K and VAE-K are not explicitly designed for generating sequences with different rewriting patterns, we can still fix the noise for the th decoder and compare the behavior of the model with different noise settings. Herein, we vary in for all models in comparison.

Model PPDB Paralex
Beam-8 [0.165, 0.148, 0.136, 0.126, [0.350, 0.317, 0.298, 0.287,
0.120, 0.114, 0.108, 0.103] 0.280, 0.276, 0.273, 0.271]
Noise-8 [0.163, 0.163, 0.164, 0.163, [0.352, 0.352,0.352,0.352,
0.163, 0.163, 0.163, 0.163] 0.352, 0.352, 0.352, 0.352]
VAE-8 [0.165, 0.165, 0.165, 0.165, [0.345, 0.345, 0.345, 0.345,
0.165, 0.165, 0.165, 0.165] 0.345, 0.345, 0.345, 0.345]
D-PAGE-2 [0.138, 0.167] [0.347, 0.329]
D-PAGE-4 [0.138, 0.117, 0.145, 0.154] [0.343, 0.350, 0.302, 0.340]
D-PAGE-8 [0.106, 0.134, 0.126, 0.134, [0.324, 0.341, 0.342, 0.332,
0.141, 0.105, 0.137, 0.126] 0.326, 0.330, 0.296, 0.243]
Table 1: Multi-reference BLEU of the baselines and D-PAGE-K, on PPDB and Paralex.
Model PPDB Paralex
Seq2Seq_Layer1 0.163 0.350
Seq2Seq_Layer2 0.164 0.351
Seq2Seq_Layer4 0.159 0.339
Table 2: Comparison of varying Seq2Seq architectures, with multi-reference BLEU.

5 Experimental Results

Our extensive experimental results on two real-world datasets show that our model achieves the highest pattern diversity according to our new evaluation metric. At least one decoder gains competitive fidelity as the competitive baselines and half decoders obtain better lexical diversity than the baselines. Our model is also the only one that captures all pre-defined rewriting patterns on the synthetic datasets.

5.1 Fidelity

The Seq2Seq model is the backbone of D-PAGE-K and the baselines. We evaluate Seq2Seq with varying number of stacked LSTM layers () for both encoder and decoder. As demonstrated in Table 2, Seq2Seq_Layer2 achieves the highest multi-reference BLEU on both benchmark corpora. We choose Seq2Seq_Layer2 as the backbone for D-PAGE-K and all three baselines also because i) the results are also on a par with those of the same architectures in Prakash et al. (2016); ii) the same architecture is the best performing one in Nisioi et al. (2017) on text simplification task.

We provide the BLEU scores of D-PAGE-K and baselines in Table 1, as well as the results of SARI in Appendix C. Compared with the Seq2Seq, at least one decoder of D-PAGE-K achieves competitive BLEU scores, on both PPDB and Paralex. One decoder of D-PAGE-2 even outperforms all Seq2Seq baselines on PPDB. As the number of decoders increases, the averaged BLEU scores of D-PAGE-K decreases. This is due to the fact that the number of training paraphrase pairs assigned to each individual decoder decreases proportionally. A sufficiently large training corpus could thus alleviate such a problem.

The fidelity of the highest ranked sequences generated by Beam-K measure up to those of the backbone model, while the BLEU scores of sequences decrease significantly as their ranks increases. On average, D-PAGE-8 outperforms Beam-8 Paralex (0.317 vs. 0.294), while keeps competitive on PPDB (0.126 vs. 0.127). Noise-K and VAE-K achieve similar BLEU scores as Seq2Seq, however their generated sequences are mostly identical, because both models are trained to be robust to input noise.

Model PPDB Paralex
distinct-1 distinct-2 JD distinct-1 distinct-2 JD
Beam-2 0.00380 0.05068 5.58E-3 0.00541 0.04464 1.93E-3
Noise-2 0.00331 0.03443 3.53E-4 0.00488 0.03350 9.06E-5
VAE-2 0.00322 0.03298 3.27E-5 0.00470 0.03248 3.17E-6
D-PAGE-2 0.00353 0.04394 1.56E-1 0.00490 0.03848 2.74E-2
Beam-4 0.00214 0.03847 5.56E-3 0.00309 0.03119 2.29E-3
Noise-4 0.00166 0.01778 3.60E-4 0.00247 0.01721 7.73E-5
VAE-4 0.00161 0.01654 2.48E-5 0.00235 0.01630 2.93E-6
D-PAGE-4 0.00188 0.02919 5.22E-1 0.00263 0.02391 4.26E-2
Beam-8 0.00109 0.02553 6.53E-3 0.00168 0.02123 4.07E-3
Noise-8 0.00084 0.00912 3.56E-4 0.00124 0.00879 7.28E-2
VAE-8 0.00081 0.00830 2.31E-5 0.00118 0.00823 5.73E-6
D-PAGE-8 0.00101 0.02055 5.26E-1 0.00142 0.01504 5.09E-2
Table 3: Distinct-1, distinct-2 and JD of baselines and D-PAGE-K, on PPDB and Paralex.

5.2 Diversity

We compare the diversity property of D-PAGE-K with baselines in terms of lexical and pattern diversities, using distinct-N and JD respectively.

Distinct-N is a measure of distinct ngrams in generated sequences. From Table 3, we can see that both D-PAGE-K and Beam-K outperform Noise-K and VAE-K in terms of this metric. Beam-K achieves the highest distinct, because it deliberately avoids repeating sentences while decoding. However, sequences between different ranks do not tend to capture different rewriting patterns, because their word distributions are similar, as evident by JD scores.

D-PAGE-K obtains dramatically higher JD than Beam-K, Noise-K and VAE-K, which supports the strong information-theoretic argument for diversity in rewriting patterns (Sec 4.2). This also demonstrates that the different decoders of D-PAGE-K have their distinct preferences in word usage patterns. With more decoders, JD of D-PAGE-K increases, while there is no such trend for other models.

Model PPDB Paralex
Beam-2 Beam-4 Beam-2 Beam-4
Seq2Seq_Layer2 0.05068 0.03847 0.04464 0.03119
D-PAGE-2 (D1) 0.03501 0.02576 0.04117 0.02800
D-PAGE-2 (D2) 0.06127 0.04581 0.04809 0.03282
D-PAGE-4 (D1) 0.04983 0.03889 0.04596 0.03135
D-PAGE-4 (D2) 0.02570 0.01919 0.03968 0.02720
D-PAGE-4 (D3) 0.03916 0.02907 0.05243 0.03602
D-PAGE-4 (D4) 0.06237 0.04795 0.03621 0.02490
Table 4: Comparison of lexical diversity of top-2 and top-4 beam search on Seq2Seq_Layer2 and D-PAGE-K, using distinct-2.

We also apply top-2 and top-4 beam search for D-PAGE-2 (with decoder D1-D2) and D-PAGE-4 (with decoder D1-D4), on both PPDB and Paralex. Distinct-2 are reported in Table 4. Half of the D-PAGE-K outperform the Seq2Seq_Layer2 models. Although D-PAGE is not essentially designed for lexical diversity, we can still obtain single models with noticeable higher vocabulary coverage, on both PPDB (D-PAGE-2(D2) and D-PAGE-4(D4)) and Paralex(D-PAGE-4(D3)).

Dataset Model AvgLen(S) AvgLen [, , ,]
Beam-4 2.438 [2.584, 2.578, 2.581, 2.594] 0.016
PPDB Noise-4 2.438 [2.609, 2.608, 2.606, 2.606] 0.003
VAE-4 2.438 [2.616, 2.617, 2.616, 2.616] 0.001
D-PAGE-4 2.438 [2.516, 3.315, 2.765, 2.084] 1.231
Beam-4 6.643 [5.844, 5.870, 5.904, 5.965] 0.121
Paralex Noise-4 6.643 [5.760, 5.763, 5.756, 5.758] 0.007
VAE-4 6.643 [5.852, 5.852, 5.853, 5.852] 0.001
D-PAGE-4 6.643 [5.961, 6.459, 5.270, 6.532] 1.262
Table 5: The average length of the sources, AvgLen(S), and average length of outputs of th decoder, , across Beam-4, Noise-4, VAE-4 and D-PAGE-4, on PPDB and Paralex.
sr: responses to sr: am pleased sr: managed to sr: partly or entirely
: responses provided : am happy : was able to : partially or totally
: replies to the : am very happy to : have been able to : partially or wholly
: the replies to : is my pleasure to : been able to : either partially or totally
: answers to : am happy : managed to : partially or totally
Table 6: Sample results, with sources (sr) and D-PAGE outputs of the th decoder , on PPDB.

5.3 Analysis

We investigate D-PAGE-K in details by using its generated sequences based on PPDB and Paralex. Inspired by the work of authorship analysis Diederich et al. (2003) and language style classification Khosmood and Levinson (2008), we analyse the average output length and the preferences of function words. We also observe the performance of D-PAGE-K on synthetic datasets and demonstrate two types of rewriting patterns that can be learned by our model.

Table 5 lists the average length of input sentences () and average length of the outputs of th decoder () for all models. We also calculate the differences between longest and shortest averaged outputs by . The average length of output sequences using D-PAGE-K is indeed longer than those of baselines. D-PAGE-K also achieves the largest on both PPDB and Paralex, larger than 1.2. This shows that D-PAGE-K encodes more diverse patterns, with some decoders prefer to generate longer, probably more complex, sentence than others. In contrast, the baselines do not feature such distinct properties.

Table 6 lists sample outputs of D-PAGE-K with on PPDB, together with source sentences (sr). In most cases, D-PAGE-K is able to replace the words with synonyms, e.g. “responses” to “replies” and “answers”, “entirely” to “wholly” and “totally”. The tense of the verbs in output phrases are also coherent with the inputs, e.g. “am” to “is”, “managed” to “was”. In some cases, our model generates duplicated outputs. For example, the output of “managed to” through is the same as the input. This is due to the limitation of Seq2Seq models, which does not guarantee the dissimilarity between the outputs and inputs. The outputs of “partly or entirely” through and are duplicated, which means that even through the models with different rewriting patterns, the outputs can still be the same.

For each decoder, we demonstrate the top 10 words contributed to JD, in Table 7. For both PPDB and Paralex, function words contribute most, such as articles, auxiliary verbs, prepositions, conjunctions, etc. Different decoders of D-PAGE-K trained on Paralex show preferences of different question words, as Parelex is composed of question paraphrases. As the selection bias of function words was used in text style analysis Diederich et al. (2003), we believe such word preference between decoders illustrates the patterns of decoders, to some extent.

D: was, are, be, its, these, those, under, during, within, all
D: the, out, been, be, a, set, has, up, to, are
D: is, been, the, to, being, of, shall, had, have, has
D: is, were, by, and, in, of, their, on, such, that
D: do, what, in, where, be, you, s, how, and, for
D: the, a, of, some, between, any, history, main, most, all
D: be, how, what, do, can, s, for, in, who, to
D: the, a, of, some, between, different, any, main, name, most
Table 7: Top 10 words contribute most to JD, for each decoder , on PPDB and Paralex. Words that appear once in the lists are bold.

In order to understand the capability of capturing specific rewriting patterns, we conduct experiments on synthetic datasets for our model. In particular, we apply decoders to generate sets of paraphrases. Since we have a reference set per rewriting pattern, we pair each paraphrase set with each reference set. For each pair, we align each system output to a single reference, so that we are able to compute a BLEU score for that pair. The corresponding results are reported in Table 8. If a model is able to capture all patterns, we expect that for each pattern, there is one decoder achieving a high score. For Syn-Sub, our model is able to learn the patterns , and , but only partially capture the pattern . Our further investigation shows that tends to provide outputs with mixed rewriting pattern between and . For Syn-Scale, although the lengths of input and output sequences are different and the dataset include two types of operations, substitution and insertion, our model can still learn all possible patterns perfectly. We contribute this to smaller vocabulary size and shorter sequence length. However, we have not observed such distinct rewriting pattern from the outputs of Noise-K or VAE-K, see Appendix D.

Dec References
Id r1 r2 r3 r4 r5
0.000 0.000 0.000 1.000 0.000
0.000 0.000 1.000 0.000 0.000
1.000 0.000 0.000 0.000 0.000
0.000 0.000 1.000 0.000 0.000
0.000 0.369 0.000 0.631 0.000
(a) Syn-Sub
Dec References
Id r1 r2 r3 r4 r5
0.047 0.577 0.719 1.000 0.612
0.066 1.000 0.693 0.598 0.386
0.055 0.671 1.000 0.736 0.502
0.033 0.407 0.507 0.607 1.000
1.000 0.066 0.057 0.049 0.032
(b) Syn-Scale
Table 8: The confusion matrices of D-PAGE on Syn-Sub and Syn-Shift using BLEU scores.

6 Conclusion

This paper presents a novel paraphrase generation method D-PAGE, which generates paraphrases with multiple rewriting patterns. Our model with different patterns can generate diverse outputs, with little loss of fidelity. D-PAGE also manages to provide models with higher lexical diversity than baselines. The experiments on synthetic datasets demonstrate the strong capacity of our model to learn common rewriting patterns.


  • Barzilay and McKeown (2001) Regina Barzilay and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of ACL.
  • Dai et al. (2017) Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. 2017. Towards diverse and natural image descriptions via a conditional gan. In Proceedings of ICCV.
  • Diederich et al. (2003) Joachim Diederich, Jörg Kindermann, Edda Leopold, and Gerhard Paass. 2003.

    Authorship attribution with support vector machines.

    Applied Intelligence, 19:109–123.
  • Dong et al. (2017) Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In Proceedings of EMNLP.
  • Fader et al. (2013) Anthony Fader, Luke S. Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proceedings of ACL.
  • Gan et al. (2017) Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. 2017. Stylenet: Generating attractive visual captions with styles. In Proceedings of CVPR.
  • Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9 8:1735–80.
  • Ibrahim et al. (2003) Ali Ibrahim, Boris Katz, and Julie Qiaojin Lin. 2003. Extracting structural paraphrases from aligned monolingual corpora.
  • Jain et al. (2017) Unnat Jain, Ziyu Zhang, and Alexander Schwing. 2017.

    Creativity: Generating diverse questions using variational autoencoders.

    In Proceedings of CVPR.
  • Jeffreys (1998) Harold Jeffreys. 1998. The theory of probability. OUP Oxford.
  • Khosmood and Levinson (2008) Foaad Khosmood and Robert A Levinson. 2008. Automatic natural language style classification and transformation.
  • Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of ACL.
  • Kullback and Leibler (1951) Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86.
  • Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of NAACL-HLT.
  • Li et al. (2016b) Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of ACL.
  • Li et al. (2017) Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2017. Paraphrase generation with deep reinforcement learning. CoRR, abs/1711.00279.
  • Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP.
  • Mathews et al. (2016) Alexander Patrick Mathews, Lexing Xie, and Xuming He. 2016. Senticap: Generating image descriptions with sentiments. In Proceedings of AAAI.
  • Narayan and Gardent (2014) Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of ACL.
  • Narayan et al. (2016) Shashi Narayan, Siva Reddy, and Shay B Cohen. 2016. Paraphrase generation from latent-variable pcfgs for semantic parsing. In Proceedings of INLG.
  • Nema et al. (2017) Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017.

    Diversity driven attention model for query-based abstractive summarization.

    In Proceedings of ACL.
  • Nisioi et al. (2017) Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P Dinu. 2017. Exploring neural text simplification models. In Proceedings of ACL, volume 2.
  • Nock et al. (2017) Richard Nock, Zac Cranko, Aditya K. Menon, Lizhen Qu, and Robert C. Williamson. 2017. f-GANs in an information-geometric nutshell. In Proceedings of NIPS.
  • Papineni et al. (2002) Kishore Papineni, Salim E. Roucos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL.
  • Pavlick et al. (2015) Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of ACL.
  • Prakash et al. (2016) Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual lstm networks. In Proceedings of COLING.
  • Robbins and Monro (1951) Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. The annals of mathematical statistics, pages 400–407.
  • Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS.
  • Tillmann and Ney (2003) Christoph Tillmann and Hermann Ney. 2003. Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational Linguistics, 29:97–133.
  • Woodsend and Lapata (2011) Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of EMNLP.
  • Wubben et al. (2010) Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2010. Paraphrase generation as monolingual translation: Data and evaluation. In INLG.
  • Wubben et al. (2012) Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of ACL.
  • Xu et al. (2015) Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association of Computational Linguistics, 3(1):283–297.
  • Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415.
  • Xu et al. (2012) Wei Xu, Alan Ritter, William B. Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In Proceedings of COLING.
  • Zhang and Lapata (2017) Xingxing Zhang and Mirella Lapata. 2017.

    Sentence simplification with deep reinforcement learning.

    In Proceedings of EMNLP.
  • Zhao et al. (2010) Shiqi Zhao, Haifeng Wang, Xiang Lan, and Ting Liu. 2010. Leveraging multiple mt engines for paraphrase generation. In Proceedings of COLING.

Appendix A Training Balance of D-PAGE

We demonstrate the proportions of training samples that get the lowest loss (match best) through the th decoder, , see Figure 2

. During the first 10 batches, the proportions vibrate sharply. After several epochs, the proportions gradually converge to the value close to 0.2 (the ideal averaged proportion). This result shows that the training samples fall into each decoder with balance. As PPDB is composed of shorter sequences than Paralex, the clustering proportions of PPDB converge better.

(a) PPDB
(b) Paralex
Figure 2: Proportions of training samples get the lowest loss through the th decoder, for the first 10 batches and epochs, on PPDB and Paralex.

Appendix B Efficiency of D-PAGE

We develop our D-PAGE-K model based on OpenNMT Klein et al. (2017). We conduct all our experiments on the server with one Nvidia Tesla P100 (SXM2). For all experiments, we train the models for 10 epochs. The total training time of Seq2Seq, with 2 layers, and D-PAGE, with 2-8 decoders, are provided in hour(h), in Table 9.

Introducing more decoders linearly increases the time for training. Paralex costs roughly 2 times more training time than PPDB, as the average sentence length of Paralex is longer than that of PPDB. The training time of D-PAGE-8 for PPDB and Paralex are acceptable, which are less than 1.5 days and 4.1 days respectively. As D-PAGE models share the calculation of encoder part, training time of D-PAGE-8 is about 5.5 times more than single decoder models and 2 times more than D-PAGE-2. The training efficiency can be improved by calculating the loss of each decoder in parallel through multiple GPUs. For decoding, each decoder for D-PAGE works as efficiently as Seq2Seq’s decoder.

Dataset Seq2Seq D-PAGE-2 D-PAGE-4 D-PAGE-8
PPDB   5.20h 10.00h 18.27h 33.12h
Paralex 14.99h 29.97h 53.40h 97.90h
Table 9: Training time consumptions of D-PAGE-K and Seq2Seq, on PPDB and Paralex.
Model PPDB Paralex
Beam-8 [0.225, 0.222, 0.221, 0.221, [0.447, 0.441, 0.438, 0.436,
0.220, 0.221, 0.220, 0.220] 0.435, 0.434, 0.435, 0.435]
Noise-8 [0.225, 0.225, 0.225, 0.225, [0.450, 0.450, 0.450, 0.450,
0.225, 0.225, 0.225, 0.225] 0.450, 0.450, 0.450, 0.450]
VAE-8 [0.224, 0.224, 0.224, 0.224, [0.446, 0.446, 0.446, 0.446,
0.224, 0.224, 0.224, 0.224] 0.446, 0.446, 0.446, 0.446]
D-PAGE-4 [0.213, 0.220, 0.228, 0.221] [0.446, 0.449, 0.435, 0.443]
D-PAGE-8 [0.215, 0.225, 0.215, 0.217, [0.443, 0.449, 0.449, 0.447,
0.223, 0.226, 0.208, 0.209] 0.439, 0.444, 0.431, 0.418]
Table 10: Fidelity experimental results on PPDB and Paralex, with SARI.

Appendix C SARI on PPDB and Paralex

The experiment result of SARI on PPDB and Paralex is demonstrated in Table 10. The overall results are similar to those of multi-reference BLEU, as discussed in Section 5.1. Surprisingly, in terms of SARI, both of the best D-PAGE-2 and D-PAGE-4 models outperform the best Beam-K demonstrates the potential of using D-PAGE to train model with higher fidelity.

Appendix D Baseline results on Synthetic Datasets

In this section, we demonstrate the performance of Noise-K and VAE-K on the Synthetic datasets, in Table 11 and Table 12. There are mainly two types of behavior for both Noise-K and VAE-K, on our synthetic datasets. i) The models totally get lost among different rewriting patterns, and fail to generate coherent results, such as Noise-K on Syn-Sub and Syn-Scale and VAE-K on Syn-Sub. ii) The models manage to learn one of the rewriting patterns, while they ignore other possible ways to paraphrase, such as VAE-K on Syn-Scale. Comparing with the results of D-PAGE, in Table 8, D-PAGE is able to detect and learn diverse rewriting operations, such as replacements and insertions.

Dec References
Id r1 r2 r3 r4 r5
D1 0.030 0.012 0.009 0.047 0.006
D2 0.031 0.011 0.009 0.047 0.004
D3 0.031 0.011 0.009 0.048 0.006
D4 0.030 0.011 0.008 0.046 0.006
D5 0.030 0.011 0.010 0.046 0.004
(a) Syn-Sub
Dec References
Id r1 r2 r3 r4 r5
D1 0.025 0.007 0.006 0.005 0.003
D2 0.022 0.004 0.004 0.003 0.002
D3 0.027 0.006 0.006 0.005 0.003
D4 0.025 0.006 0.005 0.005 0.003
D5 0.018 0.004 0.003 0.003 0.002
(b) Syn-Scale
Table 11: The confusion matrices of Noise-5 on Syn-Sub and Syn-Scale, using BLEU.
Dec References
Id r1 r2 r3 r4 r5
D1 0.184 0.263 0.204 0.231 0.119
D2 0.265 0.192 0.205 0.207 0.131
D3 0.285 0.166 0.186 0.211 0.152
D4 0.233 0.196 0.174 0.234 0.163
D5 0.202 0.193 0.164 0.264 0.178
(a) Syn-Sub
Dec References
Id r1 r2 r3 r4 r5
D1 0.899 0.907 0.911 0.904 0.882
D2 0.896 0.907 0.912 0.904 0.881
D3 0.899 0.907 0.910 0.903 0.882
D4 0.899 0.908 0.911 0.903 0.881
D5 0.900 0.909 0.912 0.903 0.880
(b) Syn-Scale
Table 12: The confusion matrices of VAE-5 on Syn-Sub and Syn-Scale, using BLEU.