Text generation refers to the task of generating a new text based on some user input. The task exists in many forms, but arguably the most common form concerns generating a coherent and consistent text based on an input context such as the first few sentences of the target output. This is often achieved by giving the context to a generative language model. Generative language models play a central role in machine learning and natural language processing (NLP). Not only they serve as the main mean for unsupervised feature representation learning but also find use in various applications, including question answering, dialogue agents, summarization, and content creation systems.
Thanks to the introduction of novel deep learning architectures and the availability of large-scale training corpora, the state-of-the-art text generation has advanced significantly in recent years. We can now train language models capable of generating fluent and coherent texts that people cannot tell them apart from those written by humans. However, despite the great achievement, existing generative models are limited and inflexible in the sense that a trained model is only capable in generating texts of one style. It can not be used to generate texts of different styles. For instance, a news generative model can only be used to generate news, and a lyric generative model can only be used to generate lyrics. In contrast, humans can compose texts in various styles.
To bridge the gap, we propose a style example-guided text generation framework that can generate styled texts based on the style of the example reference text. In our framework, the generator takes two inputs where one is the context input while the other is the style reference example. We use the style reference example to change the generation behavior of our generative model dynamically. For a fixed context, when the provided style reference is a paragraph sampled from a news article, it becomes a news generator. When the provided style reference is a review, it becomes a review generator. In other words, the same generator can generate texts of different styles based on the examples. In Tab. 1, we show example outputs of the proposed framework where we generate texts of dramatically different styles for the same input sentence.
The proposed style example-guided text generation framework is based on the generative adversarial networks (GANs), and we utilize the transformer in both the generator and discriminator design. We collect a large dataset containing documents with many different styles for training. Using a novel learning objective function, our network learns to generate styled texts based on the input style example in an unsupervised manner. We conduct extensive experimental validations with comparisons to strong baselines. We also investigate different ways of designing the generator and compare their performance. Through detailed quantitative and user study results, we prove the effectiveness of the proposed framework for the style example-guided text generation task.
2 Related Work
has seen many advancements in recent years, which has resulted in significant improvements on various NLP tasks. Early language models focused on using n-grams to represent a text distribution.Bengio et al. (2003) introduced a neural language model in a shift from more traditional n-gram models. Many works later (Mikolov et al. (2013); Pennington et al. (2014)) focused on word embeddings as a way to represent tokens within the text. More recently, Peters et al. (2018) used bi-directional LSTMs to obtain deep contextualized word representation. However, RNNs can only represent a limited context. Vaswani et al. (2017)
introduced the transformer networks which use the connections between long-distance word pairs embedded in attention mechanisms and can easily enable the learning of long-term dependency. Many later models (Devlin et al. (2019); Liu et al. (2019d); Dai et al. (2019); Yang et al. (2019)) used transformer model and obtained significant improvements on downstream tasks (Wang et al. (2019); Rajpurkar et al. (2016); Zellers et al. (2018)). Lately, (Radford et al. (2019)) introduced GPT-2, a generative left-to-right language model based on the transformer and showed that these models are able to generate coherent text when pretrained on a large corpus. Shoeybi et al. (2019) further scaled up the GPT-2 model and demonstrated improved performance. Our work differs from the prior works because we aim for allowing user flexible control over the style of the generated text.
Texts generation includes review generation (Radford et al. (2018); Zang and Wan (2017)), sentiment texts generation (Wang and Wan (2018); Hu et al. (2017); Merity et al. (2017)), Wikipedia generation (Liu et al. (2018); Lebret et al. (2016)), fake news generation (Bakhtin et al. (2019); Zellers et al. (2019)), abstractive summarization (Li et al. (2018); Zhang et al. (2019); Pasunuru et al. (2017)), and conversation/dialogue system (Vinyals and Le (2015); Budzianowski and Vulić (2019)). Although many of them trained a transformer on large-scale corpora, their results were limited in their specific domain (e.g., reviews, news, etc.) because they either utilized domain-specific priors in their model design or were not designed to generate texts in many different domains or styles.
), variational autoencoder-based (Xu et al. (2019)), normalizing flow-based (Tran et al. (2019)) approaches for general texts generation task. However, we instead focus on generating styled paragraphs conditioning on a context and a reference paragraph. A recent work by Keskar et al. (2019) is most related to ours. They propose a conditional transformer using a control code to perform language generation in a sequence-to-sequence manner. We demonstrate our method outperforms theirs by a large margin in the experiment section.
Text style transfer concerns transferring an input text of one style to a different style (Kerpedjiev (1992); Rao and Tetreault (2018); Xu (2017); Xu et al. (2012); Fu et al. (2018); Hu et al. (2017); Prabhumoye et al. (2018); Shen et al. (2017); Li et al. (2019)). Our work is different since we do not aim for changing the style of a given text. Instead, we aim for a style-controllable way for generating texts from scratch. Also, rather than handling transferring between two styles (e.g., positive negative sentiments), our model can generate texts of many different styles. Finally, our model outputs paragraphs while existing text style transfer works mostly output sentences.
Image Style transfer
is a popular topic in computer vision. There are many successful techniques, including iterative optimization on the gram matrix (Gatys et al. (2016)), perceptual loss (Johnson et al. (2016); Gupta et al. (2017)), feature transformation (Li et al. (2017)), adaptive instance-normalization (Dumoulin et al. (2017); Huang and Belongie (2017)), and GAN-based methods (Zhu et al. (2017); Kim et al. (2017)). Our proposed framework also gets inspiration from them.
Transformer is the state-of-the-art network for various natural language processing tasks. Different from RNNs (Hochreiter and Schmidhuber (1997); Bengio et al. (2003); Chung et al. (2014)), which consume a sequence token by token, in a transformer network, the entire sequence is fed into layers of transformer modules. The representation of a token at a layer is then computed by attending to the latent representations of all the other tokens in the preceding layer.
Variants of transformer networks are available. We build our model based on GPT-2 transformer network (Radford et al. (2019); Shoeybi et al. (2019)), which train a deep transformer using a left-to-right language model:
where ’s denote the word tokens. Different from BERT-like transformer networks (Devlin et al. (2019); Liu et al. (2019d)), GPT-2 makes a casual assumption, i.e., the latent representation of a token is calculated using only the latent representations of the preceding tokens. Thus, during generation, GPT-2 can be directly applied to complete the text given the context sentence.
GAN defines a zero-sum game played by a generator and a discriminator
. Under some nice conditions, the generator learns to convert a random noise vector to a realistic signal in a way that the discriminator cannot tell it apart from real signals. In this case, the distribution of the output signals produced by the generator converges to the distribution of signals observed in the real world.
We use a conditional GAN where takes a context sentence and a style reference example as inputs. To avoid non-differentiability in text decoding (e.g., beam search), we use a latent GAN formulation (Achlioptas et al. (2017)). We first divide into a feature extractor and an output embedding layer ; that is . Now, instead of using the output text from as the discriminator input, we feed the latent representation computed by to the discriminator. For real text, we use a pretrained trained GPT-2 model . Again, we decompose into a feature extractor and an output embedding layer (
). The GAN discriminator then takes features extracted byas input for real texts. Using this latent GAN formulation, we aim for aligning the feature distribution of our generator to the feature distribution of the pretrained GPT-2 model.
4 Style Example-Guided Text Generation
We propose a language generative model framework that allows us to control style of the output text using a style reference example. Given few context sentences and a reference text , our generator generates output text that has the same style as the reference example given by
We divide the feature extractor into a style encoder and a text decoder where the style encoder extracts a style representation from the style example, , and the text decoder consumes the style representation and the context sentences to compute a feature for to generate the styled text . In this section, we will first introduce the data streams employed during training and our novel learning objective function. We will then discuss various generator design choices.
4.1 Learning Data Streams
Let be a dataset of documents where is a document and is its style label. We assume a finite set of style labels where each integer represents a style class such as news, review, lyric, poem, novel, and children book. During training, our framework employs two data streams where the first one is called the reconstruction stream while the other is referred to as the cross-style generation stream. We note that such a two-stream processing pipeline is common in GAN-based image translation frameworks (Liu et al. (2017); Huang et al. (2018); Liu et al. (2019a)) but is less explored for language modeling.
Reconstruction stream (RS). For this steam, we first sample two documents with the same style from : and where . We then sample two paragraphs111For the purpose of data augmentation, in our implementation, a paragraph we sample may not be the full paragraph in the nominal sense. It could starting from the middle of a nominal paragraph.: and . We extract the first few sentences from as the input context , where is the extraction function, and use for the style reference . Feeding and to the generator , we expect should be able to reconstruct : .
Cross-style generation stream (CS). We first sample two documents and where . We then sample paragraphs and . We again extract the first few sentences from as the input context and use for the style reference . As feeding and to the generator , we expect should output where should has the same style as . Let be an oracle style comparator function that outputs 1 if the two input texts have the same style and 0 otherwise. We aim for .
4.2 Learning Objective
We propose an objective function consisting of four carefully designed loss terms for training the proposed framework using the above two data streams. The objective function is given by
where is the language modeling loss, the distillation loss, is a style comparison loss, and is the latent GAN loss. The scalars , , and
are the hyper-parameters controlling relative importance of the terms. The values for these hyperparameters and the method for determining their values are discussed in AppendixA. We visualizes training with the proposed objective function using the two data streams in Fig. 1.
Language modeling loss
formulates the probability distribution of a paragraphas the product of the conditional probability of each token given the previous tokens as shown in (1). We use to supervise the training of the data reconstruction stream. It is given by
where denotes that and are from the reconstruction stream. The variable is the total number of tokens in and is the size of the vocabulary.
Distillation loss. We use to regularize the learning as processing the data reconstruction steam. We pretrain a GPT-2 model using our dataset and use it as our distillation target. We denote the pretrained GPT-2 model as . (Note that does not have the desired style control capability.) By jointly optimizing and , we train to generate fluent texts (by minimizing ) as well as behave similarly to (by minimizing ). The distillation loss is calculated by minimizing the mutual information between output distributions of and , which is given by
We note that the distillation loss has been used in various tasks including model compression, transfer learning, life-long learning, etc (Hinton et al. (2015); Kim and Rush (2016); Liu et al. (2019c); Mirzadeh et al. (2019); Liu et al. (2019b); Hou et al. (2018)). In this paper, we extend its use to the style example-guided language generative model training task.
Style loss helps ensure the output from the cross-style generation stream has the same style as the input reference. A pretrained style comparator is used for computing the loss. The comparator takes two paragraphs as input and is trained to output 1 when the two paragraphs have the same style and 0 otherwise. We use for pretraining since it contains style labels for each document. We pretrain using the binary cross entropy loss. The comparator is highly accurate. It achieves a classification accuracy of 87.8% to 98.8% in our held-out validation sets. After pretaining, we fix and use it to train . The style loss is then given by
where denotes the pair is sampled from the cross-style generation stream.
Here, we would like to make two remarks. First, since takes the latent feature from as input, we avoid the non-differentiability of the text decoding mechanism and can directly train . Second, despite that is pretrained using feature extracted from , we use the feature extracted from as input. We can perform this operation not only because these two features have the same dimension but also because we enforce them to have a similar distribution via optimizing the GAN loss, discussed below.
GAN loss is used to match the distributions of the features generated by and those generated by , respectively, as processing the cross-style generation stream. We use a latent GAN formulation where we train a GAN discriminator to differentiate features extracted from to . The GAN loss is given by
We realize the discriminator using a GPT-2-based transformer network.
4.3 Generator Design
We realize the style encoder using a GPT-2-based transformer identical to . After extracting a representation for each token in , we utilize a 3-layer position-wise fully-connected network to obtain the final style code as illustrated in Fig. 2. The text decoder is also a GPT-2-based transformer identical to . We initialize the weights in and using the weights in the pretrained . Next, we compare four different ways of injecting outputs from into , which represent different inductive biases and result in difference performances.
Model A: style code as a bias to the input. In this model, the style code is directly summed up with the token-embedding and position embedding before inputting to the first transformer module in . In other words, the input to the first transformer module in is where denotes as the th word embedding, and denotes as the th position embedding.
Model B: style code as a summarization token. In this model, the computed style code is treated as a special token that is inserted to the beginning of the input sequence and is directed fed in the first transformer module in . That is the input sequence length becomes . This design is motivated by the traditional sequence-to-sequence modeling techniques (Chung et al. (2014); Cho et al. (2014); Sutskever et al. (2014); Bahdanau et al. (2016); Vinyals and Le (2015)).
Model C: style-aware self-attention. In this model, we input into each self-attention layer in to influence its computation given by where which denotes an affine transformation, and denotes the key and value embeddings from the th hidden layer, and denotes the hidden dimension.
), we utilize the style code to modulate the hidden representations within the text decoder via normalization layers. Specifically, we replace the scale and bias parameters in the affine transformation step of the layer normalization (Ba et al. (2016)) with a style code determined scale and bias. That is
where denotes the th hidden representation of the th token at the th transformer layer. We note since there are two layer normalization layers in each transformer in our implementation. The mean and deviation and are computed across the channel dimension.
Implementation. We set the latent dimension to , number of attention-heads to , number of transformer layers to , number of tokens in a paragraph to , and the vocabulary size to using BPE-encoding (Sennrich et al. (2015)) vocabulary from Radford et al. (2019) throughout out all the models and experiments. We use a pretrained GPT-2 model and a style comparator in our framework. The training details of these two models are given in Appendix B. All of the experiments are conducted using an NVIDIA DGX1 machine.
3-Style. The dataset consists of documents from the RealNews dataset (Zellers et al. (2019)), the BookCorpus dataset (Zhu et al. (2015)), and the Reviews dataset (Yelp (2019); McAuley and Leskovec (2013); Maas et al. (2011); Dataworld (2017); Liu (2017)). The 3 styles are news, book, and review. In detail, the news set has documents and words, the books set has documents and words, and the review set has documents and words after cleaning. The total dataset has documents and words. We hold out documents as the validation set and documents as the testing set.
. We build a dataset that contains 21 text styles. We first classify the documents in RealNews intostyles, including Sciences, Sport, Politics, Business, Technology, Entertainment, Opinion, Life, and News. Then, we divide the documents in BookCorpus into 8 different styles, which are Romance, Fantasy, Sciencefiction, Children’sbooks, Thriller, Adventure, Poetry, and Plays. We split the documents into multiple small documents by extracting the dialogues except for the Poetry and Plays. We divide the Review dataset into styles, namely Yelp, Hotel, and Movie. Finally, we crawl lyrics from http://www.azlyrics.com/. The total dataset has documents. We hold out documents as the validation set and documents as the testing set.
Auto-evaluation metrics. We evaluate different models using fluency score, style score, style diversity score, and content novelty score. The fluency score measures whether the output paragraph reads like a human-written one. The style score checks whether the output text carries the target style. Our framework supports multimodal outputs (Huang et al. (2018)
). For the same input context but different reference examples of the same style, our framework should produce different output texts but all with the same style. To measure how different these outputs are, we use the style diversity score. Finally, the content novelty score is used to measure the difference between the output and the reference example. A model that directly duplicates the reference to the output is undesirable. The details of these automatic evaluation metrics are available in AppendixC.
Human study settings. We use the Amazon Mechanical Turk (AMT) platform for user studies. We conduct two studies where one evaluates fluency of the generated paragraphs while the other verifies the style correctness. For the fluency study, we present a human-written text and a machine-generated text in random order and ask the worker to choose which one is written by a human. For this metric, the closer the preference score to , the better the performance.
For the style study, we perform two tests. In one test, we present a worker a generated paragraph that supposes to be in the target style. We also give the worker two human-written reference paragraphs where one is with the target style while the other is not. We then ask the worker to choose which reference paragraph has a style more similar to the generated one. In the other test, we again present a worker a generated paragraph but this time with the style categorical labels to choose from instead of the reference paragraphs. We compute the frequency that the worker selects the right style. The higher the score, the better the performance. More details are in Appendix D.
Strong baselines. We compare our framework to three strong baselines, namely the general GPT-2 model (G-GPT-2), a baseline consists of multiple style-specialized GPT-2 models (S-GPT-2), and the style-code encoding (SC) method based on the description in Keskar et al. (2019). G-GPT-2 is trained on the entire dataset using . It does not allow style control but can generate fluent texts. In S-GPT-2, we train a GPT-2 model per style. As training a GPT-2 model is costly, we only use this baseline for the 3-Style dataset evaluation. In SC
, an one-hot encoding of the style class label is used as a special token for style-controllable paragraph generation. Unlike the proposed method that extracts the style code from the input paragraph,SC input the style label. The rest of the model is similar to our Model B without the style encoder.
In Fig. 3, we plot the fluency and style scores achieved by our models as well as those by the baselines on the -Style and -Style datasets. The closer the model to the top-right corner, the more superior the model is. From the figure, we found that among our models, Model D performs the best. As expected, G-GPT-2 achieves the best fluency score. However, since it does not support style control, it has a poor style score. On the other hand, S-GPT-2 achieves good fluency and style scores for the 3-Style dataset. This is understandable as it utilizes a GPT-2 model for each style. However, such an approach does not scale well as GPT-2 training is expensive. We also found that SC does not achieve good style score and is inferior to our models. We suspect this is because the one-hot style class code is largely ignored during inference. Since Model D performs the best in our framework, for the rest of the paper, we use it as our representative model for performance comparison as well as ablation study.
In Tab. 2, we show the style diversity scores achieved by our models. We found that all of our 4 models can generate diverse styled paragraphs conditioning on the same context and different reference examples with the same style.
(%) Model D SC Random 3-Style 56 54 50 21-Style 57 63 50
(%) Model D SC Random 3-Style by reference 56 52 50 3-Style by category 65 54 50 21-Style by reference 66 49 50 21-Style by category 69 50 50
Human evaluation. In Tab. 3, we report user study results on fluency and style control. We found that our model achieves great fluency on both of the datasets. Compared to SC, our model performs better in controlling the style in the output texts.
Ablation study. We conduct an ablation study on the loss terms in the proposed objective function and report the results in Tab. 4 using the -Style dataset. The results show that each term is important. Removing leads to a degraded content novelty score. Removing leads to a degraded style score, thought an improved fluency score and a content novelty score. Removing leads to both degraded fluency and style diversity scores.
We presented a language generative framework for style example-guided paragraph generation. To the best of our knowledge, we were the first to achieve such style-controllability on paragraph generation. We attributed the success to our carefully designed learning objective function, the generator network, and the newly composed large-scale dataset consisting of documents of various text styles.
- Learning representations and generative models for 3d point clouds. arxiv preprint arXiv:1707.02392. Cited by: §3.
- Layer normalization. arXiv preprint arXiv:1607.06450. Cited by: §4.3.
- Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), Cited by: §4.3.
- Real or fake? learning to discriminate machine from human generated text. arxiv preprint arXiv:1906.03351. Cited by: §2.
- A neural probabilistic language model. Journal of Machine Learning Research (JMLR). Cited by: §2, §3.
- Hello, it’s gpt-2–how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. arxiv preprint arXiv:1907.05774. Cited by: §2.
- KERMIT: generative insertion-based modeling for sequences. arxiv preprint arXiv:1906.01604. Cited by: §2.
- Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §4.3.
Empirical evaluation of gated recurrent neural networks on sequence modeling. arxiv preprint arXiv:1412.3555. Cited by: §3, §4.3.
- Training language gans from scratch. In Advances in Neural Information Processing Systems (NIPS), Cited by: §2.
- Transformer-xl: attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2.
- Hotel reviews dataset. In https://data.world/datafiniti/hotel-reviews, Cited by: §5.
- Bert: pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: §2, §3.
- A learned representation for artistic style. In International Conference on Learning Representations (ICLR), Cited by: §2.
- Style transfer in text: exploration and evaluation. In Association for the Advancement of Artificial Intelligence (AAAI), Cited by: §2.
Image style transfer using convolutional neural networks. In
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.
- Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), Cited by: §3.
- Characterizing and improving stability in neural style transfer. In IEEE International Conference on Computer Vision (ICCV), Cited by: §2.
- Distilling the knowledge in a neural network. arxiv preprint arXiv:1503.02531. Cited by: §4.2.
- Long short-term memory. Neural computation. Cited by: §3.
- Lifelong learning via progressive distillation and retrospection. In European Conference on Computer Vision (ECCV), Cited by: §4.2.
- Toward controlled generation of text. In International Conference on Machine Learning (ICML), Cited by: §2, §2.
- Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference on Computer Vision (ICCV), Cited by: §2.
Multimodal unsupervised image-to-image translation. European Conference on Computer Vision (ECCV). Cited by: §4.1, §5.
Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (ECCV), Cited by: §2.
- A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4.3.
- Generation of informative texts with style. In International Conference on Computational Linguistics (COLING), Cited by: §2.
- CTRL: a conditional transformer language model for controllable generation. In Einstein.ai, Cited by: §2, §5.
- Learning to discover cross-domain relations with generative adversarial networks. In International Conference on Machine Learning (ICML), Cited by: §2.
- Sequence-level knowledge distillation. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §4.2.
- Neural text generation from structured data with application to the biography domain. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.
Guiding generation for abstractive text summarization based on key information guide network. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: §2.
- Domain adaptive text style transfer. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.
- Universal style transfer via feature transforms. In Advances in Neural Information Processing Systems (NIPS), pp. 385–395. Cited by: §2.
- 515K hotel reviews data in europe. In https://www.kaggle.com/jiashenliu/515k-hotel-reviews-data-in-europe, Cited by: §5.
- Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems (NIPS), Cited by: §4.1.
- Few-shot unsupervised image-to-image translation. In IEEE International Conference on Computer Vision (ICCV), Cited by: §4.1.
- Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations (ICLR), Cited by: §2.
- Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arxiv preprint arXiv:1904.09482. Cited by: §4.2.
- Multi-task deep neural networks for natural language understanding. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §4.2.
- Roberta: a robustly optimized bert pretraining approach. arxiv preprint arXiv:1907.11692. Cited by: §2, §3.
Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §5.
- From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In International World Wide Web Conference (WWW), Cited by: §5.
- Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), Cited by: §2.
- Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NIPS), Cited by: §2.
- Improved knowledge distillation via teacher assistant: bridging the gap between student and teacher. arxiv preprint arXiv:1902.03393. Cited by: §4.2.
- Semantic image synthesis with spatially adaptive normalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4.3.
- Towards improving abstractive summarization via entailment generation. In Annual Meeting of the Association for Computational Linguistics (ACL) Workshop, Cited by: §2.
- Glove: global vectors for word representation. Vol. 14, pp. 1532–1543. External Links: Cited by: §2.
- Deep contextualized word representations. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2.
- Style transfer through back-translation. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2.
- Learning to generate reviews and discovering sentiment. In International Conference on Learning Representations (ICLR), Cited by: §2.
- Language models are unsupervised multitask learners. In OpenAI Blog, Cited by: §2, §3, §5.
- Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.
- Dear sir or madam, may i introduce the yafc corpus: corpus, benchmarks and metrics for formality style transfer. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: §2.
- Wikipedia xml data. In Amazon, Cited by: §5.
- Get to the point: summarization with pointer-generator networks. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §5.
- On accurate evaluation of gans for language generation. arxiv preprint arXiv:1806.04936. Cited by: Appendix A.
- Neural machine translation of rare words with subword units. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §5.
- Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems (NIPS), pp. 6833–6844. Cited by: §2.
- Megatron-lm: training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053. Cited by: §2, §3.
- Insertion transformer: flexible sequence generation via insertion operations. In International Conference on Machine Learning (ICML), Cited by: §2.
- Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), Cited by: §4.3.
- Discrete flows: invertible generative models of discrete data. In International Conference on Learning Representations (ICLR) Workshop, Cited by: §2.
- Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), Cited by: §2, §3.
- A neural conversational model. arxiv preprint arXiv:1506.05869. Cited by: §2, §4.3.
- Glue: a multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), Cited by: §2.
- SentiGAN: generating sentimental texts via mixture adversarial networks.. In International Joint Conferences on Artificial Intelligence (IJCAI), Cited by: §2.
- Unsupervised controllable text generation with global variation discovery and disentanglement. arxiv preprint arXiv:1905.11975. Cited by: §2.
- Paraphrasing for style. International Conference on Computational Linguistics (COLING). Cited by: §2.
- From shakespeare to twitter: what are language styles all about?. In Empirical Methods in Natural Language Processing (EMNLP) Workshop, Cited by: §2.
- XLNet: generalized autoregressive pretraining for language understanding. arxiv preprint arXiv:1906.08237. Cited by: §2.
- Yelp dataset challenge. In https://www.yelp.com/dataset/challenge, Cited by: §5.
- Seqgan: sequence generative adversarial nets with policy gradient. In Association for the Advancement of Artificial Intelligence (AAAI), Cited by: §2.
- Towards automatic generation of product reviews from aspect-sentiment scores. In International Conference on Natural Language Generation (INLG), Cited by: §2.
- SWAG: a large-scale adversarial dataset for grounded commonsense inference. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.
- Defending against neural fake news. In Advances in Neural Information Processing Systems (NIPS), Cited by: §2, §5, §5.
- Pretraining-based natural language generation for text summarization. arxiv preprint arXiv:1902.09243. Cited by: §2.
The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix C, Appendix C.
- Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV), Cited by: §2.
- Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In IEEE International Conference on Computer Vision (ICCV), Cited by: §5, §5.
Appendix A Hyper-parameters Tunning
We tune the hyper-parameters on a pilot-test dataset. This pilot-test dataset has training examples and hold-out examples. We perform a grid search on log-scale. We utilize the Frechet Embedding Distance (Semeniuta et al. (2018)) to select best hyper-parameters. For , , and , we try , , and . For betas used in Adam Optimizer, we try and . For weight decay, we try and . For the initial learning rate, we try and . Eventually, we use as initial learning rate and for , , as well as for all models except for the Model B. We use for the Model B instead.
Appendix B Pretraining of and
Pretraining of the GPT-2 model . We pretrain on our collected dataset from scratch. We use Adam optimizer with initial learning rate, and are set by , cosine learning rate decay style, and weight decay. The batch size is set to . The total training iterations is which takes weeks.
Pretraining of the style comparator . We pretrain the Style Comparator using and batch size, respectively. The initial learning rate is with weight decay and cosine learning rate decay style. The optimizer is also Adam with and . Since the training converges very quickly, we do early stopping if the accuracy on validation set plateaus. Eventually, we use the checkpoint at and iterations to train on the -style and -style datasets, respectively. The accuracy on hold-out validation set are and on -styles and -styles datasets, respectively.
Appendix C Auto-evaluation Metrics
Fluency score. To ensure the generated paragraph is fluent and coherent, we utilize a pretrained GPT-2 model to measure the perplexity of the generated paragraph. We compute the generation likelihood over each token using the model and treat the generated paragraph as inputs and as labels. Given the input text, the pretrained GPT-2 returns the probability distribution of next token over the vocabulary. Then, we measure the perplexity by this probability distribution and label. Since our dictionary size is , the random guess of the next token would result in perplexity (). Thus, we set as an upper bound and define the fluency score of the generated paragraph as . In this sense, a higher fluency score means lower perplexity.
Style score. We train 3/21 binary style classifiers (since we have 3/21 different styles in the 3-Style/21-Style dataset) by finetuning a GPT-2 network to automatically evaluate whether the generated text carries the style of a target class. These 3/21 classifiers achieve average classification accuracies of /. During the testing phase, for a target style, if the corresponding style classifier correctly predicts 1 for the generated paragraph computed by a model, we count it as a successful trial. We compute the success rate over the test set and use the result as the style score for the model.
Style diversity score. We adopt the LPIPS distance (Zhang et al. (2018)) to measure the diversity of the generation outputs conditioning on the same context. To implement this metric, we first extract the feature representation from each token in a generated paragraph by a pretrained GPT-2 model. We compute the mean representation of the tokens in a paragraph as the paragraph-level representation. Then, we measure the distance between two paragraph-level representations of two different paragraphs generated using the same context but two different references written in the same style. In this sense, a larger distance value implies the styles of the two generated paragraphs are more different.
To get an idea of the range of this metric, we compute an upper bound and a lower bound. We consider two paragraphs from two documents of different styles should have a high style diversity score. We hence sample paragraphs from each style and use the pretrained GPT-2 model to extract deep features. After taking average over the token-dimension, we obtain by representation for each style. Then, we compute the distance between of these matrices divided by 1000. This gives us a matrix of size measuring the pairwise distance between two styles. We use the largest value in this matrix as our upper bound, which is .
For the lower bound, since two different paragraphs from the same document should have a low style diversity score, we use their scores for the lower bound computation. Specifically, we compute the average distance between two different paragraphs from the same document. We do this for each style and obtain 21 different values. We obtain the lower bound by taking average over these values, which is .
Content novelty score. To verify that our model is not simply duplicating the content from reference paragraph, we utilize LPIPS distance (Zhang et al. (2018)) to measure the difference between the generated paragraph and the input reference paragraph. We again use a pretrained GPT-2 model for extracting a feature representation for each token. To compute the distance between two paragraphs, we compute the bipartite matching cost between the tokens from the two paragraphs. Specifically, we first compute the distances between any token representation in one paragraph to all the token representations in the other paragraph. We then compute the minimum cost assignment by solving a bipartite matching problem. In order to get an idea about the range of the content novelty score, we compute an upper bound and a lower bound using a similar approach as the one used for the style diversity score. We find the upper bound value is and the lower bound value is .
Appendix D Human Evaluation
To participate in our user study, a worker has to be awarded the Master Qualification by AMT and has at least life-long HIT approval rate. We generate paragraphs for a context with different reference paragraphs from the same style. For -style dataset, we randomly sample examples from each style and do style generation for all styles. Thus, there are examples for each testing model. For -style dataset, we randomly sample examples from testing set and do style generation on all styles. Thus, there are examples for each testing model. There is a typical example for experiment of fluency human study in Fig. 4.