DeepAI AI Chat
Log In Sign Up

Generative Pre-training for Paraphrase Generation by Representing and Predicting Spans in Exemplars

by   Tien-Cuong Bui, et al.

Paraphrase generation is a long-standing problem and serves an essential role in many natural language processing problems. Despite some encouraging results, recent methods either confront the problem of favoring generic utterance or need to retrain the model from scratch for each new dataset. This paper presents a novel approach to paraphrasing sentences, extended from the GPT-2 model. We develop a template masking technique, named first-order masking, to masked out irrelevant words in exemplars utilizing POS taggers. So that, the paraphrasing task is changed to predicting spans in masked templates. Our proposed approach outperforms competitive baselines, especially in the semantic preservation aspect. To prevent the model from being biased towards a given template, we introduce a technique, referred to as second-order masking, which utilizes Bernoulli distribution to control the visibility of the first-order-masked template's tokens. Moreover, this technique allows the model to provide various paraphrased sentences in testing by adjusting the second-order-masking level. For scale-up objectives, we compare the performance of two alternatives template-selection methods, which shows that they were equivalent in preserving semantic information.


page 1

page 2

page 3

page 4


Improving Aspect Sentiment Quad Prediction via Template-Order Data Augmentation

Recently, aspect sentiment quad prediction (ASQP) has become a popular t...

Few-Shot Natural Language Generation by Rewriting Templates

Virtual assistants such as Google Assistant, Alexa and Siri enable users...

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

Recently, pre-trained models have achieved state-of-the-art results in v...

Natural Language Generation Using Link Grammar for General Conversational Intelligence

Many current artificial general intelligence (AGI) and natural language ...

LabelPrompt: Effective Prompt-based Learning for Relation Classification

Recently, prompt-based learning has become a very popular solution in ma...

Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding

E-commerce query understanding is the process of inferring the shopping ...