A Survey of Pretrained Language Models Based Text Generation

01/14/2022
by   Junyi Li, et al.
158

Text Generation aims to produce plausible and readable text in human language from input data. The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). Grounding text generation on PLMs is seen as a promising direction in both academia and industry. In this survey, we present the recent advances achieved in the topic of PLMs for text generation. In detail, we begin with introducing three key points of applying PLMs to text generation: 1) how to encode the input data as representations preserving input semantics which can be fused into PLMs; 2) how to design a universal and performant architecture of PLMs served as generation models; and 3) how to optimize PLMs given the reference text and ensure the generated text satisfying special text properties. Then, we figure out several challenges and future directions within each key point. Next, we present a summary of various useful resources and typical text generation applications to work with PLMs. Finally, we conclude and summarize the contribution of this survey.

READ FULL TEXT
research
05/21/2021

Pretrained Language Models for Text Generation: A Survey

Text generation has become one of the most important yet challenging tas...
research
12/06/2021

Search and Learn: Improving Semantic Coverage for Data-to-Text Generation

Data-to-text generation systems aim to generate text descriptions based ...
research
04/11/2022

Uniform Complexity for Text Generation

Powerful language models such as GPT-2 have shown promising results in t...
research
03/23/2023

DreamBooth3D: Subject-Driven Text-to-3D Generation

We present DreamBooth3D, an approach to personalize text-to-3D generativ...
research
07/16/2021

Urdu Hindi Poetry Generation using Neural Networks

One of the major problems writers and poets face is the writer's block. ...
research
11/21/2020

Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference

A major challenge in evaluating data-to-text (D2T) generation is measuri...
research
07/07/2021

On Training Instance Selection for Few-Shot Neural Text Generation

Large-scale pretrained language models have led to dramatic improvements...

Please sign up or login with your details

Forgot password? Click here to reset