The Case for a Single Model that can Both Generate Continuations and Fill in the Blank

06/09/2022
by   Daphne Ippolito, et al.
0

The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do the fill-in-the-blank task, a more useful model is one that can effectively perform _both_ FitB and continuation. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how FitB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/08/2020

Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning

Fine-tuning pre-trained generative language models to down-stream langua...
research
05/18/2023

Discriminative Diffusion Models as Few-shot Vision and Language Learners

Diffusion models, such as Stable Diffusion, have shown incredible perfor...
research
02/08/2020

Blank Language Models

We propose Blank Language Model (BLM), a model that generates sequences ...
research
08/23/2023

Audio Generation with Multiple Conditional Diffusion Model

Text-based audio generation models have limitations as they cannot encom...
research
06/15/2018

Discovering User Groups for Natural Language Generation

We present a model which predicts how individual users of a dialog syste...
research
05/31/2023

Fine-grained Text Style Transfer with Diffusion-Based Language Models

Diffusion probabilistic models have shown great success in generating hi...
research
02/06/2020

Introducing Aspects of Creativity in Automatic Poetry Generation

Poetry Generation involves teaching systems to automatically generate te...

Please sign up or login with your details

Forgot password? Click here to reset