Learning Sparse Prototypes for Text Generation

06/29/2020
by   Junxian He, et al.
0

Prototype-driven text generation uses non-parametric models that first choose from a library of sentence "prototypes" and then modify the prototype to generate the output text. While effective, these methods are inefficient at test time as a result of needing to store and index the entire training corpus. Further, existing methods often require heuristics to identify which prototypes to reference at training time. In this paper, we propose a novel generative model that automatically learns a sparse prototype support set that, nonetheless, achieves strong language modeling performance. This is achieved by (1) imposing a sparsity-inducing prior on the prototype selection distribution, and (2) utilizing amortized variational inference to learn a prototype retrieval function. In experiments, our model outperforms previous prototype-driven language models while achieving up to a 1000x memory reduction, as well as a 1000x speed-up at test time. More interestingly, we show that the learned prototypes are able to capture semantics and syntax at different granularity as we vary the sparsity of prototype selection, and that certain sentence attributes can be controlled by specifying the prototype for generation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/09/2022

Factuality Enhanced Language Models for Open-Ended Text Generation

Pretrained language models (LMs) are susceptible to generate text with n...
research
04/13/2021

From Solving a Problem Boldly to Cutting the Gordian Knot: Idiomatic Text Generation

We study a new application for text generation – idiomatic sentence gene...
research
04/05/2020

Syntax-driven Iterative Expansion Language Models for Controllable Text Generation

The dominant language modeling paradigms handle text as a sequence of di...
research
09/09/2021

Efficient Nearest Neighbor Language Models

Non-parametric neural language models (NLMs) learn predictive distributi...
research
06/06/2022

Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation

While large-scale neural language models, such as GPT2 and BART, have ac...
research
09/09/2021

Graphine: A Dataset for Graph-aware Terminology Definition Generation

Precisely defining the terminology is the first step in scientific commu...
research
10/01/2019

Generalization in Generation: A closer look at Exposure Bias

Exposure bias refers to the train-test discrepancy that seemingly arises...

Please sign up or login with your details

Forgot password? Click here to reset