Changing the Mind of Transformers for Topically-Controllable Language Generation

03/29/2021
by   Haw-Shiuan Chang, et al.
13

Large Transformer-based language models can aid human authors by suggesting plausible continuations of text written so far. However, current interactive writing assistants do not allow authors to guide text generation in desired topical directions. To address this limitation, we design a framework that displays multiple candidate upcoming topics, of which a user can select a subset to guide the generation. Our framework consists of two components: (1) a method that produces a set of candidate topics by predicting the centers of word clusters in the possible continuations, and (2) a text generation model whose output adheres to the chosen topics. The training of both components is self-supervised, using only unlabeled text. Our experiments demonstrate that our topic options are better than those of standard clustering approaches, and our framework often generates fluent sentences related to the chosen topics, as judged by automated metrics and crowdsourced workers.

READ FULL TEXT

page 3

page 4

research
11/14/2020

Conditioned Natural Language Generation using only Unconditioned Language Model: An Exploration

Transformer-based language models have shown to be very powerful for nat...
research
01/02/2021

The Truth is Out There: Investigating Conspiracy Theories in Text Generation

With the growing adoption of text generation models in today's society, ...
research
04/05/2020

Syntax-driven Iterative Expansion Language Models for Controllable Text Generation

The dominant language modeling paradigms handle text as a sequence of di...
research
06/05/2020

CoCon: A Self-Supervised Approach for Controlled Text Generation

Pretrained Transformer-based language models (LMs) display remarkable na...
research
06/18/2022

Collocation2Text: Controllable Text Generation from Guide Phrases in Russian

Large pre-trained language models are capable of generating varied and f...
research
03/11/2021

Topical Language Generation using Transformers

Large-scale transformer-based language models (LMs) demonstrate impressi...
research
07/08/2023

Advancements in Scientific Controllable Text Generation Methods

The previous work on controllable text generation is organized using a n...

Please sign up or login with your details

Forgot password? Click here to reset