F^2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax

09/20/2020
by   Byung-Ju Choi, et al.
0

Despite recent advances in neural text generation, encoding the rich diversity in human language remains elusive. We argue that the sub-optimal text generation is mainly attributable to the imbalanced token distribution, which particularly misdirects the learning model when trained with the maximum-likelihood objective. As a simple yet effective remedy, we propose two novel methods, F^2-Softmax and MefMax, for a balanced training even with the skewed frequency distribution. MefMax assigns tokens uniquely to frequency classes, trying to group tokens with similar frequencies and equalize frequency mass between the classes. F^2-Softmax then decomposes a probability distribution of the target token into a product of two conditional probabilities of (i) frequency class, and (ii) token from the target frequency class. Models learn more uniform probability distributions because they are confined to subsets of vocabularies. Significant performance gains on seven relevant metrics suggest the supremacy of our approach in improving not only the diversity but also the quality of generated texts.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/08/2021

Neural Text Generation with Part-of-Speech Guided Softmax

Neural text generation models are likely to suffer from the low-diversit...
research
10/15/2020

Diverse Keyphrase Generation with Neural Unlikelihood Training

In this paper, we study sequence-to-sequence (S2S) keyphrase generation ...
research
11/12/2021

Speeding Up Entmax

Softmax is the de facto standard in modern neural networks for language ...
research
05/06/2020

Token Manipulation Generative Adversarial Network for Text Generation

MaskGAN opens the query for the conditional language model by filling in...
research
07/18/2023

Text vectorization via transformer-based language models and n-gram perplexities

As the probability (and thus perplexity) of a text is calculated based o...
research
04/08/2022

Enhance Incomplete Utterance Restoration by Joint Learning Token Extraction and Text Generation

This paper introduces a model for incomplete utterance restoration (IUR)...
research
07/03/2020

On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generation

The goal of text generation models is to fit the underlying real probabi...

Please sign up or login with your details

Forgot password? Click here to reset