Best-k Search Algorithm for Neural Text Generation

11/22/2022
by   Jiacheng Xu, et al.
0

Modern natural language generation paradigms require a good decoding strategy to obtain quality sequences out of the model. Beam search yields high-quality but low diversity outputs; stochastic approaches suffer from high variance and sometimes low quality, but the outputs tend to be more natural and creative. In this work, we propose a deterministic search algorithm balancing both quality and diversity. We first investigate the vanilla best-first search (BFS) algorithm and then propose the Best-k Search algorithm. Inspired by BFS, we greedily expand the top k nodes, instead of only the first node, to boost efficiency and diversity. Upweighting recently discovered nodes accompanied by heap pruning ensures the completeness of the search procedure. Experiments on four NLG tasks, including question generation, commonsense generation, text summarization, and translation, show that best-k search yields more diverse and natural outputs compared to strong baselines, while our approach maintains high text quality. The proposed algorithm is parameter-free, lightweight, efficient, and easy to use.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/14/2021

Massive-scale Decoding for Text Generation using Lattices

Neural text generation models like those used for summarization and tran...
research
11/14/2022

Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding

In open-ended natural-language generation, existing text decoding method...
research
12/06/2022

Improved Beam Search for Hallucination Mitigation in Abstractive Summarization

Advancement in large pretrained language models has significantly improv...
research
07/09/2020

Unsupervised Text Generation by Learning from Search

In this work, we present TGLS, a novel framework to unsupervised Text Ge...
research
10/25/2022

Information Filter upon Diversity-Improved Decoding for Diversity-Faithfulness Tradeoff in NLG

Some Natural Language Generation (NLG) tasks require both faithfulness a...
research
08/30/2019

Parameter-free Structural Diversity Search

The problem of structural diversity search is to nd the top-k vertices w...
research
03/28/2022

A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation

We propose Composition Sampling, a simple but effective method to genera...

Please sign up or login with your details

Forgot password? Click here to reset