Log In Sign Up

If beam search is the answer, what was the question?

by   Clara Meister, et al.

Quite surprisingly, exact maximum a posteriori (MAP) decoding of neural language generators frequently leads to low-quality results. Rather, most state-of-the-art results on language generation tasks are attained using beam search despite its overwhelmingly high search error rate. This implies that the MAP objective alone does not express the properties we desire in text, which merits the question: if beam search is the answer, what was the question? We frame beam search as the exact solution to a different decoding objective in order to gain insights into why high probability under a model alone may not indicate adequacy. We find that beam search enforces uniform information density in text, a property motivated by cognitive science. We suggest a set of decoding objectives that explicitly enforce this property and find that exact decoding with these objectives alleviates the problems encountered when decoding poorly calibrated language generation models. Additionally, we analyze the text produced using various decoding strategies and see that, in our neural machine translation experiments, the extent to which this property is adhered to strongly correlates with BLEU.


page 1

page 2

page 3

page 4


On Decoding Strategies for Neural Text Generators

When generating text from probabilistic models, the chosen decoding stra...

Rethinking the Evaluation of Neural Machine Translation

The evaluation of neural machine translation systems is usually built up...

Calibrating Sequence likelihood Improves Conditional Language Generation

Conditional language models are predominantly trained with maximum likel...

Sampling-Based Minimum Bayes Risk Decoding for Neural Machine Translation

In neural machine translation (NMT), we search for the mode of the model...

Enabling arbitrary translation objectives with Adaptive Tree Search

We introduce an adaptive tree search algorithm, that can find high-scori...

A Streaming Approach For Efficient Batched Beam Search

We propose an efficient batching strategy for variable-length decoding o...

Determinantal Beam Search

Beam search is a go-to strategy for decoding neural sequence models. The...