Log In Sign Up

Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models

by   Luke Vilnis, et al.

Decoding methods for large language models often trade-off between diversity of outputs and parallelism of computation. Methods such as beam search and Gumbel top-k sampling can guarantee a different output for each element of the beam, but are not easy to parallelize. Alternatively, methods such as temperature sampling and its modifications (top-k sampling, nucleus sampling, typical decoding, and others), are embarrassingly parallel, but have no guarantees about duplicate samples. We present a framework for sampling according to an arithmetic code book implicitly defined by a large language model, compatible with common sampling variations, with provable beam diversity under certain conditions, as well as being embarrassingly parallel and providing unbiased and consistent expectations from the original model. We demonstrate the effectiveness of our approach on WMT machine translation, showing substantially reduced variance when estimating expected BLEU score and up to 1 point increased BLEU in oracle experiments.


Comparison of Diverse Decoding Methods from Conditional Language Models

While conditional language models have greatly improved in their ability...

Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding

In open-ended natural-language generation, existing text decoding method...

Machine Translation Decoding beyond Beam Search

Beam search is the go-to method for decoding auto-regressive machine tra...

Putting Machine Translation in Context with the Noisy Channel Model

We show that Bayes' rule provides a compelling mechanism for controlling...

Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement

The well-known Gumbel-Max trick for sampling from a categorical distribu...

Consistency of a Recurrent Language Model With Respect to Incomplete Decoding

Despite strong performance on a variety of tasks, neural sequence models...

A Simple Baseline for Beam Search Reranking

Reranking methods in machine translation aim to close the gap between co...