Consistency of a Recurrent Language Model With Respect to Incomplete Decoding

02/06/2020
by   Sean Welleck, et al.
7

Despite strong performance on a variety of tasks, neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. We study the related issue of receiving infinite-length sequences from a recurrent language model when using common decoding algorithms. To analyze this issue, we first define inconsistency of a decoding algorithm, meaning that the algorithm can yield an infinite-length sequence that has zero probability under the model. We prove that commonly used incomplete decoding algorithms - greedy search, beam search, top-k sampling, and nucleus sampling - are inconsistent, despite the fact that recurrent language models are trained to produce sequences of finite length. Based on these insights, we propose two remedies which address inconsistency: consistent variants of top-k and nucleus sampling, and a self-terminating recurrent language model. Empirical results show that inconsistency occurs in practice, and that the proposed methods prevent inconsistency.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/03/2022

A Non-monotonic Self-terminating Language Model

Recent large-scale neural autoregressive sequence models have shown impr...
research
09/30/2022

Calibrating Sequence likelihood Improves Conditional Language Generation

Conditional language models are predominantly trained with maximum likel...
research
05/12/2016

Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model

Recent advances in conditional recurrent language modelling have mainly ...
research
10/07/2021

Back from the future: bidirectional CTC decoding using future information in speech recognition

In this paper, we propose a simple but effective method to decode the ou...
research
05/19/2022

RankGen: Improving Text Generation with Large Ranking Models

Given an input sequence (or prefix), modern language models often assign...
research
09/02/2019

Subword Language Model for Query Auto-Completion

Current neural query auto-completion (QAC) systems rely on character-lev...
research
10/13/2022

Language Model Decoding as Likelihood-Utility Alignment

A critical component of a successful language generation pipeline is the...

Please sign up or login with your details

Forgot password? Click here to reset