Autoregressive Text Generation Beyond Feedback Loops

08/30/2019
by   Florian Schmidt, et al.
0

Autoregressive state transitions, where predictions are conditioned on past predictions, are the predominant choice for both deterministic and stochastic sequential models. However, autoregressive feedback exposes the evolution of the hidden state trajectory to potential biases from well-known train-test discrepancies. In this paper, we combine a latent state space model with a CRF observation model. We argue that such autoregressive observation models form an interesting middle ground that expresses local correlations on the word level but keeps the state evolution non-autoregressive. On unconditional sentence generation we show performance improvements compared to RNN and GAN baselines while avoiding some prototypical failure modes of autoregressive models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/12/2018

Deep State Space Models for Unconditional Word Generation

Autoregressive feedback is considered a necessity for successful uncondi...
research
11/25/2019

Non-autoregressive Transformer by Position Learning

Non-autoregressive models are promising on various text generation tasks...
research
02/16/2021

Non-Autoregressive Text Generation with Pre-trained Language Models

Non-autoregressive generation (NAG) has recently attracted great attenti...
research
04/15/2023

Tractable Control for Autoregressive Language Generation

Despite the success of autoregressive large language models in text gene...
research
04/20/2017

Fast Generation for Convolutional Autoregressive Models

Convolutional autoregressive models have recently demonstrated state-of-...
research
11/20/2019

Autoregressive Modeling of Forest Dynamics

In this work, we employ autoregressive models developed in financial eng...
research
05/06/2023

An Adversarial Non-Autoregressive Model for Text Generation with Incomplete Information

Non-autoregressive models have been widely studied in the Complete Infor...

Please sign up or login with your details

Forgot password? Click here to reset