Insertion-based Decoding with Automatically Inferred Generation Order

02/04/2019
by   Jiatao Gu, et al.
0

Conventional neural autoregressive decoding commonly assumes a left-to-right generation order. In this work, we propose a novel decoding algorithm -- INDIGO -- which supports flexible generation in an arbitrary order with the help of insertion operations. We use Transformer, a state-of-the-art sequence generation model, to efficiently implement the proposed approach, enabling it to be trained with either a pre-defined generation order or an adaptive order searched based on the model's own preference. Experiments on three real-world tasks, including machine translation, word order recovery and code generation, demonstrate that our algorithm can generate sequences in an arbitrary order, while achieving competitive or even better performance compared to the conventional left-to-right generation. Case studies show that INDIGO adopts adaptive generation orders based on input information.

READ FULL TEXT
research
02/08/2019

Insertion Transformer: Flexible Sequence Generation via Insertion Operations

We present the Insertion Transformer, an iterative, partially autoregres...
research
02/05/2019

Non-Monotonic Sequential Text Generation

Standard sequential generation methods assume a pre-specified generation...
research
11/01/2019

Sequence Modeling with Unconstrained Generation Order

The dominant approach to sequence generation is to produce a sequence in...
research
01/15/2020

Parallel Machine Translation with Disentangled Context Transformer

State-of-the-art neural machine translation models generate a translatio...
research
10/29/2019

An Empirical Study of Generation Order for Machine Translation

In this work, we present an empirical study of generation order for mach...
research
12/16/2021

Learning and Analyzing Generation Order for Undirected Sequence Models

Undirected neural sequence models have achieved performance competitive ...
research
08/28/2019

Solving Math Word Problems with Double-Decoder Transformer

This paper proposes a Transformer-based model to generate equations for ...

Please sign up or login with your details

Forgot password? Click here to reset