EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start

05/24/2022
by   Jonathan Mallinson, et al.
0

We present EdiT5 - a novel semi-autoregressive text-editing approach designed to combine the strengths of non-autoregressive text-editing and autoregressive decoding. EdiT5 is faster at inference times than conventional sequence-to-sequence (seq2seq) models, while being capable of modeling flexible input-output transformations. This is achieved by decomposing the generation process into three sub-tasks: (1) tagging to decide on the subset of input tokens to be preserved in the output, (2) re-ordering to define their order in the output text, and (3) insertion to infill the missing tokens that are not present in the input. The tagging and re-ordering steps, which are responsible for generating the largest portion of the output, are non-autoregressive, while the insertion uses an autoregressive decoder. Depending on the task, EdiT5 requires significantly fewer autoregressive steps demonstrating speedups of up to 25x when compared to classic seq2seq models. Quality-wise, EdiT5 is initialized with a pre-trained T5 checkpoint yielding comparable performance to T5 in high-resource settings and clearly outperforms it on low-resource settings when evaluated on three NLG tasks: Sentence Fusion, Grammatical Error Correction, and Decontextualization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/24/2020

Felix: Flexible Text Editing Through Tagging and Insertion

We present Felix — a flexible text-editing approach for generation, desi...
research
09/19/2023

RedPenNet for Grammatical Error Correction: Outputs to Tokens, Attentions to Spans

The text editing tasks, including sentence fusion, sentence splitting an...
research
03/17/2022

An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models

We propose a framework for training non-autoregressive sequence-to-seque...
research
10/21/2022

Text Editing as Imitation Game

Text editing, such as grammatical error correction, arises naturally fro...
research
02/20/2020

Imputer: Sequence Modelling via Imputation and Dynamic Programming

This paper presents the Imputer, a neural sequence model that generates ...
research
05/24/2023

Let There Be Order: Rethinking Ordering in Autoregressive Graph Generation

Conditional graph generation tasks involve training a model to generate ...
research
07/25/2020

NoPropaganda at SemEval-2020 Task 11: A Borrowed Approach to Sequence Tagging and Text Classification

This paper describes our contribution to SemEval-2020 Task 11: Detection...

Please sign up or login with your details

Forgot password? Click here to reset