Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction

05/03/2020
by   Masahiro Kaneko, et al.
0

This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine-tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of-the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/kanekomasahiro/bert-gec.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/04/2020

Chinese Grammatical Correction Using BERT-based Pre-trained Model

In recent years, pre-trained models have been extensively studied, and s...
research
01/10/2020

Towards Minimal Supervision BERT-based Grammar Error Correction

Current grammatical error correction (GEC) models typically consider the...
research
05/26/2020

GECToR – Grammatical Error Correction: Tag, Not Rewrite

In this paper, we present a simple and efficient GEC sequence tagger usi...
research
05/22/2023

Text-to-SQL Error Correction with Language Models of Code

Despite recent progress in text-to-SQL parsing, current semantic parsers...
research
07/30/2018

Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder

We present an atrous convolutional encoder-decoder trained to denoise 51...
research
05/24/2020

Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model

Grammatical error correction (GEC) literature has reported on the effect...
research
09/30/2021

BERT got a Date: Introducing Transformers to Temporal Tagging

Temporal expressions in text play a significant role in language underst...

Please sign up or login with your details

Forgot password? Click here to reset