Cross-Corpora Evaluation and Analysis of Grammatical Error Correction Models --- Is Single-Corpus Evaluation Enough?

04/05/2019
by   Masato Mita, et al.
0

This study explores the necessity of performing cross-corpora evaluation for grammatical error correction (GEC) models. GEC models have been previously evaluated based on a single commonly applied corpus: the CoNLL-2014 benchmark. However, the evaluation remains incomplete because the task difficulty varies depending on the test corpus and conditions such as the proficiency levels of the writers and essay topics. To overcome this limitation, we evaluate the performance of several GEC models, including NMT-based (LSTM, CNN, and transformer) and an SMT-based model, against various learner corpora (CoNLL-2013, CoNLL-2014, FCE, JFLEG, ICNALE, and KJ). Evaluation results reveal that the models' rankings considerably vary depending on the corpus, indicating that single-corpus evaluation is insufficient for GEC models.

READ FULL TEXT
research
02/14/2017

JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction

We present a new parallel corpus, JHU FLuency-Extended GUG corpus (JFLEG...
research
04/16/2021

Comparison of Grammatical Error Correction Using Back-Translation Models

Grammatical error correction (GEC) suffers from a lack of sufficient par...
research
01/17/2022

Proficiency Matters Quality Estimation in Grammatical Error Correction

This study investigates how supervised quality estimation (QE) models of...
research
05/26/2019

Evaluation of basic modules for isolated spelling error correction in Polish texts

Spelling error correction is an important problem in natural language pr...
research
07/27/2017

Correction of "A Comparative Study to Benchmark Cross-project Defect Prediction Approaches"

Unfortunately, the article "A Comparative Study to Benchmark Cross-proje...
research
05/10/2021

Cross-Corpora Language Recognition: A Preliminary Investigation with Indian Languages

In this paper, we conduct one of the very first studies for cross-corpor...
research
05/29/2023

Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods

Large-scale pre-trained language models such as GPT-3 have shown remarka...

Please sign up or login with your details

Forgot password? Click here to reset