Automatic Metric Validation for Grammatical Error Correction

04/30/2018
by   Leshem Choshen, et al.
0

Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with shed a new light on metric quality, showing for example that the standard M^2 metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/18/2023

HTEC: Human Transcription Error Correction

High-quality human transcription is essential for training and improving...
research
10/07/2016

There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction

Current methods for automatically evaluating grammatical error correctio...
research
02/14/2019

On Many-to-Many Mapping Between Concordance Correlation Coefficient and Mean Square Error

The concordance correlation coefficient (CCC) is one of the most widely ...
research
07/02/2017

Grammatical Error Correction with Neural Reinforcement Learning

We propose a neural encoder-decoder model with reinforcement learning (N...
research
12/01/2022

The Subfield Metric and its Application to Quantum Error Correction

We introduce a new weight and corresponding metric over finite extension...
research
07/31/2019

On conducting better validation studies of automatic metrics in natural language generation evaluation

Natural language generation (NLG) has received increasing attention, whi...
research
08/29/2019

A General Model Validation and Testing Tool

We construct and propose the "Bayesian Validation Metric" (BVM) as a gen...

Please sign up or login with your details

Forgot password? Click here to reset