Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite

08/31/2019
by   Prathyusha Jwalapuram, et al.
5

The ongoing neural revolution in machine translation has made it easier to model larger contexts beyond the sentence-level, which can potentially help resolve some discourse-level ambiguities such as pronominal anaphora, thus enabling better translations. Unfortunately, even when the resulting improvements are seen as substantial by humans, they remain virtually unnoticed by traditional automatic evaluation measures like BLEU, as only a few words end up being affected. Thus, specialized evaluation measures are needed. With this aim in mind, we contribute an extensive, targeted dataset that can be used as a test suite for pronoun translation, covering multiple source languages and different pronoun errors drawn from real system translations, for English. We further propose an evaluation measure to differentiate good and bad pronoun translations. We also conduct a user study to report correlations with human judgments.

READ FULL TEXT
research
08/19/2022

Discourse Cohesion Evaluation for Document-Level Neural Machine Translation

It is well known that translations generated by an excellent document-le...
research
08/08/2019

A Test Suite and Manual Evaluation of Document-Level NMT at WMT19

As the quality of machine translation rises and neural machine translati...
research
09/03/2019

Context-Aware Monolingual Repair for Neural Machine Translation

Modern sentence-level NMT systems often produce plausible translations o...
research
10/04/2018

A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation

The translation of pronouns presents a special challenge to machine tran...
research
10/16/2019

Fine-grained evaluation of Quality Estimation for Machine translation based on a linguistically-motivated Test Suite

We present an alternative method of evaluating Quality Estimation system...
research
10/11/2021

It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data

Most existing simultaneous machine translation (SiMT) systems are traine...
research
10/26/2020

Data Troubles in Sentence Level Confidence Estimation for Machine Translation

The paper investigates the feasibility of confidence estimation for neur...

Please sign up or login with your details

Forgot password? Click here to reset