Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point

08/13/2018

∙

by   Liane Guillou, et al.

∙

0

∙

share

We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semi-automatic metrics and test suites in place of fully automatic metrics.