Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task

by   Thibault Sellam, et al.

The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem. This paper describes our contribution to the WMT 2020 Metrics Shared Task, the main benchmark for automatic evaluation of translation. We make several submissions based on BLEURT, a previously published metric based on transfer learning. We extend the metric beyond English and evaluate it on 14 language pairs for which fine-tuning data is available, as well as 4 "zero-shot" language pairs, for which we have no labelled examples. Additionally, we focus on English to German and demonstrate how to combine BLEURT's predictions with those of YiSi and use alternative reference translations to enhance the performance. Empirical results show that the models achieve competitive results on the WMT Metrics 2019 Shared Task, indicating their promise for the 2020 edition.


page 1

page 2

page 3

page 4


Machine Translation Evaluation with BERT Regressor

We introduce the metric using BERT (Bidirectional Encoder Representation...

RoBLEURT Submission for the WMT2021 Metrics Task

In this paper, we present our submission to Shared Metrics Task: RoBLEUR...

LIUM-CVC Submissions for WMT18 Multimodal Translation Task

This paper describes the multimodal Neural Machine Translation systems d...

CushLEPOR: Customised hLEPOR Metric Using LABSE Distilled Knowledge Model to Improve Agreement with Human Judgements

Human evaluation has always been expensive while researchers struggle to...

Unbabel's Participation in the WMT19 Translation Quality Estimation Shared Task

We present the contribution of the Unbabel team to the WMT 2019 Shared T...

Addressing Exposure Bias With Document Minimum Risk Training: Cambridge at the WMT20 Biomedical Translation Task

The 2020 WMT Biomedical translation task evaluated Medline abstract tran...

CUNI systems for WMT21: Terminology translation Shared Task

This paper describes Charles University submission for Terminology trans...