DeepAI AI Chat
Log In Sign Up

Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement

by   Zhen Yang, et al.

Word-level Quality Estimation (QE) of Machine Translation (MT) aims to find out potential translation errors in the translated sentence without reference. Typically, conventional works on word-level QE are designed to predict the translation quality in terms of the post-editing effort, where the word labels ("OK" and "BAD") are automatically generated by comparing words between MT sentences and the post-edited sentences through a Translation Error Rate (TER) toolkit. While the post-editing effort can be used to measure the translation quality to some extent, we find it usually conflicts with the human judgement on whether the word is well or poorly translated. To overcome the limitation, we first create a golden benchmark dataset, namely HJQE (Human Judgement on Quality Estimation), where the expert translators directly annotate the poorly translated words on their judgements. Additionally, to further make use of the parallel corpus, we propose the self-supervised pre-training with two tag correcting strategies, namely tag refinement strategy and tree-based annotation strategy, to make the TER-based artificial QE corpus closer to HJQE. We conduct substantial experiments based on the publicly available WMT En-De and En-Zh corpora. The results not only show our proposed dataset is more consistent with human judgment but also confirm the effectiveness of the proposed tag correcting strategies.[The data can be found at <>.]


MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset

We present MLQE-PE, a new dataset for Machine Translation (MT) Quality E...

IntelliCAT: Intelligent Machine Translation Post-Editing with Quality Estimation and Translation Suggestion

We present IntelliCAT, an interactive translation interface with neural ...

Quality Estimation without Human-labeled Data

Quality estimation aims to measure the quality of translated content wit...

TranSmart: A Practical Interactive Machine Translation System

Automatic machine translation is super efficient to produce translations...

An Empirical Study of Automatic Post-Editing

Automatic post-editing (APE) aims to reduce manual post-editing efforts ...

A Self-Supervised Automatic Post-Editing Data Generation Tool

Data building for automatic post-editing (APE) requires extensive and ex...

GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation

Computer-aided translation (CAT), the use of software to assist a human ...