DeepAI AI Chat
Log In Sign Up

Removing Biases from Trainable MT Metrics by Using Self-Training

by   Miloš Stanojević, et al.
University of Amsterdam

Most trainable machine translation (MT) metrics train their weights on human judgments of state-of-the-art MT systems outputs. This makes trainable metrics biases in many ways. One of them is preferring longer translations. These biased metrics when used for tuning are evaluating different types of translations -- n-best lists of translations with very diverse quality. Systems tuned with these metrics tend to produce overly long translations that are preferred by the metric but not by humans. This is usually solved by manually tweaking metric's weights to equally value recall and precision. Our solution is more general: (1) it does not address only the recall bias but also all other biases that might be present in the data and (2) it does not require any knowledge of the types of features used which is useful in cases when manual tuning of metric's weights is not possible. This is accomplished by self-training on unlabeled n-best lists by using metric that was initially trained on standard human judgments. One way of looking at this is as domain adaptation from the domain of state-of-the-art MT translations to diverse n-best list translations.


page 1

page 2


Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature

Literary translation is a culturally significant task, but it is bottlen...

Detecting over/under-translation errors for determining adequacy in human translations

We present a novel approach to detecting over and under translations (OT...

Isometric MT: Neural Machine Translation for Automatic Dubbing

Automatic dubbing (AD) is among the use cases where translations should ...

GLEU Without Tuning

The GLEU metric was proposed for evaluating grammatical error correction...

Analysing Coreference in Transformer Outputs

We analyse coreference phenomena in three neural machine translation sys...

Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT

Although measuring intrinsic quality has been a key factor in the advanc...