DeepAI AI Chat
Log In Sign Up

Using Mechanical Turk to Build Machine Translation Evaluation Sets

10/20/2014
by   Michael Bloodgood, et al.
Johns Hopkins University
0

Building machine translation (MT) test sets is a relatively expensive task. As MT becomes increasingly desired for more and more language pairs and more and more domains, it becomes necessary to build test sets for each case. In this paper, we investigate using Amazon's Mechanical Turk (MTurk) to make MT test sets cheaply. We find that MTurk can be used to make test sets much cheaper than professionally-produced test sets. More importantly, in experiments with multiple MT systems, we find that the MTurk-produced test sets yield essentially the same conclusions regarding system performance as the professionally-produced test sets yield.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/07/2021

Variance-Aware Machine Translation Test Sets

We release 70 small and discriminative test sets for machine translation...
06/19/2019

The Effect of Translationese in Machine Translation Test Sets

The effect of translationese has been studied in the field of machine tr...
05/30/2019

DiaBLa: A Corpus of Bilingual Spontaneous Written Dialogues for Machine Translation

We present a new English-French test set for the evaluation of Machine T...
10/13/2021

Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits

Training data for machine translation (MT) is often sourced from a multi...
12/14/2016

Unsupervised Clustering of Commercial Domains for Adaptive Machine Translation

In this paper, we report on domain clustering in the ambit of an adaptiv...
12/20/2017

Adaptive Mantel Test for Penalized Inference, with Applications to Imaging Genetics

Mantel's test (MT) for association is conducted by testing the linear re...
05/04/2022

Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance

Human-translated text displays distinct features from naturally written ...