DeepAI AI Chat
Log In Sign Up

Fine-grained linguistic evaluation for state-of-the-art Machine Translation

by   Eleftherios Avramidis, et al.

This paper describes a test suite submission providing detailed statistics of linguistic performance for the state-of-the-art German-English systems of the Fifth Conference of Machine Translation (WMT20). The analysis covers 107 phenomena organized in 14 categories based on about 5,500 test items, including a manual annotation effort of 45 person hours. Two systems (Tohoku and Huoshan) appear to have significantly better test suite accuracy than the others, although the best system of WMT20 is not significantly better than the one from WMT19 in a macro-average. Additionally, we identify some linguistic phenomena where all systems suffer (such as idioms, resultative predicates and pluperfect), but we are also able to identify particular weaknesses for individual systems (such as quotation marks, lexical ambiguity and sluicing). Most of the systems of WMT19 which submitted new versions this year show improvements.


Linguistic evaluation of German-English Machine Translation using a Test Suite

We present the results of the application of a grammatical test suite fo...

Fine-grained evaluation of German-English Machine Translation based on a Test Suite

We present an analysis of 16 state-of-the-art MT systems on German-Engli...

A Test Suite for the Evaluation of Portuguese-English Machine Translation

This paper describes the development of the first test suite for the lan...

A Challenge Set Approach to Evaluating Machine Translation

Neural machine translation represents an exciting leap forward in transl...

Neural versus Phrase-Based Machine Translation Quality: a Case Study

Within the field of Statistical Machine Translation (SMT), the neural ap...

Pronoun Translation in English-French Machine Translation: An Analysis of Error Types

Pronouns are a long-standing challenge in machine translation. We presen...