DeepAI AI Chat
Log In Sign Up

Evaluating Machine Translation Performance on Chinese Idioms with a Blacklist Method

by   Yutong Shao, et al.

Idiom translation is a challenging problem in machine translation because the meaning of idioms is non-compositional, and a literal translation is likely to be wrong. In this paper, we assess the quality of idiom translation of a modern neural MT system. We introduce a new evaluation method based on an idiom-specific blacklist of literal translations, based on the insight that the occurrence of any blacklisted words in the translation output indicates a likely translation error. We introduce a dataset, CIBB (Chinese Idioms Blacklists Bank), and perform an evaluation of a state-of-the-art Chinese-English neural MT system. Our evaluation confirms that our blacklist method is effective at identifying literal translation errors, and that a sizable number of idioms in our test set are mistranslated (36.5


page 1

page 2

page 3

page 4


Upping the Ante: Towards a Better Benchmark for Chinese-to-English Machine Translation

There are many machine translation (MT) papers that propose novel approa...

Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

One of the major challenges of machine translation (MT) is ambiguity, wh...

Understanding the Impact of UGC Specificities on Translation Quality

This work takes a critical look at the evaluation of user-generated cont...

Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015

This year, the Nara Institute of Science and Technology (NAIST)'s submis...

SAO WMT19 Test Suite: Machine Translation of Audit Reports

This paper describes a machine translation test set of documents from th...

MTNT: A Testbed for Machine Translation of Noisy Text

Noisy or non-standard input text can cause disastrous mistranslations in...

An Overview on Machine Translation Evaluation

Since the 1950s, machine translation (MT) has become one of the importan...