Automatic Testing and Improvement of Machine Translation

10/07/2019

∙

by   Zeyu Sun, et al.

∙

0

∙

share

This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic testing to detect inconsistency bugs (without access to human oracles). It then adopts probability-reference or cross-reference to post-process the translations, in a grey-box or black-box manner, to repair the inconsistencies. Our evaluation on two state-of-the-art translators, Google Translate and Transformer, indicates that TransRepair has a high precision (99 using automatic consistency metrics and manual assessment, we find that Google Translate and Transformer have approximately 36 Black-box repair fixes 28 Transformer. Grey-box repair fixes 30 inspection indicates that the translations repaired by our approach improve consistency in 87 better translation acceptability in 27