Log In Sign Up

PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents

by   Ryo Fujii, et al.

Neural Machine Translation (NMT) has shown drastic improvement in its quality when translating clean input, such as text from the news domain. However, existing studies suggest that NMT still struggles with certain kinds of input with considerable noise, such as User-Generated Contents (UGC) on the Internet. To make better use of NMT for cross-cultural communication, one of the most promising directions is to develop a model that correctly handles these expressions. Though its importance has been recognized, it is still not clear as to what creates the great gap in performance between the translation of clean input and that of UGC. To answer the question, we present a new dataset, PheMT, for evaluating the robustness of MT systems against specific linguistic phenomena in Japanese-English translation. Our experiments with the created dataset revealed that not only our in-house models but even widely used off-the-shelf systems are greatly disturbed by the presence of certain phenomena.


page 1

page 2

page 3

page 4


Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back Translation

Neural Machine Translation (NMT) models have been proved strong when tra...

Improving Robustness of Machine Translation with Synthetic Noise

Modern Machine Translation (MT) systems perform consistently well on cle...

Robust Neural Machine Translation with Joint Textual and Phonetic Embedding

Neural machine translation (NMT) is notoriously sensitive to noises, but...

Context in Neural Machine Translation: A Review of Models and Evaluations

This review paper discusses how context has been used in neural machine ...

Addressing the Vulnerability of NMT in Input Perturbations

Neural Machine Translation (NMT) has achieved significant breakthrough i...

Analyzing Neural MT Search and Model Performance

In this paper, we offer an in-depth analysis about the modeling and sear...