Otem&Utem: Over- and Under-Translation Evaluation Metric for NMT

07/24/2018
by   Jing Yang, et al.
0

Although neural machine translation(NMT) yields promising translation performance, it unfortunately suffers from over- and under-translation is- sues [Tu et al., 2016], of which studies have become research hotspots in NMT. At present, these studies mainly apply the dominant automatic evaluation metrics, such as BLEU, to evaluate the overall translation quality with respect to both adequacy and uency. However, they are unable to accurately measure the ability of NMT systems in dealing with the above-mentioned issues. In this paper, we propose two quantitative metrics, the Otem and Utem, to automatically evaluate the system perfor- mance in terms of over- and under-translation respectively. Both metrics are based on the proportion of mismatched n-grams between gold ref- erence and system translation. We evaluate both metrics by comparing their scores with human evaluations, where the values of Pearson Cor- relation Coefficient reveal their strong correlation. Moreover, in-depth analyses on various translation systems indicate some inconsistency be- tween BLEU and our proposed metrics, highlighting the necessity and significance of our metrics.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/15/2021

Reward Optimization for Neural Machine Translation with Learned Metrics

Neural machine translation (NMT) models are conventionally trained with ...
research
09/14/2019

Beyond BLEU: Training Neural Machine Translation with Semantic Similarity

While most neural machine translation (NMT) systems are still trained us...
research
05/04/2020

Evaluating Explanation Methods for Neural Machine Translation

Recently many efforts have been devoted to interpreting the black-box NM...
research
05/04/2020

Using Context in Neural Machine Translation Training Objectives

We present Neural Machine Translation (NMT) training using document-leve...
research
06/06/2022

Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation

We introduce Bi-SimCut: a simple but effective training strategy to boos...
research
05/10/2018

First Experiments with Neural Translation of Informal to Formal Mathematics

We report on our first experiments to train deep neural networks that au...

Please sign up or login with your details

Forgot password? Click here to reset