Discourse Structure in Machine Translation Evaluation

10/04/2017
by   Shafiq Joty, et al.
0

In this article, we explore the potential of using sentence-level discourse structure for machine translation evaluation. We first design discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory (RST). Then, we show that a simple linear combination with these measures can help improve various existing machine translation evaluation metrics regarding correlation with human judgments both at the segment- and at the system-level. This suggests that discourse information is complementary to the information used by many of the existing evaluation metrics, and thus it could be taken into account when developing richer evaluation metrics, such as the WMT-14 winning combined metric DiscoTKparty. We also provide a detailed analysis of the relevance of various discourse elements and relations from the RST parse trees for machine translation evaluation. In particular we show that: (i) all aspects of the RST tree are relevant, (ii) nuclearity is more useful than relation type, and (iii) the similarity of the translation RST tree to the reference tree is positively correlated with translation quality.

READ FULL TEXT

page 23

page 24

page 25

research
11/28/2019

DiscoTK: Using Discourse Structure for Machine Translation Evaluation

We present novel automatic metrics for machine translation evaluation th...
research
01/26/2022

DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence

Recently, there has been a growing interest in designing text generation...
research
10/07/2018

Assessing Crosslingual Discourse Relations in Machine Translation

In an attempt to improve overall translation quality, there has been an ...
research
10/01/2022

FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation

We present FRMT, a new dataset and evaluation benchmark for Few-shot Reg...
research
09/20/2022

Can we do that simpler? Simple, Efficient, High-Quality Evaluation Metrics for NLG

We explore efficient evaluation metrics for Natural Language Generation ...
research
12/20/2022

BMX: Boosting Machine Translation Metrics with Explainability

State-of-the-art machine translation evaluation metrics are based on bla...
research
12/03/2020

SemMT: A Semantic-based Testing Approach for Machine Translation Systems

Machine translation has wide applications in daily life. In mission-crit...

Please sign up or login with your details

Forgot password? Click here to reset