DeepAI AI Chat
Log In Sign Up

Evaluation Discrepancy Discovery: A Sentence Compression Case-study

by   Yevgeniy Puzikov, et al.

Reliable evaluation protocols are of utmost importance for reproducible NLP research. In this work, we show that sometimes neither metric nor conventional human evaluation is sufficient to draw conclusions about system performance. Using sentence compression as an example task, we demonstrate how a system can game a well-established dataset to achieve state-of-the-art results. In contrast with the results reported in previous work that showed correlation between human judgements and metric scores, our manual analysis of state-of-the-art system outputs demonstrates that high metric scores may only indicate a better fit to the data, but not better outputs, as perceived by humans.


page 1

page 2

page 3

page 4


BMX: Boosting Machine Translation Metrics with Explainability

State-of-the-art machine translation evaluation metrics are based on bla...

Human acceptability judgements for extractive sentence compression

Recent approaches to English-language sentence compression rely on paral...

Learning Neural Word Salience Scores

Measuring the salience of a word is an essential step in numerous NLP ta...

Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers

This paper presents the first large-scale meta-evaluation of machine tra...

Better Smatch = Better Parser? AMR evaluation is not so simple anymore

Recently, astonishing advances have been observed in AMR parsing, as mea...