On the Effectiveness of Automated Metrics for Text Generation Systems

10/24/2022
by   Pius von Däniken, et al.
0

A major challenge in the field of Text Generation is evaluation because we lack a sound theory that can be leveraged to extract guidelines for evaluation campaigns. In this work, we propose a first step towards such a theory that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. The theory has practical applications, such as determining the number of samples needed to reliably distinguish the performance of a set of Text Generation systems in a given setting. We showcase the application of the theory on the WMT 21 and Spot-The-Bot evaluation data and outline how it can be leveraged to improve the evaluation protocol regarding the reliability, robustness, and significance of the evaluation outcome.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/06/2023

Correction of Errors in Preference Ratings from Automated Metrics for Text Generation

A major challenge in the field of Text Generation is evaluation: Human e...
research
10/19/2019

Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation

Neural conditional text generation systems have achieved significant pro...
research
09/25/2020

Weird AI Yankovic: Generating Parody Lyrics

Lyrics parody swaps one set of words that accompany a melody with a new ...
research
05/25/2022

R2D2: Robust Data-to-Text with Replacement Detection

Unfaithful text generation is a common problem for text generation syste...
research
01/17/2021

GENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation

Leaderboards have eased model development for many NLP datasets by stand...
research
06/30/2021

Evaluation of Thematic Coherence in Microblogs

Collecting together microblogs representing opinions about the same topi...
research
09/04/2023

Automatic Scam-Baiting Using ChatGPT

Automatic scam-baiting is an online fraud countermeasure that involves a...

Please sign up or login with your details

Forgot password? Click here to reset