Evaluating the Efficacy of Summarization Evaluation across Languages

06/02/2021
by   Fajri Koto, et al.
12

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/18/2023

Revisiting Automatic Question Summarization Evaluation in the Biomedical Domain

Automatic evaluation metrics have been facilitating the rapid developmen...
research
05/22/2023

SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation

Reliable automatic evaluation of summarization systems is challenging du...
research
10/24/2018

A Multilingual Study of Compressive Cross-Language Text Summarization

Cross-Language Text Summarization (CLTS) generates summaries in a langua...
research
04/28/2022

Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization

In zero-shot multilingual extractive text summarization, a model is typi...
research
10/24/2018

Effective extractive summarization using frequency-filtered entity relationship graphs

Word frequency-based methods for extractive summarization are easy to im...
research
09/16/2021

Does Summary Evaluation Survive Translation to Other Languages?

The creation of a large summarization quality dataset is a considerable,...
research
05/20/2020

Examining the State-of-the-Art in News Timeline Summarization

Previous work on automatic news timeline summarization (TLS) leaves an u...

Please sign up or login with your details

Forgot password? Click here to reset