ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks

03/05/2018
by   Kavita Ganesan, et al.
0

Evaluation of summarization tasks is extremely crucial to determining the quality of machine generated summaries. Over the last decade, ROUGE has become the standard automatic evaluation measure for evaluating summarization tasks. While ROUGE has been shown to be effective in capturing n-gram overlap between system and human composed summaries, there are several limitations with the existing ROUGE measures in terms of capturing synonymous concepts and coverage of topics. Thus, often times ROUGE scores do not reflect the true quality of summaries and prevents multi-faceted evaluation of summaries (i.e. by topics, by overall content coverage and etc). In this paper, we introduce ROUGE 2.0, which has several updated measures of ROUGE: ROUGE-N+Synonyms, ROUGE-Topic, ROUGE-Topic+Synonyms, ROUGE-TopicUniq and ROUGE-TopicUniq+Synonyms; all of which are improvements over the core ROUGE measures.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/23/2020

Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries

Reference-based metrics such as ROUGE or BERTScore evaluate the content ...
research
06/11/2019

Generating Summaries with Topic Templates and Structured Convolutional Decoders

Existing neural generation approaches create multi-sentence text as a si...
research
05/10/2021

Improving Factual Consistency of Abstractive Summarization via Question Answering

A commonly observed problem with the state-of-the art abstractive summar...
research
01/16/2020

Intweetive Text Summarization

The amount of user generated contents from various social medias allows ...
research
02/26/2021

Neural Code Summarization

Code summarization is the task of generating readable summaries that are...
research
09/12/2023

Evaluating Dynamic Topic Models

There is a lack of quantitative measures to evaluate the progression of ...
research
10/24/2018

Effective extractive summarization using frequency-filtered entity relationship graphs

Word frequency-based methods for extractive summarization are easy to im...

Please sign up or login with your details

Forgot password? Click here to reset