Evaluation of Automatic Text Summarization using Synthetic Facts

04/11/2022
by   Jay Ahn, et al.
0

Despite some recent advances, automatic text summarization remains unreliable, elusive, and of limited practical use in applications. Two main problems with current summarization methods are well known: evaluation and factual consistency. To address these issues, we propose a new automatic reference-less text summarization evaluation system that can measure the quality of any text summarization model with a set of generated facts based on factual consistency, comprehensiveness, and compression rate. As far as we know, our evaluation system is the first system that measures the overarching quality of the text summarization models based on factuality, information coverage, and compression rate.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/19/2020

Dataset for Automatic Summarization of Russian News

Automatic text summarization has been studied in a variety of domains an...
research
09/04/2018

Étude de l'informativité des transcriptions : une approche basée sur le résumé automatique

In this paper we propose a new approach to evaluate the informativeness ...
research
10/11/2018

Semantic Structural Evaluation for Text Simplification

Current measures for evaluating text simplification systems focus on eva...
research
03/27/2023

Large Language Models are Diverse Role-Players for Summarization Evaluation

Text summarization has a wide range of applications in many scenarios. T...
research
05/24/2022

MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification

In text summarization and simplification, system outputs must be evaluat...
research
10/31/2022

Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency

The topic of summarization evaluation has recently attracted a surge of ...
research
05/30/2019

Assessing The Factual Accuracy of Generated Text

We propose a model-based metric to estimate the factual accuracy of gene...

Please sign up or login with your details

Forgot password? Click here to reset