Facet-Aware Evaluation for Extractive Text Summarization

08/27/2019
by   Yuning Mao, et al.
0

Commonly adopted metrics for extractive text summarization like ROUGE focus on the lexical similarity and are facet-agnostic. In this paper, we present a facet-aware evaluation procedure for better assessment of the information coverage in extracted summaries while still supporting automatic evaluation once annotated. Specifically, we treat facet instead of token as the basic unit for evaluation, manually annotate the support sentences for each facet, and directly evaluate extractive methods by comparing the indices of extracted sentences with support sentences. We demonstrate the benefits of the proposed setup by performing a thorough quantitative investigation on the CNN/Daily Mail dataset, which in the meantime reveals useful insights of state-of-the-art summarization methods.[%s]

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/04/2018

Abstractive Summarization Improved by WordNet-based Extractive Sentences

Recently, the seq2seq abstractive summarization models have achieved goo...
research
08/25/2015

Better Summarization Evaluation with Word Embeddings for ROUGE

ROUGE is a widely adopted, automatic evaluation measure for text summari...
research
12/19/2022

Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences

Lack of factual correctness is an issue that still plagues state-of-the-...
research
08/01/2022

SMART: Sentences as Basic Units for Text Evaluation

Widely used evaluation metrics for text generation either do not work we...
research
04/19/2020

Extractive Summarization as Text Matching

This paper creates a paradigm shift with regard to the way we build neur...
research
05/04/2021

Semantic Extractor-Paraphraser based Abstractive Summarization

The anthology of spoken languages today is inundated with textual inform...
research
03/26/2023

SASS: Data and Methods for Subject Aware Sentence Simplification

Sentence simplification tends to focus on the generic simplification of ...

Please sign up or login with your details

Forgot password? Click here to reset