Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings

04/12/2021
by   Oleg Vasilyev, et al.
0

We propose a new reference-free summary quality evaluation measure, with emphasis on the faithfulness. The measure is designed to find and count all possible minute inconsistencies of the summary with respect to the source document. The proposed ESTIME, Estimator of Summary-to-Text Inconsistency by Mismatched Embeddings, correlates with expert scores in summary-level SummEval dataset stronger than other common evaluation measures not only in Consistency but also in Fluency. We also introduce a method of generating subtle factual errors in human summaries. We show that ESTIME is more sensitive to subtle errors than other common evaluation measures.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/23/2020

Fill in the BLANC: Human-free quality estimation of document summaries

We present BLANC, a new approach to the automatic estimation of document...
research
10/25/2022

Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics

Despite extensive recent advances in summary generation models, evaluati...
research
12/29/2020

Is human scoring the best criteria for summary evaluation?

Normally, summary quality measures are compared with quality scores prod...
research
12/22/2021

Consistency and Coherence from Points of Contextual Similarity

Factual consistency is one of important summary evaluation dimensions, e...
research
09/12/2018

Semantic WordRank: Generating Finer Single-Document Summarizations

We present Semantic WordRank (SWR), an unsupervised method for generatin...
research
03/19/2021

Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation

The goal of a summary is to concisely state the most important informati...
research
09/10/2020

Classification of descriptions and summary using multiple passes of statistical and natural language toolkits

This document describes a possible approach that can be used to check th...

Please sign up or login with your details

Forgot password? Click here to reset