Sensitivity of BLANC to human-scored qualities of text summaries

10/13/2020
by   Oleg Vasilyev, et al.
5

We explore the sensitivity of a document summary quality estimator, BLANC, to human assessment of qualities for the same summaries. In our human evaluations, we distinguish five summary qualities, defined by how fluent, understandable, informative, compact, and factually correct the summary is. We make the case for optimal BLANC parameters, at which the BLANC sensitivity to almost all of summary qualities is about as good as the sensitivity of a human annotator.

READ FULL TEXT
research
02/23/2020

Fill in the BLANC: Human-free quality estimation of document summaries

We present BLANC, a new approach to the automatic estimation of document...
research
08/28/2022

Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods

Automatic summary assessment is useful for both machine-generated and hu...
research
04/09/2019

A sensitivity analysis of the PAWN sensitivity index

The PAWN index is gaining traction among the modelling community as a mo...
research
04/28/2020

Human-Like Summaries from Heterogeneous and Time-Windowed Software Development Artefacts

Automatic text summarisation has drawn considerable interest in the area...
research
01/12/2020

Sensitivity analysis methods in the biomedical sciences

Sensitivity analysis is an important part of a mathematical modeller's t...
research
09/08/2023

From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting

Selecting the “right” amount of information to include in a summary is a...
research
12/29/2020

Is human scoring the best criteria for summary evaluation?

Normally, summary quality measures are compared with quality scores prod...

Please sign up or login with your details

Forgot password? Click here to reset