Log In Sign Up

Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods

by   Potsawee Manakul, et al.

Automatic summary assessment is useful for both machine-generated and human-produced summaries. Automatically evaluating the summary text given the document enables, for example, summary generation system development and detection of inappropriate summaries. Summary assessment can be run in a number of modes: ranking summary generation systems; ranking summaries of a particular document; and estimating the quality of a document-summary pair on an absolute scale. Existing datasets with annotation for summary assessment are usually based on news summarization datasets such as CNN/DailyMail or XSum. In this work, we describe a new dataset, the podcast summary assessment corpus, a collection of podcast summaries that were evaluated by human experts at TREC2020. Compared to existing summary assessment data, this dataset has two unique aspects: (i) long-input, speech podcast based, documents; and (ii) an opportunity to detect inappropriate reference summaries in podcast corpus. First, we examine existing assessment methods, including model-free and model-based methods, and provide benchmark results for this long-input summary assessment dataset. Second, with the aim of filtering reference summary-document pairings for training, we apply summary assessment for data selection. The experimental results on these two aspects provide interesting insights on the summary assessment and generation tasks. The podcast summary assessment data is available.


page 1

page 2

page 3

page 4


Sensitivity of BLANC to human-scored qualities of text summaries

We explore the sensitivity of a document summary quality estimator, BLAN...

Video Summarization via Actionness Ranking

To automatically produce a brief yet expressive summary of a long video,...

HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization

Document structure is critical for efficient information consumption. Ho...

Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics

Despite extensive recent advances in summary generation models, evaluati...

Guided Exploration of Data Summaries

Data summarization is the process of producing interpretable and represe...

TGSum: Build Tweet Guided Multi-Document Summarization Dataset

The development of summarization research has been significantly hampere...

How to Write Summaries with Patterns? Learning towards Abstractive Summarization through Prototype Editing

Under special circumstances, summaries should conform to a particular st...