Neural Text Summarization: A Critical Evaluation

08/23/2019
by   Wojciech Kryściński, et al.
0

Text summarization aims at compressing long documents into a shorter form that conveys the most important parts of the original document. Despite increased interest in the community and notable research effort, progress on benchmark datasets has stagnated. We critically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlight three primary shortcomings: 1) automatically collected datasets leave the task underconstrained and may contain noise detrimental to training and evaluation, 2) current evaluation protocol is weakly correlated with human judgment and does not account for important characteristics such as factual correctness, 3) models overfit to layout biases of current datasets and offer limited diversity in their outputs.

READ FULL TEXT
research
07/03/2022

An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics

Long documents such as academic articles and business reports have been ...
research
07/24/2020

SummEval: Re-evaluating Summarization Evaluation

The scarcity of comprehensive up-to-date studies on evaluation metrics f...
research
05/18/2021

BookSum: A Collection of Datasets for Long-form Narrative Summarization

The majority of available text summarization datasets include short-form...
research
06/01/2023

Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study

Text summarization is a downstream natural language processing (NLP) tas...
research
01/26/2023

LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization

Text Summarization is a popular task and an active area of research for ...
research
04/15/2021

SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

Novel neural architectures, training strategies, and the availability of...
research
07/18/2022

Towards the Human Global Context: Does the Vision-Language Model Really Judge Like a Human Being?

As computer vision and NLP make progress, Vision-Language(VL) is becomin...

Please sign up or login with your details

Forgot password? Click here to reset