CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems

10/11/2020
by   Yiran Chen, et al.
0

Neural network-based models augmented with unsupervised pre-trained knowledge have achieved impressive performance on text summarization. However, most existing evaluation methods are limited to an in-domain setting, where summarizers are trained and evaluated on the same dataset. We argue that this approach can narrow our understanding of the generalization ability for different summarization systems. In this paper, we perform an in-depth analysis of characteristics of different datasets and investigate the performance of different summarization models under a cross-dataset setting, in which a summarizer trained on one corpus will be evaluated on a range of out-of-domain corpora. A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways (i.e. abstractive and extractive) on model generalization ability. Further, experimental results shed light on the limitations of existing summarizers. Brief introduction and supplementary code can be found in https://github.com/zide05/CDEvalSumm.

READ FULL TEXT

page 6

page 13

research
05/03/2023

TempoSum: Evaluating the Temporal Generalization of Abstractive Summarization

Recent pre-trained language models (PLMs) achieve promising results in e...
research
08/30/2019

Exploring Domain Shift in Extractive Text Summarization

Although domain shift has been well explored in many NLP applications, i...
research
09/30/2019

A Closer Look at Data Bias in Neural Extractive Summarization Models

In this paper, we take stock of the current state of summarization datas...
research
04/15/2021

RefSum: Refactoring Neural Summarization

Although some recent works show potential complementarity among differen...
research
01/02/2022

On the Cross-dataset Generalization in License Plate Recognition

Automatic License Plate Recognition (ALPR) systems have shown remarkable...
research
02/18/2021

Meta-Transfer Learning for Low-Resource Abstractive Summarization

Neural abstractive summarization has been studied in many pieces of lite...
research
12/07/2020

CX DB8: A queryable extractive summarizer and semantic search engine

Competitive Debate's increasingly technical nature has left competitors ...

Please sign up or login with your details

Forgot password? Click here to reset