AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization

by   Tiezheng Yu, et al.

State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model's catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.


page 1

page 2

page 3

page 4


Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation

We consider unsupervised domain adaptation (UDA), where labeled data fro...

A Broad Study of Pre-training for Domain Generalization and Adaptation

Deep models must learn robust and transferable representations in order ...

A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization

We study the problem of domain adaptation for neural abstractive summari...

Meta-Transfer Learning for Low-Resource Abstractive Summarization

Neural abstractive summarization has been studied in many pieces of lite...

Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization

The most advanced abstractive dialogue summarizers lack generalization a...

Low-Resource Neural Headline Generation

Recent neural headline generation models have shown great results, but a...

LRS-DAG: Low Resource Supervised Domain Adaptation with Generalization Across Domains

Current state of the art methods in Domain Adaptation follow adversarial...

Code Repositories


The code repository for NAACL 2021 paper "AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization".

view repo