AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization

03/21/2021
by   Tiezheng Yu, et al.
6

State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model's catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/01/2022

Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation

We consider unsupervised domain adaptation (UDA), where labeled data fro...
03/22/2022

A Broad Study of Pre-training for Domain Generalization and Adaptation

Deep models must learn robust and transferable representations in order ...
07/21/2017

A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization

We study the problem of domain adaptation for neural abstractive summari...
02/18/2021

Meta-Transfer Learning for Low-Resource Abstractive Summarization

Neural abstractive summarization has been studied in many pieces of lite...
04/09/2022

Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization

The most advanced abstractive dialogue summarizers lack generalization a...
07/31/2017

Low-Resource Neural Headline Generation

Recent neural headline generation models have shown great results, but a...
09/15/2019

LRS-DAG: Low Resource Supervised Domain Adaptation with Generalization Across Domains

Current state of the art methods in Domain Adaptation follow adversarial...

Code Repositories

AdaptSum

The code repository for NAACL 2021 paper "AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization".


view repo