Long Document Summarization in a Low Resource Setting using Pretrained Language Models

by   Ahsaas Bajaj, et al.

Abstractive summarization is the task of compressing a long document into a coherent short document while retaining salient information. Modern abstractive summarization methods are based on deep neural networks which often require large training datasets. Since collecting summarization datasets is an expensive and time-consuming task, practical industrial settings are usually low-resource. In this paper, we study a challenging low-resource setting of summarizing long legal briefs with an average source document length of 4268 words and only 120 available (document, summary) pairs. To account for data scarcity, we used a modern pretrained abstractive summarizer BART (Lewis et al., 2020), which only achieves 17.9 ROUGE-L as it struggles with long documents. We thus attempt to compress these long documents by identifying salient sentences in the source which best ground the summary, using a novel algorithm based on GPT-2 (Radford et al., 2019) language model perplexity scores, that operates within the low resource regime. On feeding the compressed documents to BART, we observe a 6.0 ROUGE-L improvement. Our method also beats several competitive salience detection baselines. Furthermore, the identified salient sentences tend to agree with an independent human labeling by domain experts.


page 1

page 2

page 3

page 4


Unsupervised Abstractive Summarization of Bengali Text Documents

Abstractive summarization systems generally rely on large collections of...

On Generating Extended Summaries of Long Documents

Prior work in document summarization has mainly focused on generating sh...

Incorporating Domain Knowledge for Extractive Summarization of Legal Case Documents

Automatic summarization of legal case documents is an important and prac...

Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization

Abstractive summarization, the task of generating a concise summary of i...

TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation

Many scientific papers such as those in arXiv and PubMed data collection...

Reinforced Abstractive Summarization with Adaptive Length Controlling

Document summarization, as a fundamental task in natural language genera...

Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark

The GLUE benchmark (Wang et al., 2019b) is a suite of language understan...