SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

11/18/2021
by   Philippe Laban, et al.
2

In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. On our newly introduced benchmark called SummaC (Summary Consistency) consisting of six large inconsistency detection datasets, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4 a 5 available: https://github.com/tingofurro/summac

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/12/2022

Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization

Neural abstractive summarization models are prone to generate summaries ...
research
11/15/2022

Evaluating the Factual Consistency of Large Language Models Through Summarization

While large language models (LLMs) have proven to be effective on a larg...
research
06/17/2021

DocNLI: A Large-scale Dataset for Document-level Natural Language Inference

Natural language inference (NLI) is formulated as a unified framework fo...
research
12/28/2020

On Generating Extended Summaries of Long Documents

Prior work in document summarization has mainly focused on generating sh...
research
11/30/2022

Revisiting text decomposition methods for NLI-based factuality scoring of summaries

Scoring the factuality of a generated summary involves measuring the deg...
research
09/21/2022

Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees

Current abstractive summarization models either suffer from a lack of cl...
research
03/22/2021

Nutri-bullets: Summarizing Health Studies by Composing Segments

We introduce Nutri-bullets, a multi-document summarization task for heal...

Please sign up or login with your details

Forgot password? Click here to reset