DeepAI AI Chat
Log In Sign Up

On Improving Summarization Factual Consistency from Natural Language Feedback

12/20/2022
by   Yixin Liu, et al.
0

Despite the recent progress in language generation models, their outputs may not always meet user expectations. In this work, we study whether informational feedback in natural language can be leveraged to improve generation quality and user preference alignment. To this end, we consider factual consistency in summarization, the quality that the summary should only contain information supported by the input documents, for user preference alignment. We collect a high-quality dataset, DeFacto, containing human demonstrations and informational feedback in natural language consisting of corrective instructions, edited summaries, and explanations with respect to the factual consistency of the summary. Using our dataset, we study two natural language generation tasks: 1) editing a summary using the human feedback, and 2) generating human feedback from the original summary. Using the two tasks, we further evaluate if models can automatically correct factual inconsistencies in generated summaries. We show that the human-edited summaries we collected are more factually consistent, and pre-trained language models can leverage our dataset to improve the factual consistency of original system-generated summaries in our proposed generation tasks. We make the DeFacto dataset publicly available at https://github.com/microsoft/DeFacto.

READ FULL TEXT
05/04/2022

Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking

Despite the recent advances in abstractive summarization systems, it is ...
05/04/2023

ChatGPT-steered Editing Instructor for Customization of Abstractive Summarization

Tailoring outputs of large language models, such as ChatGPT, to specific...
11/07/2022

CELLS: A Parallel Corpus for Biomedical Lay Language Generation

Recent lay language generation systems have used Transformer models trai...
05/24/2023

SummIt: Iterative Text Summarization via ChatGPT

Existing text summarization systems have made significant progress in re...
04/12/2022

Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback

For summarization, human preference is critical to tame outputs of the s...
12/03/2022

CoP: Factual Inconsistency Detection by Controlling the Preference

Abstractive summarization is the process of generating a summary given a...
12/15/2021

Interscript: A dataset for interactive learning of scripts through error feedback

How can an end-user provide feedback if a deployed structured prediction...