Analyzing and Evaluating Faithfulness in Dialogue Summarization

10/21/2022
by   Bin Wang, et al.
0

Dialogue summarization is abstractive in nature, making it suffer from factual errors. The factual correctness of summaries has the highest priority before practical applications. Many efforts have been made to improve faithfulness in text summarization. However, there is a lack of systematic study on dialogue summarization systems. In this work, we first perform the fine-grained human analysis on the faithfulness of dialogue summaries and observe that over 35 respective the source dialogues. Furthermore, we present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations. Experimental results show that our evaluation schema is a strong proxy for the factual correctness of summarization models. The human-annotated faithfulness samples and the evaluation toolkit are released to facilitate future research toward faithful dialogue summarization.

READ FULL TEXT
research
10/17/2022

Leveraging Non-dialogue Summaries for Dialogue Summarization

To mitigate the lack of diverse dialogue summarization datasets in acade...
research
12/16/2021

CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning

Factual inconsistencies in generated summaries severely limit the practi...
research
06/16/2021

Coreference-Aware Dialogue Summarization

Summarizing conversations via neural approaches has been gaining researc...
research
12/19/2022

Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences

Lack of factual correctness is an issue that still plagues state-of-the-...
research
05/02/2023

The Role of Summarization in Generative Agents: A Preliminary Perspective

Generative agents that simulate human society show tremendous potential ...
research
06/08/2023

Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework

Factuality is important to dialogue summarization. Factual error correct...
research
05/26/2022

Unsupervised Abstractive Dialogue Summarization with Word Graphs and POV Conversion

We advance the state-of-the-art in unsupervised abstractive dialogue sum...

Please sign up or login with your details

Forgot password? Click here to reset