Detecting Hallucinated Content in Conditional Neural Sequence Generation

by   Chunting Zhou, et al.

Neural sequence models can generate highly fluent sentences but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input, which can cause a lack of trust in the model. To better assess the faithfulness of the machine outputs, we propose a new task to predict whether each token in the output sequence is hallucinated conditioned on the source input, and collect new manually annotated evaluation sets for this task. We also introduce a novel method for learning to model hallucination detection, based on pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations. Experiments on machine translation and abstract text summarization demonstrate the effectiveness of our proposed approach – we obtain an average F1 of around 0.6 across all the benchmark datasets and achieve significant improvements in sentence-level hallucination scoring compared to baseline methods. We also release our annotated data and code for future research at


page 1

page 2

page 3

page 4


CoDesc: A Large Code-Description Parallel Dataset

Translation between natural language and source code can help software d...

Salience Estimation with Multi-Attention Learning for Abstractive Text Summarization

Attention mechanism plays a dominant role in the sequence generation mod...

Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models

This paper studies how to automatically generate a natural language text...

BenchCLAMP: A Benchmark for Evaluating Language Models on Semantic Parsing

We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Mo...

Sequential Models in the Synthetic Data Vault

The goal of this paper is to describe a system for generating synthetic ...

PLOD: An Abbreviation Detection Dataset for Scientific Documents

The detection and extraction of abbreviations from unstructured texts ca...

An End-to-End Dialogue Summarization System for Sales Calls

Summarizing sales calls is a routine task performed manually by salespeo...