Fact Checking with Insufficient Evidence

04/05/2022
by   Pepa Atanasova, et al.
0

Automating the fact checking (FC) process relies on information obtained from external sources. In this work, we posit that it is crucial for FC models to make veracity predictions only when there is sufficient evidence and otherwise indicate when it is not enough. To this end, we are the first to study what information FC models consider sufficient by introducing a novel task and advancing it with three main contributions. First, we conduct an in-depth empirical analysis of the task with a new fluency-preserving method for omitting information from the evidence at the constituent and sentence level. We identify when models consider the remaining evidence (in)sufficient for FC, based on three trained models with different Transformer architectures and three FC datasets. Second, we ask annotators whether the omitted evidence was important for FC, resulting in a novel diagnostic dataset, SufficientFacts, for FC with omitted evidence. We find that models are least successful in detecting missing evidence when adverbial modifiers are omitted (21 it is easiest for omitted date modifiers (63 novel data augmentation strategy for contrastive self-learning of missing evidence by employing the proposed omission method combined with tri-training. It improves performance for Evidence Sufficiency Prediction by up to 17.8 F1 score, which in turn improves FC performance by up to 2.6 F1 score.

READ FULL TEXT
research
04/07/2021

BreakingBERT@IITK at SemEval-2021 Task 9 : Statement Verification and Evidence Finding with Tables

Recently, there has been an interest in factual verification and predict...
research
10/25/2022

Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation

Misinformation emerges in times of uncertainty when credible information...
research
09/01/2023

FactLLaMA: Optimizing Instruction-Following Language Models with External Knowledge for Automated Fact-Checking

Automatic fact-checking plays a crucial role in combating the spread of ...
research
06/07/2019

FAKTA: An Automatic End-to-End Fact Checking System

We present FAKTA which is a unified framework that integrates various co...
research
06/02/2021

A Multi-Level Attention Model for Evidence-Based Fact Checking

Evidence-based fact checking aims to verify the truthfulness of a claim ...
research
02/15/2018

Truth Validation with Evidence

In the modern era, abundant information is easily accessible from variou...
research
03/25/2016

Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning

Most successful information extraction systems operate with access to a ...

Please sign up or login with your details

Forgot password? Click here to reset