MythQA: Query-Based Large-Scale Check-Worthy Claim Detection through Multi-Answer Open-Domain Question Answering

07/21/2023
by   Yang Bai, et al.
0

Check-worthy claim detection aims at providing plausible misinformation to downstream fact-checking systems or human experts to check. This is a crucial step toward accelerating the fact-checking process. Many efforts have been put into how to identify check-worthy claims from a small scale of pre-collected claims, but how to efficiently detect check-worthy claims directly from a large-scale information source, such as Twitter, remains underexplored. To fill this gap, we introduce MythQA, a new multi-answer open-domain question answering(QA) task that involves contradictory stance mining for query-based large-scale check-worthy claim detection. The idea behind this is that contradictory claims are a strong indicator of misinformation that merits scrutiny by the appropriate authorities. To study this task, we construct TweetMythQA, an evaluation dataset containing 522 factoid multi-answer questions based on controversial topics. Each question is annotated with multiple answers. Moreover, we collect relevant tweets for each distinct answer, then classify them into three categories: "Supporting", "Refuting", and "Neutral". In total, we annotated 5.3K tweets. Contradictory evidence is collected for all answers in the dataset. Finally, we present a baseline system for MythQA and evaluate existing NLP models for each system component using the TweetMythQA dataset. We provide initial benchmarks and identify key challenges for future models to improve upon. Code and data are available at: https://github.com/TonyBY/Myth-QA

READ FULL TEXT
research
04/22/2020

AmbigQA: Answering Ambiguous Open-domain Questions

Ambiguity is inherent to open-domain question answering; especially when...
research
05/07/2023

FACTIFY-5WQA: 5W Aspect-based Fact Verification through Question Answering

Automatic fact verification has received significant attention recently....
research
11/10/2020

Generating Fact Checking Briefs

Fact checking at scale is difficult – while the number of active fact ch...
research
05/22/2023

AVeriTeC: A dataset for real-world claim verification with evidence from the web

Existing datasets for automated fact-checking have substantial limitatio...
research
12/19/2022

Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments

We present a human-in-the-loop evaluation framework for fact-checking no...
research
10/22/2022

Varifocal Question Generation for Fact-checking

Fact-checking requires retrieving evidence related to a claim under inve...
research
08/04/2019

Automatic Fact-Checking Using Context and Discourse Information

We study the problem of automatic fact-checking, paying special attentio...

Please sign up or login with your details

Forgot password? Click here to reset