Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection

09/21/2018
by   Lev Konstantinovskiy, et al.
0

In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This paper is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics and annotators than previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5 methods ClaimBuster and ClaimRank. The system was deployed in production and received positive user feedback.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/02/2022

The ParlaSent-BCS dataset of sentiment-annotated parliamentary debates from Bosnia-Herzegovina, Croatia, and Serbia

Expression of sentiment in parliamentary debates is deemed to be signifi...
research
07/01/2019

Claim Extraction in Biomedical Publications using Deep Discourse Model and Transfer Learning

Claims are a fundamental unit of scientific discourse. The exponential g...
research
01/28/2021

LESA: Linguistic Encapsulation and Semantic Amalgamation Based Generalised Claim Detection from Online Content

The conceptualization of a claim lies at the core of argument mining. Th...
research
12/16/2021

NewsClaims: A New Benchmark for Claim Detection from News with Background Knowledge

Claim detection and verification are crucial for news understanding and ...
research
08/20/2020

Checkworthiness in Automatic Claim Detection Models: Definitions and Analysis of Datasets

Public, professional and academic interest in automated fact-checking ha...
research
09/30/2019

Automatic Fact-guided Sentence Modification

Online encyclopediae like Wikipedia contain large amounts of text that n...
research
05/04/2022

A Framework to Generate High-Quality Datapoints for Multiple Novel Intent Detection

Systems like Voice-command based conversational agents are characterized...

Please sign up or login with your details

Forgot password? Click here to reset