Probing Spurious Correlations in Popular Event-Based Rumor Detection Benchmarks

09/19/2022
by   Jiaying Wu, et al.
0

As social media becomes a hotbed for the spread of misinformation, the crucial task of rumor detection has witnessed promising advances fostered by open-source benchmark datasets. Despite being widely used, we find that these datasets suffer from spurious correlations, which are ignored by existing studies and lead to severe overestimation of existing rumor detection performance. The spurious correlations stem from three causes: (1) event-based data collection and labeling schemes assign the same veracity label to multiple highly similar posts from the same underlying event; (2) merging multiple data sources spuriously relates source identities to veracity labels; and (3) labeling bias. In this paper, we closely investigate three of the most popular rumor detection benchmark datasets (i.e., Twitter15, Twitter16 and PHEME), and propose event-separated rumor detection as a solution to eliminate spurious cues. Under the event-separated setting, we observe that the accuracy of existing state-of-the-art models drops significantly by over 40 comparable to a simple neural classifier. To better address this task, we propose Publisher Style Aggregation (PSA), a generalizable approach that aggregates publisher posting records to learn writing style and veracity stance. Extensive experiments demonstrate that our method outperforms existing baselines in terms of effectiveness, efficiency and generalizability.

READ FULL TEXT
research
03/13/2019

Sub-event detection from Twitter streams as a sequence labeling problem

This paper introduces improved methods for sub-event detection in social...
research
10/02/2019

Graph-Hist: Graph Classification from Latent Feature Histograms With Application to Bot Detection

Neural networks are increasingly used for graph classification in a vari...
research
05/27/2023

Zero- and Few-Shot Event Detection via Prompt-Based Meta Learning

With emerging online topics as a source for numerous new events, detecti...
research
02/01/2023

Learning from Stochastic Labels

Annotating multi-class instances is a crucial task in the field of machi...
research
11/01/2022

Semantic Pivoting Model for Effective Event Detection

Event Detection, which aims to identify and classify mentions of event i...
research
04/16/2022

Learning to Classify Open Intent via Soft Labeling and Manifold Mixup

Open intent classification is a practical yet challenging task in dialog...
research
04/27/2023

Rumor Detection with Hierarchical Representation on Bipartite Adhoc Event Trees

The rapid growth of social media has caused tremendous effects on inform...

Please sign up or login with your details

Forgot password? Click here to reset