Stance Detection Benchmark: How Robust Is Your Stance Detection?

01/06/2020
by   Benjamin Schiller, et al.
0

Stance Detection (StD) aims to detect an author's stance towards a certain topic or claim and has become a key component in applications like fake news detection, claim validation, and argument search. However, while stance is easily detected by humans, machine learning models are clearly falling short of this task. Given the major differences in dataset sizes and framing of StD (e.g. number of classes and inputs), we introduce a StD benchmark that learns from ten StD datasets of various domains in a multi-dataset learning (MDL) setting, as well as from related tasks via transfer learning. Within this benchmark setup, we are able to present new state-of-the-art results on five of the datasets. Yet, the models still perform well below human capabilities and even simple adversarial attacks severely hurt the performance of MDL models. Deeper investigation into this phenomenon suggests the existence of biases inherited from multiple datasets by design. Our analysis emphasizes the need of focus on robustness and de-biasing strategies in multi-task learning approaches. The benchmark dataset and code is made available.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/04/2022

MDFEND: Multi-domain Fake News Detection

Fake news spread widely on social media in various domains, which lead t...
research
10/11/2020

Connecting the Dots Between Fact Verification and Fake News Detection

Fact verification models have enjoyed a fast advancement in the last two...
research
05/12/2019

A Benchmark Study on Machine Learning Methods for Fake News Detection

The proliferation of fake news and its propagation on social media have ...
research
11/22/2022

An Emotion-Aware Multi-Task Approach to Fake News and Rumour Detection using Transfer Learning

Social networking sites, blogs, and online articles are instant sources ...
research
09/04/2019

Different Absorption from the Same Sharing: Sifted Multi-task Learning for Fake News Detection

Recently, neural networks based on multi-task learning have achieved pro...
research
09/05/2018

Stance Prediction for Russian: Data and Analysis

Stance detection is a critical component of rumour and fake news identif...
research
07/24/2019

WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale

The Winograd Schema Challenge (WSC), proposed by Levesque et al. (2011) ...

Please sign up or login with your details

Forgot password? Click here to reset