On the Importance of Delexicalization for Fact Verification

by   Sandeep Suntwal, et al.

In this work we aim to understand and estimate the importance that a neural network assigns to various aspects of the data while learning and making predictions. Here we focus on the recognizing textual entailment (RTE) task and its application to fact verification. In this context, the contributions of this work are as follows. We investigate the attention weights a state of the art RTE method assigns to input tokens in the RTE component of fact verification systems, and confirm that most of the weight is assigned to POS tags of nouns (e.g., NN, NNP etc.) or their phrases. To verify that these lexicalized models transfer poorly, we implement a domain transfer experiment where a RTE component is trained on the FEVER data, and tested on the Fake News Challenge (FNC) dataset. As expected, even though this method achieves high accuracy when evaluated in the same domain, the performance in the target domain is poor, marginally above chance.To mitigate this dependence on lexicalized information, we experiment with several strategies for masking out names by replacing them with their semantic category, coupled with a unique identifier to mark that the same or new entities are referenced between claim and evidence. The results show that, while the performance on the FEVER dataset remains at par with that of the model trained on lexicalized data, it improves significantly when tested in the FNC dataset. Thus our experiments demonstrate that our strategy is successful in mitigating the dependency on lexical information.


page 1

page 2

page 3


Connecting the Dots Between Fact Verification and Fake News Detection

Fact verification models have enjoyed a fast advancement in the last two...

Beyond Fact Verification: Comparing and Contrasting Claims on Contentious Topics

As the importance of identifying misinformation is increasing, many rese...

Joint Verification and Reranking for Open Fact Checking Over Tables

Structured information is an important knowledge source for automatic ve...

Mitigation of Diachronic Bias in Fake News Detection Dataset

Fake news causes significant damage to society.To deal with these fake n...

The use of Data Augmentation as a technique for improving neural network accuracy in detecting fake news about COVID-19

This paper aims to present how the application of Natural Language Proce...

Logically at Factify 2022: Multimodal Fact Verification

This paper describes our participant system for the multi-modal fact ver...

Integrating Pattern- and Fact-based Fake News Detection via Model Preference Learning

To defend against fake news, researchers have developed various methods ...

Please sign up or login with your details

Forgot password? Click here to reset