In the modern world, social media is playing its part in several ways, for instance in news dissemination and information sharing, social media outlets, such as Twitter, Facebook, and Instagram, have been proved very effective (said2019natural; imran2016twitter; liu2015events; ahmad2019social). However, it also comes with several challenges, such as collecting information from several sources, detecting and filtering misinformation (gangireddy2020unsupervised; han2020graph; yang2019unsupervised). Similar to other events and pandemics, being one of the deadly pandemics in the history, COVID-19 has been the subject of discussion over social media since its emergence. Without any surprise, a lot of misinformation about the pandemic are circulated over social networks. In order to identify misinformation spreaders and filter fake news about COVID-19 and 5G conspiracy, a task namely "FakeNews: Corona Virus and 5G Conspiracy Multimedia Twitter-Data-Based Analysis" has been proposed in the benchmark MediaEval 2020 competition (pogorelov2020fakenews).
This paper provides a detailed description of the methods proposed by team DCSE_UETP for the fake news detection task. The task consists of two parts, namely (i) text-based misinformation detection (TMD), and (ii) structure-based misinformation detection (SMD). The first task (TMD) is based on textual analysis of COVID-19 related information shared on Twitter during January 2020 and 15th of July 2020, and aims to detect different types of conspiracy theories about COVID-19 and its vaccines, such as that "the 5G weakens the immune system and thus caused the current corona-virus pandemic etc., (pogorelov2020fakenews). In the SMD task, the participants are provided with a set of graphs, each representing a sub-graph of Twitter, and corresponds to a single tweet where the vertices of the graphs represent accounts. Similar to TMD, in this task, the participants need to detect and differentiate between 5G and other COVID-19 conspiracy theories.
2. Proposed Approach
2.1. Methodology for TMD Task
For the text-based analysis, we employed two different methods including a (i) Bag of Words (BoW), and a (ii) BERT model-based solution (devlin2018bert). Before proceeding with the proposed methods, it is to be noted that the dataset provided for the text-based analysis is not balanced where one of the classes namely non-conspiracy contains a very high number of samples while the rest are composed of relatively fewer samples. In total, the majority class contains 4412, while the other two classes, namely 5G conspiracies, and other conspiracies, are composed of 1263 and 785 samples, respectively. In order to balance the dataset, we rely on an ensemble of different re-sampled datasets, where models are built/trained by dividing the class with a higher number of samples into n-differing parts as illustrated in Figure 1. After training
models, the results of the models are combined using two different late fusion methods including a majority voting method, and summation of the posterior probabilities. In the majority voting, since we have four models, in the case of tie we consider the accumulative probabilities/scores to assign a label to a test sample.
Before deploying BoW and BERT, text has been cleaned by removing punctuation’s keys, such as commas, full-stops, emojis, URLs, and stop words. Once the text is pre-processed, we proceed with the tokenization and creation of BoW vocabulary, which is followed by generation of the feature vector for each sentence. A Naives Bayes classifier is then trained on the extracted features. On the other hand, a logistic regression model is trained on word embeddings generated via BERT.
2.2. Methodology for SMD Task
Graphs representation learning using Graph Neural Networks (GNNs) have been shown to be effective in various domains such as social networks, biological networks, and financial networks. GNNs aggregate the neighborhood representation within k hops and then apply a pooling such as SUM, MEAN, MAX to obtain the final representation of the node. Furthermore, GNN’s can be used to learn the representation of a simple graph structures (DBLP:journals/corr/abs-1806-08804; DBLP:journals/corr/abs-1810-00826; cangea2018sparse)
, which then can be used to classify the graphs. For graph classification, these methods learn the representation of nodes, followed by graphREADOUT method, which is aggregating the node features obtained after the final iteration of GNN.
We model this problem as a graph classification task. Following Keyule et al.(DBLP:journals/corr/abs-1806-08804), we train our model using three classes of the graphs 5G Conspiracy, non-conspiracy, other-conspiracy, and learn the representation of the graphs.
3. Results and Analysis
3.1. Runs Description in TMD Task
For TMD, we submitted six different runs mainly relying on two approaches, namely BERT and BoW, under two late fusion schemes. Three of the runs are based on binary classification while the three deal the task as ternary classification problem. It is to be noted that the fusion schemes are used to combine the scores/output of the four individual models trained as result of the data balancing method as described earlier.
The first three runs are based on the ternary classification task, where run 1 and run 2 are based on BoW with majority voting and accumulative classification scores of the individual models. The third and final ternary run is based on BERT features, where a logistic regression model is trained on word embeddings generated by BERT. As can be seen in Table 1(a), overall, better results are obtained with BoW approach under the majority voting scheme.
The last three runs are based on the binary classification task, where the first two (i.e., Run 4 and Run 5) are based on BoW with majority voting and accumulative classification based fusion methods while the final one (i.e., Run 6) is based on BERT with accumulative score based fusion scheme. As expected, the performance of all the methods is significantly higher on the binary classification task compared to ternary classification task.
Similar trend has been also observed on the test set, where overall better results are obtained with BoW under majority voting scheme.
3.2. Runs Description in SMD Task
For training the model, we divide the dataset into train/valid/valid (80/10/10). We used the grid search to obtain the best hyperparameters. The model has four MLP layers, and useMAX and MEAN
operations for neighbor pooling and graph pooling respectively. The model is trained on 1000 epochs with a learning rate of 0.01, and dropout 0.3 is applied on the final layer output. The final embedding size is 128. We evaluate our model on AUC-ROC and the result of the test set is given in Table 1(b). The results show that the model has discriminative power to learn to classify the graph structures. Furthermore, it shows that the diffusion of information depending on the type of information being spread forms a diffusion pattern.
4. Conclusions and Future Work
The challenge is composed of two tasks, one aiming to analyze and detect COVID-19 related fake news using tweets’ text while the other aims to analyze network structure for the possible detection of the fake news. For the first task, we mainly relied on two state-of-the-art methods namely BoW and BERT embeddings under different fusion schemes. Overall better results are obtained with BoW under the majority voting scheme. For the SMD task, we rely on GNNs to differentiate among different conspiracy theories on COVID-19. In the current implementations, both textual and structural information are used independently, in the future we aim to enrich the structural information with the textual information for better detection of fake news.