Fighting false rumours at the internet is a tedious task. Sometimes, even understanding what an actual rumour is about may prove challenging. And only then one can actually judge its veracity with an appropriate evidence. The works of Ferreira and Vlachos (2016); Enayet and El-Beltagy (2017) focused on prediction of rumour veracity in thread discussions. These works indicated that the veracity is correlated with stances of the discussion participants towards the rumour. Following this assumption, the participants of the SubTask A in the SemEval competition Task 7 were asked to classify whether the stance of each post in a given Twitter or Reddit thread supports, denies, queries or comments hidden rumour. Potential applications of such a function are wide, ranging from an analysis of popular events (political discussions, academy awards, etc.) to quickly disproving fake news during disasters.
Stance classification (SC) in its traditional form is concerned with determining the attitude of a source text towards a target text Mohammad et al. (2016) and it has been studied thoroughly for discussion threads Walker et al. (2012); Hasan and Ng (2013); Chuang and Hsieh (2015). However, the objective of SemEval 2019 Task 7 is to determine the stance to hidden rumour which is not explicitly given (it can be often inferred from the source post of the discussion – the root of the tree-shaped discussion thread – as demonstrated in Figure 1). The competitors were asked to classify the stance of the source post itself too.
The approach followed in our work builds on recent advances in language representation models. We fine-tune the pre-trained end-to-end Bidirectional Encoder Representations from Transformers (BERT) model Devlin et al. (2018), while using the discussion’s source post, target’s previous post and the target post itself as inputs to determine the rumour stance of the target post. Our implementation is available online.111www.github.com/MFajcik/RumourEval2019
2 Related Work
Previous SemEval competitions: In recent years, there were two SemEval competitions targeting the stance classification. The first one focused on the setting in which the actual rumour was provided Mohammad et al. (2016)
. The organizers of SemEval-2016 Task 6 prepared a benchmarking system based on SVM using hand-made features and word embeddings from their previous system for sentiment analysisMohammad et al. (2013), outperfoming all the challenge participants.
The second was previous RumourEval competition won by a system based on word vectors, handcrafted features222The features included: a flag indicating whether a tweet is a source tweet of a conversation, the length of the tweet, an indicator of the presence of urls and images, punctuation, cosine distance to source tweet and all other tweets in the conversation, the count of negation and swear words, and an average of word vectors corresponding to the tweet. and an LSTM Hochreiter and Schmidhuber (1997) summarizing information of the discussion’s branches Kochkina et al. (2017). Other submissions were either based on similar handcrafted features Singh et al. (2017); Wang et al. (2017); Enayet and El-Beltagy (2017), features based on sets of words for determining language cues such as Belief or Denial Bahuleyan and Vechtomova (2017)
, post-processing via rule-based heuristics after the feature-based classificationSrivastava et al. (2017)
, Convolutional Neural Networks (CNNs) with rulesLozano et al. (2017), or end-to-end CNNs that jointly learnt word embeddings Chen et al. (2017).
End-to-End approaches: Augenstein et al. (2016) encodes the target text by means of a bidirectional LSTM (BiLSTM), conditioned on the source text and empirically shows that the conditioning on the source text matters. Du et al. (2017) proposes target augmented embeddings – embeddings concatenated with an average of the source text embeddings and applies these to compute an attention based on the weighted sum of the target embeddings that were previously transformed via the BiLSTM. Mohtarami et al. (2018) proposes an architecture that encodes the source and the target text via a LSTM and a CNN separately and then uses a memory network together with a similarity matrix to capture the similarity between the source and the target text, and infers a fixed-size vector suitable for the stance prediction.
Provided dataset was collected from Twitter and Reddit tree-shaped discussions. The stance labels were obtained via crowdsourcing. The Twitter discussions are based on recent popular topics – Sydney siege, Germanwings crash etc. and there are 9 total topics covered in the training data. The Twitter part of test data contains different topics. The Reddit discussions cover various topics and the discussions are in most cases not related to each other. We provide a deeper insight at dataset in Appendix A.1.
4 BUT-FIT’s System Description
token to terminate each sentence. Then we use tokenizer that comes with Hugging Face pytorch re-implementation of BERT555https://github.com/huggingface/pytorch-pretrained-BERT. The tokenizer lowercases the input and applies the WordPiece encoding Wu et al. (2016)
to split input words into most frequent n-grams present in the pre-training corpus, effectively representing text at the sub-word level while keeping only 30,000 token vocabulary.
Following the recent trend in transfer learning from language models (LM), we employ the pre-trained BERT model. The model is first trained on the concatenation of BooksCorpus (800M words)Zhu et al. (2015) and English Wikipedia (2,500M words) using the multi-task objective consisting of LM and machine comprehension (MC) sub-objectives. The LM objective aims at predicting the identity of 15% randomly masked tokens present at the input666The explanation of token masking is simplified and we refer readers to read details in the original paper Devlin et al. (2018).. Given two sentences from the corpus, the MC objective is to classify whether the second sentence follows the first sentence in the corpus. The sentence is replaced randomly in half of the cases. During the pre-training, the input consists of two documents, each represented as a sequence of tokens divided by special token and preceeded by token used by the MC objective, i.e. . The input tokens are represented via jointly learned token embeddings , segment embeddings capturing whether the word belongs into or and positional embeddings since self-attention is position-invariant operation (see Devlin et al. (2018) for details).
In our solution, we follow the assumption that the stance of the discussion’s post depends only on itself, the source thread post and previous thread post. Since the original input is composed of two documents, we experimented with various ways of encoding the input (Section 6) ending up with just a concatenation of source and previous post as (left empty in case of source post being the target post) and target post as . The discriminative fine-tuning of BERT is done using the
-token level output and passing it via two dense layers yielding the posterior probabilities as depicted in Figure2. Weighted cross-entropy loss is used to ensure the flat prior over the classes.
Before submission we trained 100 models, which differed just by learning rate. We experimented with 4 different system fusions in order to increase F1 measure and compensate for overfitting:
TOP-N fusion chose 1 model randomly to add into the ensemble, then randomly shuffled the rest and tried adding them into ensemble one at the time, while iteratively calculating ensemble’s F1 by averaging the output probabilities to approximate the bayesian model averaging. If adding model into ensemble increased the F1, model has been permanently included in the ensemble. The process has been repeated until no further model improving the ensemble’s F1 has been found. This resulted into set of 17 best models.
EXC-N fusion chose all models into the ensemble and then iterativly dropped one model at the time s.t. dropping it resulted in the largest increase of the ensemble’s F1, stopping when dropping any ensemble’s model did not increased the F1. Using this approach, we ended up using 94 models.
TOP-N is analogous to TOP-N fusion, but we average the pre-softmax scores instead of output class probabilities.
OPT-F1 fusion aims at learning weights summing up to 1 for weighted average of the output probabilities from models selected via the procedure used in TOP-N
. The weights are estimated using modified Powell’s method from SciPy to maximize the F1 score on dev data.
|#||Acc||macro F1||macro F1||F1||F1||F1||F1|
Our achieved results. Results for single model were obtained by training at least 10 models and we report mean and standard deviation for these. #denotes the number of parameters. The columns F1 through F1 contain individual F1 scores for problem classes. All ensemble models are optimized for F1-score on dev data. BiLSTM+SelfAtt contains 4.2M parameters without pre-trained BERT embeddings. BERT and BERT denote ablations with empty source or target post respectively. Note that the accuracy is biased towards different training data prior as shown in Table 1. Our SemEval submissions are denoted with . Winning BLCU-nlp system achieved 61.87 F1 score on test data. More available at http://tinyurl.com/y3m5mskd.
5 Experimental Setup
We implemented our models in pytorch, where we use Hugging Face re-implementation (Footnote 5) in ”bert-large-uncased” setting pre-trained with transformer layers, hidden unit size of , attention heads and parameters. When building an ensemble, we picked the learning rates from the interval
. Each epoch, we iterate over dataset in an ordered manner, starting with shortest sequence as we found this to be helpful. We truncate sequences at maximum lengthwith a heuristic – firstly we truncate the to length , if that is not enough, then we truncate the to the same size. We kept batch size at
and keep other hyperparameters the same as in BERT paper. We use the same Adam optimizer with L2 weight decay ofand no warmup. We trained the model on GeForce RTX 2080 Ti GPU.
6 Results and Analysis
We compare our solution with three baselines. The first is branch-LSTM baseline provided by the task organizers777http://tinyurl.com/y4p5ygn7 – inspired by the winning system of RumourEval 2017. The second baseline (FeaturesNN
) is our re-implementation of first baseline in pytorch without LSTM – posts are classified via 2 layer network (ReLU/Softmax) only by features named in Footnote2. In the third case (BiLSTM+SelfAtt
), we used the same input representation as our submitted model, but replaced BERT with 1-layer BiLSTM followed by self-attention and a softmax layer as proposed byLin et al. (2017), except the orthogonality constraint is not used as we did not found it helpful.
The results are shown in Table 2
. Our BERT models encountered high variance of the results during the training. We assume the cause of this might be the problem difficulty, small training set and the model complexity. To counteract, we decided to discard all the models with less than 55 F1 score on dev data and we averaged the output class probability distributions when ensembling. Our initial experiments used sequences up to length 512, but we found no difference when truncating them down to 200.
What features weren’t helpful: We tried adding a number of features to the pooled output (after dense/tanh layer) including positive, neutral and negative sentiment and all the features used by FeaturesNN baseline. We also tried adding jointly learned POS, NER and dependency tag embeddings as well as third segment embedding888We tried adding the learned representations to the input the same way the segment/positional embeddings are added. or explicit token to separate source and previous post in BERT’s input without observing any improvement.
Our approach achieves 61.67 macro F1 score improving over baseline by 12.37%, while using only discussion’s source post, previous post and the target post to classify the target post’s stance to rumour. In our case study, we noticed that few examples are not answerable by human while using only these information sources. Therefore, in future we would like to extend our system with relevance scoring system, scoring the all discussion’s posts and picking up the most relevant ones to preserve the context of understanding.
This work was supported by [Acknowledgments will be filled upon acceptance.]
- Augenstein et al. (2016) Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. arXiv preprint arXiv:1606.05464.
- Bahuleyan and Vechtomova (2017) Hareesh Bahuleyan and Olga Vechtomova. 2017. Uwaterloo at semeval-2017 task 8: Detecting stance towards rumours with topic independent features. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 461–464.
- Chen et al. (2017) Yi-Chin Chen, Zhao-Yang Liu, and Hung-Yu Kao. 2017. Ikm at semeval-2017 task 8: Convolutional neural networks for stance detection and rumor verification. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 465–469.
- Chuang and Hsieh (2015) Ju-han Chuang and Shukai Hsieh. 2015. Stance classification on ptt comments. In 29th Pacific Asia Conference on Language, Information and Computation Proceedings of PACLIC 2015: Poster Papers, page 27.
- Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Du et al. (2017)
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017.
Stance classification with target-specific neural attention networks.
International Joint Conferences on Artificial Intelligence.
- Enayet and El-Beltagy (2017) Omar Enayet and Samhaa R El-Beltagy. 2017. Niletmrg at semeval-2017 task 8: Determining rumour and veracity support for rumours on twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 470–474.
- Ferreira and Vlachos (2016) William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1163–1168.
- Gorrell et al. (2019) Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2019. SemEval-2019 Task 7: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of SemEval. ACL.
Hasan and Ng (2013)
Kazi Saidul Hasan and Vincent Ng. 2013.
Stance classification of ideological debates: Data, models, features,
Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1348–1356.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
- Kochkina et al. (2017) Elena Kochkina, Maria Liakata, and Isabelle Augenstein. 2017. Turing at semeval-2017 task 8: Sequential approach to rumour stance classification with branch-lstm. arXiv preprint arXiv:1704.07221.
- Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
- Lozano et al. (2017) Marianela García Lozano, Hanna Lilja, Edward Tjörnhammar, and Maja Karasalo. 2017. Mama edha at semeval-2017 task 8: Stance classification with cnn and rules. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 481–485.
- Mohammad et al. (2016) Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41.
- Mohammad et al. (2013) Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013), Atlanta, Georgia, USA.
- Mohtarami et al. (2018) Mitra Mohtarami, Ramy Baly, James Glass, Preslav Nakov, Lluís Màrquez, and Alessandro Moschitti. 2018. Automatic stance detection using end-to-end memory networks. arXiv preprint arXiv:1804.07581.
- Singh et al. (2017) Vikram Singh, Sunny Narayan, Md Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharyya. 2017. Iitp at semeval-2017 task 8: A supervised approach for rumour evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 497–501.
- Srivastava et al. (2017) Ankit Srivastava, Georg Rehm, and Julian Moreno Schneider. 2017. Dfki-dkt at semeval-2017 task 8: Rumour detection and classification using cascading heuristics. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 486–490.
- Walker et al. (2012) Marilyn A Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 592–596. Association for Computational Linguistics.
- Wang et al. (2017) Feixiang Wang, Man Lan, and Yuanbin Wu. 2017. Ecnu at semeval-2017 task 8: Rumour evaluation using effective features and supervised ensemble models. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 491–496.
- Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Zhu et al. (2015)
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun,
Antonio Torralba, and Sanja Fidler. 2015.
Aligning books and movies: Towards story-like visual explanations by
watching movies and reading books.
Proceedings of the IEEE international conference on computer vision, pages 19–27.
Appendix A Supplemental Material
a.1 Dataset Insights
For each discussion from Twitter and Reddit, the dataset contains its whole tree structure and metadata, which are different for both sites (e.g. upvotes in Reddit). When analyzing the data, we also uncovered a few anomalies: 12 data points to do not contain any text and according to organizers they were deleted by users at the time of download and been left it in place so as not to break the conversational structure, the query stance of few examples taken from subreddit DebunkThis999https://www.reddit.com/r/DebunkThis/ is strictly dependent on domain knowledge and the strict class of some examples is amibgious and they should probably be labelled with multiple classes.
a.1.1 Domain knowledge dependency
Examples from subreddit DebunkThis have all the same format ”Debunk this: [statement]”, e.g. ”Debunk this: Nicotine isn’t really bad for you, and it’s the other substances that makes tobacco so harmful.”. All these examples are labelled as queries.
a.1.2 Class ambiguity
For instance source/previous post ”This is crazy! #CapeTown #capestorm #weatherforecast https://t.co/3bcKOKrCJB” and target post ”@RyGuySA Oh my gosh! Is that not a tornado?! Cause wow, It almost looks like one!”, officialy labelled in the test data as a comment, but we believe it might be a query as well.
a.2 Additional Introspection
The following figures 3, 4, 5, 6 contain selected insights at the attention matrices from multi-head attention defined as (1), where are matrices containing query/value vectors and is the key/value dimension. The insights are selected from the heads at the first layer of transformer encoder.