Stance Prediction for Contemporary Issues: Data and Experiments

We investigate whether pre-trained bidirectional transformers with sentiment and emotion information improve stance detection in long discussions of contemporary issues. As a part of this work, we create a novel stance detection dataset covering 419 different controversial issues and their related pros and cons collected by procon.org in nonpartisan format. Experimental results show that a shallow recurrent neural network with sentiment or emotion information can reach competitive results compared to fine-tuned BERT with 20x fewer parameters. We also use a simple approach that explains which input phrases contribute to stance detection.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

09/17/2017

Group Affect Prediction Using Emotion Heatmaps and Scene Information

In this paper, we describe our work on emotion detection for a group of ...
11/15/2021

Improving Prosody for Unseen Texts in Speech Synthesis by Utilizing Linguistic Information and Noisy Data

Recent advancements in end-to-end speech synthesis have made it possible...
02/21/2019

ntuer at SemEval-2019 Task 3: Emotion Classification with Word and Sentence Representations in RCNN

In this paper we present our model on the task of emotion detection in t...
04/05/2019

Exploring Fine-Tuned Embeddings that Model Intensifiers for Emotion Analysis

Adjective phrases like "a little bit surprised", "completely shocked", o...
09/09/2016

INSIGHT-1 at SemEval-2016 Task 4: Convolutional Neural Networks for Sentiment Classification and Quantification

This paper describes our deep learning-based approach to sentiment analy...
03/06/2020

S-APIR: News-based Business Sentiment Index

This paper describes our work on developing a new business sentiment ind...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Stance detection identifies whether an opinion is in favor of an idea or opposes it. It has a tight connection with sentiment analysis; however, stance detection usually investigates the two-sided relationship between an opinion and a question. For example, ‘should abortion be legal?’ or ‘is human activity primarily responsible for global climate change?’

Contemporary debatable issues, even though non-political, usually carry some political weight and controversy. For example, legislators may allow soda vending machines in our school or consider obesity as a health issue that directly impacts soda manufacturers and insurance companies respectively. On a larger scale, an issue such as climate change is being discussed in US presidential debates constantly. Meanwhile, information about these issues is mostly one-sided and provided by left or right partisan resources. Such information forms public beliefs, has persuasive power, and promotes confirmation bias Stanojevic et al. (2019), the humans’ tendency to search for the information which confirms their existing beliefs 111www.procon.org/education.php. Confirmation bias permits internet debates and promote discrimination, misinformation, and hate speech, all of which are emerging problems in user posts of social media platforms.

Although there are many attempts to automatic identification and removal of such contents from online platforms, the need for accessing bi-partisan information that cultivates critical thinking and avoids confirmation bias remains. In this regard, a few web sources, such as procon.org, present information in a non-partisan format and being used as a resource for improving critical thinking in educational training by teachers 222https://www.procon.org/view.background-resource.php?resourceID=004241.

Here, we aim to improve such resources by automatic stance detection of pro or con-perspectives regarding a debatable issue. We extend our previous work Hosseinia et al. (2019) by creating a new dataset from procon.org with distinct issues and their two-sided perspectives annotated by its experts 333https://github.com/marjanhs/procon20/. Then, we leverage external knowledge to identify the stance of a perspective towards an issue that is mainly represented in the form of a question.

The latest progress in pre-trained language models Howard and Ruder (2018) and transformers Devlin et al. (2019); Yang et al. (2019) allows one to create general models with less amount of effort for task-specific text classification. In this work, we show that bidirectional transformers can produce competitive results even without fine-tuning by leveraging auxiliary sentiment and emotion information Dragut et al. (2010). Experimental results show the effectiveness of our model and its remarkable performance. The model has a significantly smaller size compared to the BERT-base model.

The main contributions of this work are as following:

  • Proposing a simple but efficient recurrent neural network that leverages sentence-wise sentiment or token-level emotion of input sequence with BERT representation for detecting the stance of a long perspective against its related question.

  • Creating a novel dataset for stance detection with more than K instances.

  • Explaining the word/phrase contribution of input sequence using max-pooling engagement score for stance detection.

2 Related Works

We group stance detection methods based on underlying data and approaches as follows:

  • Tweets are collected from SemEval 2016, Task 6, Mohammad et al. (2016) and organized in two categories. The first category, which represents a supervised setting, includes tweets that cover opinions about five topics, “Atheism”, “Climate Change”, “Feminist Movement”, “Hillary Clinton”, and “Legalization of Abortion”. The second category, which represents weakly supervised settings, includes tweets that cover one topic, but the training data is unlabeled.

  • Claims are obtained from Wikipedia in Bar-Haim et al. (2017). Each claim is defined as a brief statement that is often part of a Wikipedia sentence. The claim dataset contains different topics.

  • Debates are gathered from various online debate resources, including idebate, debatewise and procon in the form of perspective, claim, and evidence for substantiated perspective discovery. 49 out of its 947 claims are from procon Chen et al. (2019). Claims and perspectives are short sentences and have been used for stance detection in Popat et al. (2019).

Current approaches on stance detection use different types of linguistic features, including word/character n-grams, dependency parse trees, and lexicons

Sun et al. (2018); Sridhar et al. (2015); Hasan and Ng (2013); Walker et al. (2012). There are also end-to-end neural network approaches that learn topics and opinions independently while joining them with memory networks Mohtarami et al. (2018), bidirectional conditional LSTM Augenstein et al. (2016), or neural attention Du et al. (2017). There are also some neural network approaches that leverage lexical features Riedel et al. (2017); Hanselowski et al. (2018). A consistency constraint is proposed to jointly model the topic and opinion using BERT architecture Popat et al. (2019). It trains the whole massive network for label prediction. None of these approaches incorporate bidirectional transformers with sentiment and emotion in a shallow neural network as we propose in this paper. Additionally, our focus is to find the stance of 100-200 words long discussions, which are commonly present in nonpartisan format.

3 Dataset

We collect data from procon.org, a non-profit organization that presents opinions on controversial issues in a nonpartisan format. Issues (questions) and their related responses are professionally researched from different online platforms by its experts. The dataset covers different detailed issues ranging from politics to sport and healthcare. The dataset instances are pairs of issues, in the form of questions, and their corresponding perspectives from proponents and opponents. Each perspective is either a pro or a con with 100-200 words that supports its claim with compelling arguments. Table 1 provides some examples of the questions from the dataset. The dataset statistics are also presented in Table 2. We may use the words opinion and perspective interchangeably as both refer to the same concept in this work.

HEALTH and MEDICINE
1- Should euthanasia or physician-assisted
suicide be legal?
2- Is vaping with e-cigarettes safe?
EDUCATION
1-Should parents or other adults be able to
ban books from schools and libraries?
2- Should public college be tuition-free?
POLITICS
1- Should recreational marijuana be legal?
2- Should more gun control laws be enacted?
SCIENCE and TECHNOLOGY
1- Is cell phone radiation safe?
2- Should net neutrality be restored?
ENTERTAINMENT and SPORTS
1- Are social networking sites good for
our society?
2- Do violent video games contribute to
youth violence?
Table 1: Procon dataset questions
Figure 1: Stance detection architecture; snt: sentiment; em: emotion
Set #of Topics #of Words #of Pro-perspectives #of Con-perspectives Total
train 417 127 2,140 2,125 4,265
dev 265 125 326 284 610
test 336 123 613 606 1,219
Table 2: Procon dataset statistics

4 Model

Utilizing pre-trained models has been widely popular in machine translation and various text classification tasks. Prior efforts were hindered by the lack of labeled data Zhang et al. (2019). With the growth of successful pre-trained models, a model fine-tuned on a small portion of data can compete with models trained on 10 more training data without pre-training Howard and Ruder (2018). Recently, transformer models trained on both directions of language simultaneously, such as BERT Devlin et al. (2019) and XLNet Yang et al. (2019), overcome previous unidirectional language models (e.g. ULMFiT Howard and Ruder (2018)) or models trained on two independent directions (ELMo) Peters et al. (2018) significantly. So, we build our baselines based on BERT architecture in two different ways: single and pair of inputs. A question and its related opinion are concatenated for single inputs. However, for input pairs, the question and the opinion are being separated with the BERT separator tag [SEP]. This approach has been used for question-answering applications Devlin et al. (2019).

Opinion is connected with sentiment and emotion Schneider and Dragut (2015)

. Moreover, prior efforts show the successful employment of linguistic features, extracted with external tools, in neural networks for emotional cognition

Yang et al. (2017). So, we leverage sentiment and emotion information separately with BERT representations obtained from the last BERT-base layer to form the input of a shallow recurrent neural network. In the following, we provide the details.

  • Employing sentiment: We analyze how the sentiment of sentences in proponents’ and opponents’ opinions can affect stance detection. Accordingly, we use a rule-based sentiment tool, VADER Hutto and Gilbert (2014), for obtaining the sentiment of a sentence. VADER translates its compound sentiment score, ranging from to , into negative sentiment labels for scores , positive labels for scores , and neutral for the scores between and .

    Here, we compute sentence-wise sentiment using VADER to let the model learn the flow of sentiment across the opinion. So, each token borrows the sentiment of its corresponding sentence. Then, an embedding layer converts the discrete labels into

    -dimensional vectors

    using a randomly initialized matrix

    ; These representations are concatenated with BERT token embeddings to form the bidirectional Gated Recurrent Units (

    ) input ( for token ):

    For an input sequence with tokens, is the hidden state of the last BERT-base layer corresponding to the input token at time , is sentiment embedding of the token, [;] denotes concatenation operator, , and are parameters of a fully connected layer.

    Recall that our task is to identify the stance of long opinions; So, important information towards the final stance might be anywhere in the opinion. Because of that, we collect such information from the recurrent hidden states of all input tokens using max and average-pooling. Max-pooling returns a vector with maximum weights across all hidden states of input tokens for each dimension. In this way, the input tokens with higher weights will be engaged for stance prediction. Aside from that, the last hidden state of the recurrent network () is concatenated with the pooled information (). Finally, a dense layer transforms vector into the class dimension. Figure 1 shows the model architecture.

    We refer to this model as VADER-Sent-GRU and report the experimental results in Section 6.

  • Employing emotion: We take a similar approach to engage emotion information for stance detection using the NRC emotion lexicon Mohammad and Turney (2013). The Lexicon is collected by crowdsourcing and consists of English words with their eight basic emotions including anger, fear, anticipation, trust, surprise, sadness, joy, and disgust. So, the GRU input is a concatenation of BERT representation with emotion embedding (gained from a matrix with random initialization; one dimension is added for neutral emotion). Here, we use unidirectional as it shows more stable results in our pilot experiments.

5 Experiments

In this section, we describe the corresponding baselines followed by the training setup.

5.1 Baselines

We use the following baselines utilized in opinion mining including sentiment analysis and stance detection:

  • BERT Devlin et al. (2019) followed by a nonlinear transformation on a dense layer is used for downstream stance detection. Here, the whole network is fine-tuned and all

    BERT-base layers’ weights will be updated in backpropagation. The information is pooled from the final hidden state of the classification token (

    ) after passing a fully connected layer with non-linear activation (

    ). Then, a classifier layer shrinks the activations to a binary dimension.

    where are the layers’ parameters.

  • BERT is a BERT base model that considers two different inputs using a perspective and its respective claim Popat et al. (2019). The first input is similar to BERT sentence model’s, [CLS] claim [SEP] perspective [SEP], and the second one is the sequence of [CLS] claim [SEP]. Each input will be given to the BERT model separately. The goal is to incorporate the consistency between the representation of the perspective and claim using cosine distance of the two inputs. Accordingly, the following loss (loss) is added to the regular cross-entropy loss of the BERT model:

    where and are the final hidden state representations corresponding to the [CLS] token of the BERT model for the specified input. In our experiments, we replace the underlying question of a perspective with the claim in the two input sequences.

  • XML-CNN model consists of three convolution layers with kernel size. With a dynamic max-pooling layer, crucial information is extracted across the document. XML-CNN was able to beat most of its deep neural network baselines in six benchmark datasets Liu et al. (2017). We use, BERT, Word2vec, and FastText Mikolov et al. (2018) embeddings for input tokens.

  • AWD-LSTM is a weight-dropped LSTM that deploys DropConnect on hidden-to-hidden weights as a form of recurrent regularization Merity et al. (2017). Word2vec Embedding is used for its input.

We define the corresponding hidden states of the last BERT layer as BERT embedding/representation of input sequence for both single and pair of inputs mode.

5.2 Training

We develop our code based on the Hedwig444https://github.com/castorini/hedwig implementation and train the models on epochs with batch size=. We apply early stopping technique to avoid overfitting during training. Training is stopped after consequent epochs of no improvement of the highest F1 score. We inspect the test set on the model with the best F1 score of development set and keep the settings for BERT the same as the BERT-base-uncased model. Adam optimizer with the learning rate of (for BERT) and (for other models) is used. We see a dramatic drop in BERT performance with some other learning rates. Scikit-learn Pedregosa et al. (2011) library is employed for evaluation measures.

Model P. R. F1
Pair of Input
BERT 76.49 75.37
BERT 70.34 81.24 75.40
XML-CNN(BERT) 68.48 82.22 74.72
VADER-Sent- 69.14 86.62 76.90
NRC-Emotion- 73.79 79.45
Unary Input
BERT 73.89 76.18 75.02
AWD-LSTM(Word2Vec) 65.93 73.25 69.40
XML-CNN(Word2Vec) 58.30 83.03 68.51
XML-CNN(FastText) 66.85 77.32 71.71
XML-CNN(BERT) 70.30 79.93 74.81
VADER-Sent- 66.36 82.38 73.51
NRC-Emotion- 68.46 83.20 75.11
Table 3: Evaluation results; P.:Precision, R.:Recall, : p-value ; : p-value
Model P. R. F1
VADER-Sent- 69.14 86.62 76.90
72.18 78.30 75.12
NRC-Emotion- 73.79 79.45 76.51
69.14 83.69 75.72
Table 4: Effect of sentiment and emotion in our models with pair of input
Question : Is drinking milk healthy for humans?
Top words: nutrients, calcium, niacin, riboflavin, and pantothenic
Table 5: A pro-perspective
Question : Do electronic voting machines improve the voting process?
Top words: vulnerabilities, investment, standpoint, crashes, malicious software, and tampering
Table 6: A con-perspective

6 Results and Discussion

Experimental results are provided in Table 3

. It was expected that fine-tuning BERT with a pair of input achieves a competitive performance among other baselines; but it shows that even with a shallow concatenation of the question and perspective (unary input), BERT can achieve consistent results. Moreover, models that take BERT representation in feature selection mode (without fine-tuning), e.g. XML-CNN(BERT), show better stance detection performance than other token embeddings.

We apply McNemar’s test to measure whether the disagreement between the predictions of the two models is statistically significant.

Among the models with pairs of input, VADER-Sent-GRU gains the highest recall and F1 score. It indicates that the external knowledge gained from a massive corpus, fine-tuned on fewer parameters and enriched with sentiment information can compete with the original architecture ( vs , ). As the model is significantly smaller, it trains faster and needs fewer resources for training. NRC-Emotion-GRU, highlighted in gray, achieves the second-highest F1 score among the models; It reveals that adding emotion information improves stance detection ( vs , ). However, employing sentiment information is more helpful than emotion in detecting the stance of opinions with compelling arguments ( vs , ).

Unlike the superiority of BERT over BERT reported in Popat et al. (2019), we do not see a similar performance here. BERT

uses cosine similarity between the BERT representations of [claim] and [perspective; claim] in the loss function such that their representations become similar when perspective supports the claim and dissimilar when it opposes the claim. This method works for claims and perspectives of the Perspectrum dataset where the two input components are short sentences with

words long. However, in our dataset, we have a question and its perspective that spans multiple sentences. So, forcing the model to make the BERT representations of [question] and [perspective; question] similar or dissimilar, according to the stance, harms the model training. Because the input components have different characteristics utilizing this method results in lower performance than the base model (BERT).

Next, we present some experiments to better understand the model’s units.

Figure 2: Average VADER sentiment scores across five different issues. In each issue the first bar belongs to proponents and the second bar belongs to the opponents

6.1 Effect of Sentiment and Emotion

As stated in Section 4, our recurrent models (VADER-Sent-GRU and NRC-Emotion-GRU) employ sentiment and emotion information of tokens respectively. To see the effect of learning the flow of sentiment and emotion across an opinion, we lift their embeddings from the input of the models. So, and are unidirectional and bidirectional Gated Recurrent Units network respectively, followed by pooling and classification layers:

Similarly, for an input sequence with tokens, is the hidden state of the last BERT layer corresponding to the input token at time , , and are parameters of a fully connected layer.

According to the results in Table 4, both precision and F1 score reduce for the model without emotion (); however, we see a reduction in recall and F1 in the model after lifting sentiment () indicating that integrating sentence-wise sentiment and token-level emotion impact stance detection. We also provide the average sentiment score of the perspectives regarding five different questions in Figure 2. The figure shows the difference between the sentiment of the two stance classes in each issue resulting in a better stance classification. In the next part, we analyze the effect of pooling.

6.2 Pooling Explanation

In Popat et al. (2019), authors find the most important phrases of input by removing phrases from the sequence and finding the ones with maximum effect on misclassification. In our model, we find the crucial information engaged in identifying the stance of a perspective using the max-pooling operation applied to the output sequence of recurrent neural networks (see Section 4). We hypothesize that the more a token is engaged in max-pooling, the more critical the token is for final stance prediction.

Tables 5 and 6 show the heatmap plots of two test instances. The number in each square is the engagement score, the frequency of the presence of a token in max-pooling operation. Darker colors show a higher frequency and indicate how the model identifies the stance across the perspective towards a question. The underlying question in Table 5 asks ‘Is drinking milk healthy for humans?’ According to its figure, we find sub-tokens of nutrients, calcium, niacin, riboflavin, and pantothenic with high scores. All of these words are positively aligned with the final (pro) stance; Specifically, the last three words are a type of Vitamin B. In another example in Table 6, the question is ‘Do electronic voting machines improve the voting process?’ Its corresponding heatmap displays sub-tokens of vulnerabilities, investment, standpoint, crashes, malicious software, and tampering with high scores; all of which are almost consistent with the perspective’s (con) stance.

Similarly, we find the most important words/phrases, regarding their engagement score, for a few other examples of the test set that are correctly classified. The sub-tokens of these phrases have the highest frequency in max-pooling operation. We add (pro) or (con) at the end of each phrase list to indicate the stance of their respective perspective.

  • Should students have to wear school uniforms? uniforms restrict students’ freedom of expression (con)

  • Are social networking sites good for our society? lead to stress and offline relationship (con)

  • Should recreational marijuana be legal? legalization, odious occasion (con)

  • What are the pros and cons of milk’s effect on cancer? dairy consumption is linked with rising death rates from prostate cancer (con)

  • Is human activity responsible for climate change? significant, because, (likely greater than 95 percent probability) (pro)

  • Is obesity a disease? no question that obesity is a disease, blood sugar is not functioning properly, dysregulation, diabetes (pro)

  • Is the death penalty immoral? anymore, failed policy (pro)

The above list shows that the stance-related phrases have been well identified by the model in the pooling step.

7 Conclusion

We propose a model that leverages BERT representation with sentiment or emotion information for stance detection. We create a new dataset for the perspectives that are as long as a paragraph covering a wide variety of contemporary topics. The experiments on our benchmark dataset highlight the effect of emotion and sentiment in stance prediction. The model can improve BERT base performance with significantly fewer parameters. We also explain the contribution of essential phrases of perspectives in detecting their stance using max-pooling operation.

Acknowledgments

This work is supported in part by the U.S. NSF grants 1838147 and 1838145. We also thank anonymous reviewers for their helpful feedback.

References

  • I. Augenstein, T. Rocktäschel, A. Vlachos, and K. Bontcheva (2016) Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on EMNLP, Austin, Texas, pp. 876–885. External Links: Link, Document Cited by: §2.
  • R. Bar-Haim, I. Bhattacharya, F. Dinuzzo, A. Saha, and N. Slonim (2017) Stance classification of context-dependent claims. In Proceedings of the 15th Conference of EACL: Volume 1, Long Papers, Valencia, Spain, pp. 251–261. External Links: Link Cited by: 2nd item.
  • S. Chen, D. Khashabi, W. Yin, C. Callison-Burch, and D. Roth (2019) Seeing things from a different angle:discovering diverse perspectives about claims. In NAACL, Minneapolis, Minnesota, pp. 542–557. External Links: Link, Document Cited by: 3rd item.
  • J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT), Minneapolis, Minnesota, pp. 4171–4186. External Links: Link, Document Cited by: §1, §4, 1st item.
  • E. C. Dragut, C. T. Yu, A. P. Sistla, and W. Meng (2010) Construction of a sentimental word dictionary. In CIKM, pp. 1761–1764. Cited by: §1.
  • J. Du, R. Xu, Y. He, and L. Gui (2017) Stance classification with target-specific neural attention networks. Cited by: §2.
  • A. Hanselowski, A. PVS, B. Schiller, F. Caspelherr, D. Chaudhuri, C. M. Meyer, and I. Gurevych (2018) A retrospective analysis of the fake news challenge stance-detection task. In ACL, Santa Fe, New Mexico, USA, pp. 1859–1874. External Links: Link Cited by: §2.
  • K. S. Hasan and V. Ng (2013) Stance classification of ideological debates: data, models, features, and constraints. In Proceedings of the Sixth IJCNLP, Nagoya, Japan, pp. 1348–1356. External Links: Link Cited by: §2.
  • M. Hosseinia, E. Dragut, and A. Mukherjee (2019) Pro/con: neural detection of stance in argumentative opinions. In SBP-BRiMS, pp. 21–30. External Links: Link Cited by: §1.
  • J. Howard and S. Ruder (2018) Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of ACL, Melbourne, Australia, pp. 328–339. External Links: Link, Document Cited by: §1, §4.
  • C. J. Hutto and E. Gilbert (2014) VADER: a parsimonious rule-based model for sentiment analysis of social media text. In ICWSM, Cited by: 1st item.
  • J. Liu, W. Chang, Y. Wu, and Y. Yang (2017) Deep learning for extreme multi-label text classification. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 115–124. Cited by: 3rd item.
  • S. Merity, N. S. Keskar, and R. Socher (2017) Regularizing and optimizing lstm language models. CoRR abs/1708.02182. Cited by: 4th item.
  • T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, and A. Joulin (2018) Advances in pre-training distributed word representations. In Proceedings of LREC 2018, Cited by: 3rd item.
  • S. Mohammad, S. Kiritchenko, P. Sobhani, X. Zhu, and C. Cherry (2016) SemEval-2016 task 6: detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), San Diego, California, pp. 31–41. External Links: Link, Document Cited by: 1st item.
  • S. M. Mohammad and P. D. Turney (2013) Crowdsourcing a word-emotion association lexicon. 29 (3), pp. 436–465. Cited by: 2nd item.
  • M. Mohtarami, R. Baly, J. Glass, P. Nakov, L. Màrquez, and A. Moschitti (2018) Automatic stance detection using end-to-end memory networks. In Proceedings of the 2018 Conference of NAACL: Human Language Technologies, New Orleans, Louisiana, pp. 767–776. External Links: Link, Document Cited by: §2.
  • F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay (2011)

    Scikit-learn: machine learning in Python

    .
    Journal of Machine Learning Research 12, pp. 2825–2830. Cited by: §5.2.
  • M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer (2018) Deep contextualized word representations. In Proceedings of the 2018 Conference of NAACL: Human Language Technologies, New Orleans, Louisiana, pp. 2227–2237. External Links: Link, Document Cited by: §4.
  • K. Popat, S. Mukherjee, A. Yates, and G. Weikum (2019) STANCY: stance classification based on consistency cues. In Proceedings of the 2019 Conference on EMNLP-IJCNLP, Hong Kong, China, pp. 6413–6418. External Links: Link, Document Cited by: 3rd item, §2, 2nd item, §6.2, §6.
  • B. Riedel, I. Augenstein, G. P. Spithourakis, and S. Riedel (2017) A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. CoRR abs/1707.03264. External Links: Link Cited by: §2.
  • A. T. Schneider and E. C. Dragut (2015) Towards debugging sentiment lexicons. In ACL, pp. 1024–1034. Cited by: §4.
  • D. Sridhar, J. Foulds, B. Huang, L. Getoor, and M. Walker (2015) Joint models of disagreement and stance in online debate. In Proceedings of the 53rd ACL and the 7th IJCNLP, Beijing, China, pp. 116–125. External Links: Link, Document Cited by: §2.
  • M. Stanojevic, J. Alshehri, E. C. Dragut, and Z. Obradovic (2019) Biased news data influence on classifying social media posts. In NEwsIR@SIGIR, Vol. 2411, pp. 3–8. Cited by: §1.
  • Q. Sun, Z. Wang, Q. Zhu, and G. Zhou (2018) Stance detection with hierarchical attention network. In ACL, Santa Fe, New Mexico, USA, pp. 2399–2409. External Links: Link Cited by: §2.
  • M. Walker, P. Anand, R. Abbott, and R. Grant (2012) Stance classification using dialogic properties of persuasion. In NAACL-HLT, Montréal, Canada, pp. 592–596. External Links: Link Cited by: §2.
  • F. Yang, A. Mukherjee, and E. Dragut (2017) Satirical news detection and analysis using attention mechanism and linguistic features. In Proceedings of the 2017 Conference on EMNLP, Copenhagen, Denmark, pp. 1979–1989. External Links: Link, Document Cited by: §4.
  • Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le (2019) Xlnet: generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5754–5764. Cited by: §1, §4.
  • S. Zhang, L. He, E. Dragut, and S. Vucetic (2019) How to invest my time: lessons from human-in-the-loop entity extraction. In SIGKDD, pp. 2305–2313. External Links: ISBN 9781450362016 Cited by: §4.