A Computational Analysis of Polarization on Indian and Pakistani Social Media

05/20/2020
by   Aman Tyagi, et al.
Carnegie Mellon University
0

Between February 14, 2019 and March 4, 2019, a terrorist attack in Pulwama, Kashmir followed by retaliatory air strikes led to rising tensions between India and Pakistan, two nuclear-armed countries. In this work, we examine polarizing messaging on Twitter during these events, particularly focusing on the positions of Indian and Pakistani politicians. We use label propagation technique focused on hashtag cooccurences to find polarizing tweets and users. Our analysis reveals that politicians in the ruling political party in India (BJP) used polarized hashtags and called for escalation of conflict more so than politicians from other parties. Our work offers the first analysis of how escalating tensions between India and Pakistan manifest on Twitter and provides a framework for studying polarizing messages.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/21/2020

Inferring Political Preferences from Twitter

Sentiment analysis is the task of automatic analysis of opinions and emo...
05/04/2022

Semi-supervised learning approaches for predicting South African political sentiment for local government elections

This study aims to understand the South African political context by ana...
10/28/2020

Using Twitter to Analyze Political Polarization During National Crises

Democrats and Republicans have seemed to grow apart in the past three de...
03/03/2020

Introducing the Spatial Conflict Dynamics indicator of political violence

Modern armed conflicts have a tendency to spill across state boundaries ...
09/03/2020

A Python Library for Exploratory Data Analysis and Knowledge Discovery on Twitter Data

Twitter is perhaps the social media more amenable for research. It requi...
07/16/2018

Modeling the social media relationships of Irish politicians using a generalized latent space stochastic blockmodel

Dáil Éireann is the principal chamber of the Irish parliament. The 31st ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

While social media platforms foster open communication and have the potential to offer more democratic information systems, they have simultaneously facilitated divisions in society by allowing the spread of polarizing and incendiary content [demszky-etal-2019-analyzing, stewart2017drawing]. Polarizing content can be beneficial by encouraging pride and solidarity, but it has also become a social cyber-security concern: foreign and domestic actors may employ polarizing social media content to sow divisions in a country, to demean other nations, or to promote political agendas [arif2018acting, bradshaw2018challenging, carley2018social, king2017chinese].

Using automated methods to analyze social media offers a way to understand the type of content users are exposed to, the positions taken by various users, and the agendas pursued through coordinated messaging across entire platforms. Understanding the dynamics of this information landscape has become critical, because social media can strongly influence public opinion [bradshaw2018challenging]. However, prior computational social science research on polarization has focused primarily on U.S. politics, and much attention has focused on the influence of Russian or Chinese state actors [arif2018acting, badawy2018analyzing, demszky-etal-2019-analyzing, golovchenko2018state, king2017chinese, le2019postmortem, starbird2019disinformation, stewart2017drawing]. In contrast, we focus on polarizing social media content in India and Pakistan and how it can contribute to rising tensions between these two nations. Specifically, we examine communication patterns on Twitter following the terrorist attack in the Pulwama district, Jammu and Kashmir, India, on February 14, 2019.

We primarily investigate the research question: to what extent did entities on social media advocate for or against escalating tensions?. India and Pakistan are both nuclear-armed countries and have a decades-long history involving multiple armed conflicts. The Pulwama attack in 2019 was followed by an escalation of tensions between these two nations that nearly approached full-fledged war [feyyaz2019contextualizing, palakodety2019kashmir, pandya2019future]. Moreover, the relationship between these countries is an important agenda for political parties in both India and Pakistan. India has two primary political parties: the Indian National Congress (INC), which was dominant in the early 21st century, and the Bharatiya Janata Party (BJP), which rose to prominence on a populist and nationalist platform in 2014 and has been in power since [McDonnell2019BJP]. We examine how the social media messaging of members of these parties differed, and how it changed over the sequence of events.

Our core methodology uses a network-based label propagation algorithm to quantifying the polarity of hashtags along specified dimensions: Pro-India vs. Pro-Pakistan and Pro-Aggression vs. Pro-Peace. We then aggregate the hashtag-level scores into tweet-level and user-level scores, e.g. the polarity of a given user on a given day. Unlike methodology that assumes users’ opinions do not change [darius2019twitter, darwish2019quantifying, Weber2013], focuses on binary stances [Holthoefer2015], or requires in-language annotations and feature-crafting [magdy2016isisisnotislam], our methodology allows us to analyze degrees of polarization in a multilingual corpus and how they change over time.

We begin by providing an overview of the events between February 14 and March 1, 2019 (§2). Next, we describe the Twitter data collection (§3) and discuss methods (§4) and evaluation (§5). Our results (§6) suggest that more members of the BJP propagated a narrative of escalation than members of other political parties. This finding supports anecdotes reported by journalists [WashP] about these events. Through this research, we develop (1) the first analysis of escalating tensions between India and Pakistan on Twitter, (2) a data-driven investigation of social media messaging following the 2019 Pulwama attack, and (3) a novel and general methodology to examine polarization on multilingual social media.

2 Timeline of Events

We briefly provide background on relevant events, relying primarily on third party newspapers unaffiliated with either nation (The New York Times and BBC News) and noting where official accounts differ.

Feb. 14, 2019 A 22-year old native of Pulwama carried out a suicide attack against a convoy carrying approximately 2,500 security personnel in the Pulwama district in Kashmir, India. The attack resulted in the death of more than 40 Indian soldiers. Jaish-e-Mohammad (JeM) a militant group based in Pakistan (the group is formally banned in Pakistan) claimed responsibility  [bbcbackground, HT, NYTBomber, NYTConvoy].

Feb. 14-26, 2019 The Indian government responded to the attack with threats of retaliation against Pakistan, even though Pakistani officials denied any role [bbcMourning, bbcThreats, toiRelat]. Diplomatic ties deteriorated, e.g., India revoked Pakistan’s most favored nation status, which had provided trade advantages [mfn]. Pakistan threatened to retaliate if India pursued military action [bbcimrankhan].

Feb. 26, 2019 The Indian Air Force (IAF) conducted a retaliatory airstrike against a JeM training camp inside Pakistan, which the Indian government termed “non-military, preemptive” [balakot]. According to Indian government officials, the JeM camp targeted by this airstrike was located 70km inside the Line of Control (LoC) – the military line dividing the Indian and Pakistani controlled parts of Jammu and Kashmir. Indian officials reported that the airstrike was “100 percent successful”, went on “exactly as planned”, and killed over 200 terrorists [ndtvAirStrikes, balakot]. In contrast, Pakistani officials reported that the target of the attacks was located only 5–6km inside the LoC, that the Pakistani air force turned back the Indian fighters, and that the attacks landed in an empty area [dawn_indian_aircraft, tribune_report_strike].

Feb. 27, 2019 The Pakistan Air Force (PAF) carried out retaliatory airstrikes along the LoC. Indian and Pakistani officials presented different details of the strikes, but both emphasized de-escalation: a Pakistani official reported that the PAF intentionally targeted open spaces, to demonstrate Pakistan’s capabilities without inviting escalation, while an Indian official reported no deaths or civilian casualties [cnnPAFStrike2, cnnPAFStrike1, econTimesPAF]. However, in aerial combat following the strikes, an IAF pilot was captured by the Pakistani Army [bbcPilot, IAF].

Mar. 1, 2019 Pakistan returned the IAF pilot to India on March 1 in what Pakistani Prime Minster Imran Khan called “a gesture of peace” [bbcPilotRet].

3 Data

We collected tweets related to these events by first identifying a set of relevant hashtags. Our hashtag set is based on hashtags related to #pulwama found on best-hashtags.com.111best-hashtags.com uses an algorithm to provide popular hashtags that are similar to the provided seed (#pulwama). Since our analysis, the website has stopped reporting Twitter hashtags. We modified the hashtag set to ensure that it included both hashtags more likely to be used by Pro-India users (e.g., IndiaWantsRevenge) and hashtags more likely to be used by Pro-Pakistan users (e.g., PakistanZindabad). We then collected all tweets using these terms, either as words or as hashtags during the events. We provide further details, including the full list of keywords and evidence that our data set is comprehensive in Appendix 8.

Our final data set contains 2.5M unique tweets (including retweets) from 567K users that use 67K unique hashtags. All tweets occurred between February 14th and March 4th. The data contains a mix of languages including English, Urdu, and Hindi, and many users use multiple languages in the same tweet. While some tweets express neutral opinions, others contain incendiary language, such as: “@PMOIndia @PMOIndia @narendramodi We r eagerly waiting for ur action of revenge…#PulwamaRevenge #IndiaWantsRevenge #PulwamaAttack” and “I feel time has come to give all support to #Balochistan activist. Let us #bleed Pakistan from all fronts. #NeverForget @PMOIndia @narendramodi #IndiaWantsRevenge”. In in Appendix 8, we provide further statistics on the data, including network densities.

4 Methodology

We develop a method to assign a polarity score to an aggregate group of tweets, and we analyze how polarities change over time for different groups of users. For instance, given pole (e.g., Pro-Pakistan) and pole (e.g., Pro-India), we aggregate all tweets by a given user and assign the user a polarity score between [a, b], where a score close to indicates the user more likely supports and a score close to indicates the user more likely supports B. We could also aggregate only tweets by the user on one day and determine the user’s Pro-A/Pro-B polarity on that day.

In the absence of annotated data, we use a weakly supervised approach. First, for pole , we hand-select a small seed set of tokens that are strongly associated with , and we equivalently hand-select . We assign each a polarity score of , and we assign each a polarity score of . Then, we use and

to infer polarity scores over a larger lexicon of words or hashtags

, where each is assigned a score in

. Finally, we estimate the polarity of an aggregated set of tweets by summing or averaging the inferred polarity scores for all

used in those tweets.

In order to propagate the hand-annotated labels in and to the larger lexicon , we use 3 variants of graph-based label propagation. In each variant, we construct a graph , whose nodes consist of and whose edges and edge weights are defined based on similarity metrics between members of . We describe each variant in detail below.

Network-based Hashtag Propagation

In the first variant, we define to be the set of all hashtags used in our data set. Then, we construct as a hashtag*hashtag co-occurrence network. Each node in corresponds to a hashtag. Edges occur between hashtags that co-occur in the same tweet, and edge weights are proportional to how frequently the hashtags co-occur. Then, we use the label propagation algorithm detailed in Algorithm 1 to infer polarity scores for from and , where and . The algorithm uses a greedy approach to assign labels to each node in G. If all nodes connected to a node n have been labeled, then node n is assigned a weighted average of all the adjacent nodes. This step is repeated until the maximum possible number of nodes are labeled ( controls the number of repetitions).

Our algorithm is similar to methods used to infer user-level polarities, in which small seed of users is hand-annotated and a graph-based algorithm propagates labels to other users by assuming that users who retweet each other share the same views [darwish2019quantifying, Weber2013]. Similar to above work, work by [kiranMorales] also quantifies polarity based on the graph structure and assumes that the controversial topics induce clusters of discussions, commonly referred to as “echo-chambers” However, we conduct propagation at a hashtag level, by assuming that hashtags that frequently occur in the same tweets indicate similar polarities. Also, our approach does not assume homophily in retweet network nor that user polarities are constant over time.

Network-based Word Propagation

The second variant is similar to the first; however, instead of restricting to be the set of hashtags in the corpus, we define to be the set of all tokens, including words and hashtags. We then construct G as a token*token co-occurrence network, and as above, we infer labels using Algorithm 1 and obtain token-level polarity scores in the range [-1, 1]. Expanding to all tokens instead of just hashtags allows our algorithm to incorporate more information, but also risks introducing noise, as we do not attempt to process nuances in language like negation.

Embedding-based Word Propagation (SentProp)

In the third variant, we define to be the set of all tokens, as in the Network-based Word Propagation approach. Then, we train GloVe embeddings [pennington2014glove] over our entire corpus (limiting vocabulary size to 50K). We then use SentProp [hamilton2016inducing], a method for inferring domain-specific lexicons to infer labels over . In this method, as before, we construct a graph where each is a node. However, rather than relying on raw co-occurrence scores, SentProp uses embedding similarity metrics to define edge weights and a random-walk method to propagate labels. We implement SentProp using the SocialSent package [hamilton2016inducing], where and .

Input: Graph G with nodes n and edges e with as the edge weight between and
initialize and i=0;
for each n do
       define = integer(i/); +=1;
       for each n do
             if n not labeled then
                   compute = neighbors of ;
                   compute = labeled neighbors of ;
                   if  then
                         initialize score,
                         for each  do
                               score += label ; c +=
                              
                        update label
                  
            
      
Algorithm 1 Label Propagation Algorithm

Once we have obtained hashtag-level or word-level polarity scores, we infer the polarity of a tweet or a group of tweets (e.g. all tweets by a given user) by averaging the polarity scores inferred by our algorithms for all the hashtags and words used in data subset. This approach is similar to the aggregation conducted in [Holthoefer2015], but our label propagation allows for the incorporation of thousands of words and hashtags, rather than relying on only a small hand-annotated set. If the data subset does not contain any of the keywords labelled by our algorithm (e.g. in a hashtag-based approach, the tweet contains no hashtags), we consider it unclassified. In some cases, primarily for evaluation, we convert the polarity scores into a ternary negative/neutral/positive position by using the cut-offs for the scale and for the scale.

This methodology allows us to infer the polarity of any group of tweets along any dimensions, provided a small set of seed words or hashtags for each dimension. Thus, we can examine how polarities differed for different groups of users and how they changed over time. The two dimensions we focus on are Pro-India/Pro-Pakistan and Pro-Peace/Pro-Aggression, and we provide our manually defined seed sets in Appendix 9. In practice we found that variations in the exact words in seed set had no noticeable impact on our final results. For the network-based methods, we label Pro-India seeds as , Pro-Pakistan seeds as , Pro-Peace seeds as , and Pro-Aggression seeds as . For the embedding-based approach, we label Pro-India seeds as Pro-Pakistan seeds as , Pro-Peace seeds as , and Pro-Aggression seeds as . For all word-based approaches, we limit the vocabulary size to 50K.

Pro-India Pro-Pakistan
Prec. Recall % Unk. Prec. Recall % Unk.
Hashtag 0.91 0.78 0.68 0.90 0.96 0.36
Word 0.69 0.91 0.24 0.84 0.54 0.34
Sentprop 0.49 0.80 - 0.43 0.15 -
Table 1:

Classification results for the 100 most followed Indian and Pakistan Twitter accounts, where Pro-India or Pro-Pakistan are treated as the dominant class, and the nationality of the account owner is treated as a gold label. Accounts that our algorithm was unable to classify (% Unk) are not included in precision and recall scores.

5 Evaluation

Automated Evaluation

We first evaluate our methods by focusing on the Pro-India and Pro-Pakistan dimension and assuming that popular users in India are more likely to post Pro-India content and popular users in Pakistan are more likely to post Pro-Pakistan content. From the Socialbakers.com

platform, we identified the 100 most followed Twitter accounts in India and in Pakistan. 16 of the Indian accounts and 15 of the Pakistani accounts do not occur in our data, leaving 84 Indian accounts with 2,199 tweets and 85 Pakistani accounts with 1,456 tweets for evaluation. For each account, we average words and hashtag polarities over all tweets from the account, and binarize the resulting score into Pro-Pakistan or Pro-India position.

Table 1 reports precision and recall scores when we treat Pro-India as the primary class and when we treat Pro-Pakistan as the primary class. Both of the network-based approaches rely on hashtag or word co-occurrences to propagate labels. Thus, hashtags and words that do not have any co-occurrence links to the original seed list are unable to be labeled. For instance, in the hashtag propagation approach, our method labels 41,700 hashtags out of 67,059 total hashtags in the dataset. Any users who only use unlabeled words or hashtags are therefore unable to be classified by our algorithm, resulting in 88/169 unlabeled accounts for the hashtag approach and and 49/169 unlabeled accounts for the word approach (% Unk in Table 1). In contrast, SentProp obtains polarity scores for all accounts, as it relies on embedding similarity and can propagate labels between words, even if they do not ever co-occur.

However, although Sentprop labels more accounts, its precision is much lower than the network-based methods. The network-based hashtag propagation approach overall obtains the highest precision. Although the word-propagation approach labels more accounts and works well over the Indian accounts, its classification of the Pakistani accounts (e.g. Recall) is close to random. We suspect that our method works well for hashtags, because they tend to be strongly polar and indicative of the overall sentiment of the tweet. Also, users tend to use English hashtags even though they may be tweeting in other languages. A word-based approach likely requires more careful handling of subtle language cues like negation or sarcasm.

In our subsequent analysis, we use the network-based hashtag propagation method in order to infer polarities, thus favoring high precision and strong polarization, and choosing not to analyze data where we cannot infer polarity with high-confidence. Additionally, in examining the data set, we found that many of the top-followed accounts in India and Pakistan consisted of celebrities who avoided taking stances on politicized issues, which makes the high number of unclassified accounts in this subset of the data unsurprising.

Krippendorff % Agree. Hashtag Acc. Soft Hashtag Acc.
India/Pakistan 0.77 88% 74% 89%
Aggression/Peace 0.60 74% 57% 76%
Table 2: Inter-annotator agreement and classification accuracy over 100 manually annotated data points

Manual Evaluation

In order to further evaluate our methods, we compare the performance of the network-based hashtag model with a small sample of manually annotated tweets. We randomly sampled 100 users from our data set. For each user, we randomly sampled 1 day on which the user tweeted and aggregated all tweets from that day. Thus, we conduct this evaluation at a per-user-per-day level. Two annotators independently annotated each data sample as Pro-India/Pro-Pakistan/Neutral/Can’t Determine and Pro-Peace/Pro-Aggression/Neutral/Can’t Determine. For simplicity, we collapsed Neutral/Can’t Determine and Unclassified into a single “Neutral” label. Notably, these dimensions are distinct. For example, users may write tweets that are Pro-Peace and Pro-Pakistan: “Don’t let people pull you into their War, pull them into your Peace… Peace for our World, Peace for our Children, Peace for our Future !! #PakSymbolOfPeace #SayNoToWar” or that are Pro-Peace and Pro-India: “Very mature conciliatory speech by #ImranKhan. We now urge him to walk the talk. Please return our #Abhinandan safely back to us. This will go a long way in correcting perceptions and restoring peace. #SayNoToWar”.

Table 2 reports inter-annotator agreement, which is generally high. Additionally, most disagreements occurred when one annotator labeled Neutral/Can’t Determine and the other did not, meaning polar opposite annotations were rare. If we only count polar opposite labels as disagreements, the percent agreement rises to 94% for both dimensions.

Then, the two annotators discussed any data points for which they initially disagreed and decided on a single gold label for each data point. We compare performance of the network-based hashtag propagation method against these gold annotations in Table 2. In this 3-way classification task, our method outperforms random guessing. In particular, the “Soft” accuracy, in which we only consider the model output to be incorrect if it predicted the polar-opposite label, meaning neutral/unclassified predictions are not considered incorrect, is high for both dimensions. We provide further metrics and discussion in Appendix 10.

6 Results and Analysis

We investigate multiple aspects of our data set, including network structure, polarities of various entities, and changes over time. Based on prior work suggesting that political entities in India and Pakistan may use social media to influence public opinion [indian_2014_elections, india_twitter_rahul, Kumar2016PoliticalMarket, pakistan_social], we pay particular attention to the Twitter accounts of politicians as a method for uncovering political agendas.

Position Unique Users Total Tweets Position Unique Users Total Tweets
Pro-India 125K (23%) 1.16M (46%) Pro-Aggression 78K (14%) 626K (25%)
Pro-Pakistan 117K (20%) 764K (30%) Pro-Peace 252K (45%) 1.48M (59%)
Unclassified 325K (57%) 578K (23%) Unclassified 237K (40%) 351K (16%)
Table 3: Overall polarities of users and tweets.
Figure 1: 30-core all communication networks, colored by Pro-India/Pro-Pakistan polarity (left) and Pro-Peace/Pro-Aggression polarity (right). The Pro-India/Pro-Pakistan network displays more homophily than the Pro-Peace/Pro-Aggression network.

What are the overall polarities of our data set?

In Table 3, we obtain polarity scores for each user and tweet and then ternarize them into Pro-India/Pro-Pakistan/Unclassified and as Pro-Peace/Pro-Aggression/Unclassified in the same way as in §5. At the user level, the classified accounts are approximately balanced between Pro-India and Pro-Pakistan. However, at a tweet level, the classified data contains a high percentage of Pro-India tweets, which suggests that Pro-India users tweeted about this issue more prolifically. On the Pro-Peace/Pro-Aggression dimension, there is a much higher percentage of Pro-Peace users than Pro-Aggression users. This pattern also holds at a tweet level, where only a small percentage of tweets are unclassified.

What are characteristics of the communication network?

Next, we examine the communication network between users, particularly prevalence of echo chambers. Did users with opposite positions interact on Twitter? Figure 1 shows a 30-core all communication network constructed using ORA-PRO [ora_tool_kit]. Accounts are colored based on their Pro-India/Pro-Pakistan polarity (left) and Pro-Peace/Pro-Aggression polarity (right). An edge occurs between two users if one user retweeted, mentioned, or replied to the other and users with fewer than 30 links are not shown. Unsurprisingly, the Pro-India/Pro-Pakistan position is highly segregated, with little interaction between users with different positions. In contrast, the Pro-Peace/Pro-Aggression dimension is more mixed. Although there are some areas of high density for each position, there is interaction between users of different positions. These interactions are potential avenues for users to influence each other’s views. Appendix 11 additionally reports network densities for the derived network: We find a higher density in the Pro-Aggression community than the Pro-Peace community for all communication types, meaning the Pro-Aggression community is closer-knit. Similarly, we find a closer Pro-India community as compared to the Pro-Pakistan community.

Figure 2: Aggregate Pro-Peace and Pro-Aggression polarities of the most popular Indian (33/78) and Pakistani (36/66) politicians in our data set (left) and of members of Indian political parties (right)

How polarized were different political entities?

We investigate the polarities projected by different political entities: specifically BJP politicians (currently in power in India), INC politicians (largest opposition party), other Indian politicians, and Pakistani politicians.

We first used Socialbakers platform to obtain the Twitter handles of the 100 most followed politicians in India and in Pakistan (as in §5). Our data set contained tweets from 66 Pakistani politicians and 78 Indian politicians, and our model was able to infer polarity scores for 36 Pakistani politicians and 33 Indian politicians. On the left in Figure 2, we report aggregate polarity scores over all tweets from these politicians. Politicians from Pakistan were predominately Pro-Peace, while Indian politicians expressed more mixed polarities, resulting in a near neutral aggregate score.

We then examined a broader set of Indian politicians, subdivided by political party based on a list of members running for parliament elections in 2019 [loksabhaelections2019]. Out of the 1,360 twitter handles in list list, our data set contained activity from 316 BJP accounts, 281 INC accounts, 204 other Indian party accounts.

On the right in Figure 2 we show the overall polarities, aggregated from all tweets by verified members of each party. Strikingly, members of the BJP party are positioned as much more Pro-Aggression than the members of either the INC or other Indian parties, and the party overall obtains a Pro-Aggression polarity score 222Examples of pro-aggression tweets by BJP members can be found in §11. This score is not dominated by 1-2 strongly polarized members of the party: if we aggregate the polarity scores by individuals instead of by party, 15% of BJP members had net Pro-Aggression scores and 13% had net Pro-Peace scores. In comparison, 10% of the INC members had net Pro-Aggression scores and 25% had net Pro-Peace scores. 6% of other Indian party members were classified as Pro-Aggression and 29% were classified as Pro-Peace.

These results support observations made by journalists and community members about the role of the BJP party in these events. BJP is well-known for promoting nationalism, and several journalists have speculated that conflict with Pakistan would increase Prime Minister Modi’s chances of winning the upcoming elections in April and accused the BJP of war-mongering [EcoTimesModiElection, ForbesModiElection, orfOpinion, theWeekWarMonger].

Figure 3:

Daily Pro-Peace/Pro-Aggression positions of political entities. Negative values denote Pro-Aggression stance and positive values denote Pro-Peace. The error bars represent ±1 standard deviation.

How did polarization change over time?

Figure 3 shows how this polarity changed over the two-week period of events: we infer a Pro-Peace/Pro-Aggression polarity score for all tweets posted by members of the specified political subgroup, and we plot the average score across tweets posted each day.

Immediately following the initial attack on 2/14, the tweets from all Indian political party members were inclined towards Pro-Aggression, suggesting initial outrage. However, over the next few days, while members of the INC and other Indian political parties switch towards Pro-Peace, members of the BJP party were consistently positioned with Pro-Aggression polarity. There is high volatility between 2/20 and 2/26. However, there was a much lower volume of tweets about the Pulwama incidents during this time period (tweet volume is reported in Appendix 11), and we do not believe these fluctuations are meaningful. The volume of tweets increases once again following the Indian airstrike on 2/26 and the Pakistani airstrike on 2/27. Tweets by Pakistani politicians generally fall on the Pro-Peace side, but they become more polarized after the Indian airstrike and reach a peak following the Pakistani airstrike. This is consistent with reported quotes by Pakistani officials (§2), saying that the airstrike was designed to avoid escalation. Similarly, tweets by Indian politicians from the INC and other parties become strongly Pro-Peace directly following the Indian airstrike, with polarity increasing after the Pakistani airstrike. In contrast, on the day of the Pakistani airstrike, tweets by BJP politicians remain Pro-Aggression, possibly focusing either on praise for the Indian airstrike or condemnation of the Pakistani airstrike. The polarity of the BJP tweets belatedly switches to Pro-Peace on the following day (2/28), though the strength of the Pro-Aggression polarity still remained weaker for BJP tweets than for tweets by other politicians.

7 Discussion and Related Work

The potential that social media platforms have for manipulating public opinion has led to growing interest in information operations and the development of social cyber security as a field of research [carley2018social, starbird2019disinformation]. While we do not claim that social media coverage of the Pulwama incident constituted an information operation, e.g. coordinated efforts to manipulate public opinion and change how people perceive events [starbird2019disinformation], we do find similarities between our observations and other work in this area. Notably, as described in §2, the Indian and Pakistani governments maintain starkly different accounts about the events that occurred, particularly whether or not the 2/26 airstrikes resulted in 200 casualties. Similarly, Russian and Ukranian governments circulated conflicting narratives about the cause of the crash Malaysian Airlines Flight MH17 in 2014, which prompted analyses of information operations about this incident. In a work similar to ours, [golovchenko2018state] examine social media coverage of the incident by using a set of hashtags to collect all relevant tweets during a set time frame. Other work has examined the media influence of Chinese and Russian state actors in various domains, including US and UK elections and the Syrian War [field2018framing, king2017chinese, kriel2019reverse, rozenas2019autocrats, starbird2018ecosystem]. [arif2018acting] examine Russian influence in polarizing movements on Twitter, particularly the #BlackLivesMatterMovement, and observe how Russian actors attempted to increase tensions between involved parties. Furthermore, the polarization that we observe in our data align with the “Excite” and “Dismay” strategies, which are tools of public opinion manipulation described in the BEND forms of maneuver [beskow2019social].

Almost all of these works are focused on U.S. social media, possibly involving Chinese and Russian actors. In general, most work on polarization and public opinion change has focused on U.S. politics [chen2019seeing, darwish2019quantifying, khosla2019events], with a few exceptions focusing on Germany [darius2019twitter], Egypt [Holthoefer2015, Weber2013], and Venezuela [morales2015measuring]. Work on social media in India and Pakistan has focused on healthcare [Abbasi2018], natural disasters [Murthy2013], self-promotion (e.g. “brand marketing”) primarily in relation to elections [indian_2014_elections, india_twitter_rahul, Kumar2016PoliticalMarket], or on election forecasting [Kagan2015, pakistan_social], though [pakistan_army_twitter] does argue that the Pakistan Army uses social media to subvert democracy. While these works only focus on intra-country analysis, our work also examines tensions between India and Pakistan. A small selection of work has also looked at the incidents in Pulwama and the implications of rising tensions. [feyyaz2019contextualizing] and [pandya2019future] discuss the sociopolitical context and implications of events from a non-computational perspective. [palakodety2019kashmir] additionally conduct a social media analysis, but they use YouTube data and focus on identifying deescalating language. Their timeline of escalation and deescalation is generally consistent with our findings.

Our primary methodology involves using label propagation to infer aggregated polarity scores. In language corpora, label propagation has typically relied on embedding similarity [hamilton2016inducing, rothe2016ultradense]. Our approach takes advantage of the short-text nature of Twitter through co-occurrences networks, as well as the strong semantic signals provided by hashtags [ICWSM1510473].

Conclusions Polarizing language on social media can have long-lasting sociopolitical impacts. Our analysis shows how Twitter users in India and Pakistan used polarizing language during a period of escalating tensions between the two nations, and our methodology offers tools for future work in this area.

References

Appendix

8 Data Collection and Statistics

We collected Twitter data using the Twitter API. The API collects all tweets that have provided keywords from the last 7 days. We hence collected data on February 20th, February 26th Feb and March 4th to obtain all tweets during the relevant time period. All the data collection was done using the keywords given below except “Balakot” and “Abhinandan”, which were added after the airstrikes. We de-duplicated the collected tweets to remove tweets collected twice during multiple collection runs.

To check whether our keywords covered most of the debate on these events, we scraped the timelines of 100 most-followed politicians in India and Pakistan during the incidents. For Pakistan politicians, we found that 1566 tweets were in our original keyword-based collection of tweets and 333 tweets were not part of our collection. On manual inspection, out of the total 333, only 40 tweets were found to be related to Pulwama and the rest were not relevant. For Indian politicians, we found that 2615 tweets were in the original keyword-based collection and 70 tweets were not, out of which only 23 tweets were found to be related to the incident. Thus, we believe our data set contains comprehensive coverage of tweets related to these events.

Keywords used for data collection:

PulwamaAttack, Pulwama, TerroristArmy, PhulwamaTerrorAttack, PhulwamaRevenge, KashmirTerror, PakistanZindaba, KashmiriStudents, KashmirBleeds, KashmirBanegaPakistan, JihadKashmir, SurgicalStrike, SayNoToWar, JeM, JaisheMohammad, Jaish, IndiaWantsRevenge, Gaddar, FreeKashmir, Fidayin, CrpfConvoyAttack, AzadKashmir, AJK, Balakot, AdilahMaddar, LethopraSuicidalAttack, Abhinandan

Unique Users 567K Mentions 725K (2.25*1e-6)
Total Tweets (Including Retweets) 2.5M Reciprocal 36K (2.2*1e-7)
Total Unique Hashtags 67K Replies 43K (1.3*1e-7)
Total Communication Links 2.3M (7.24*1e-6) Retweets 1.6M (4.9*1e-6)
Min. Tweets/User 1 Max. Tweets/User 3166
Avg. Tweets/User 4.4 SD Tweets/User 13.2
Table 4: Data statistics (counts) for tweets collected during and after Pulwama attack. The values in parentheses are network density values.

Table 4 reports statistics about the collected data, including the types of communication present. Twitter allows three types of interactive communication: mentions (e.g., ), replies, and retweets. We classify retweets with comments as retweets. We also define reciprocal communication as two users mentioning each other, and all communication is the sum of the above three types of communication. Furthermore, we construct networks for each type of communication, where users are nodes, and an edge exists between two nodes if those two users interacted with the specified type of communication. Table 4 reports both the frequency of each communication type in our data set and the densities of the constructed communication networks. Retweets occur more frequently and also result in higher network density than other forms of communication.

9 Seed words for label propagation

Pro-India seeds

IndiaWantsRevenge, PhulwamaRevenge, ModiUnstoppable, AbhinandhanMyHero, RespectToIndianArmy, IndiaAgainstTerroristan, JusticeForPhulwamaAttack

Pro-Pakistan seeds

AzadKashmir, KashmirBanegaPakistan, PakistanZindabad, HindustaanMurdabaad, PakistanArmyGreatArmy, PakistanArmyZindabad, SaluteToOurPakistanArmy

Pro-Aggression seeds

SayYesToWar, 4kebadle400

Pro-Peace seeds

SayNoToWar, SaveHumanity

10 Evaluation Analysis

Sample inferred Pro-Aggression Hashtags stonehearted, pakistantakesrevenge , trustarmy, baaphaitumhare, barkhaspreadsporn, planprepare, pakistanamurdabad, indiaagainstantinationals, indianarmyourpridd, boycott_pakistani_players, yh_aag_na_thandi_hopaye, timestorevenge, ab_hoga_tanav, candleofrevenge, pulawamaislamicterrorattack, indiamusttakerevenge, avengeforpulwamaattack, russiawithmodi,boycottsnehawagh, iamwaitingindia

Sample inferred Pro-Peace Hashtags saynotoindopakmedia, savefamily, wekashmiridontwantwar, southasian, uniteasia, sayyestorobo, letstabletalk, saynotopakproxywar, saynotononsense, doyourjob, neverunderestimate, wecandobetter, wngcmdrabhinandan, stopmakingnonsensestatements, yestocricket, pleasenohate, flowersfrompakistan, peacepreaching, saynotocheapmedia, indianeedseducatedpm

True Pro-India True Pro-Pakistan True Neutral
Predicted Pro-India 45 1 3
Predicted Pro-Pakistan 10 24 0
Predicted Neutral/Unclassified 10 2 5
Table 5:

India/Pakistan Confusion Matrix for the manual evaluation described in §

5
True Pro-Aggression True Pro-Peace True Neutral
Predicted Pro-Aggression 21 0 6
Predicted Pro-Peace 24 31 10
Predicted Neutral/Unclassified 2 1 5
Table 6: Aggression/Peace War Confusion Matrix for the manual evaluation described in §5

Tables 5 and 6 show the confusion matrices for the network-based hashtag propagation method over the manually annotated data set described in §5. In the Pro-India/Pro-Pakistan dimension, classification accuracy is generally high with few misclassified examples. In the Pro-Aggression/Pro-Peace dimension, the most common error is predicting tweets that are actually Pro-Aggression as Pro-Peace. This error likely results because numerous Pro-Aggression tweets exhibited “hashjacking”, in which users co-opting the hashtags preferred adversaries [Bode15, darius2019twitter]. For example:

“1993 Mumbai

1998 Coimbatore

2001 Parliament attack

2002 Akshardham

2003, 2006 Mumbai trains

2005 Delhi

2006 Varanasi

2007 Samjhauta + Hyderabad

2008 Mumbai 26/11

2016 Uri

2019 Pulwama

Hundreds more…

Did our Pak lovers say #SayNoToWar then? Only applies when India strikes back?”

“While we were celebrating #Abhinandan’s return, #Pakistan continued to violate the ceasefire in J&K. Two jawans were martyred and three civilians including a 5 year old and a nine month old were killed. And they talk about #peace and #PeaceGestures. Enough of this double speak!”

As most of our analysis focuses on analyzing pro-aggression polarity, and we have no evidence that this type of hashjacking was more prevalent for any given sub-group, we do not expect errors of this type to substantially change our findings. On the contrary, these types of errors suggest that our analysis is a conservative estimate of the level of pro-aggression polarity.

Furthermore, of the 100 annotated data points, in the Pro-India/Pro-Pakistan dimension, none of them were unclassified. In the pro-Aggression/Pro-Peace dimension, 8 were unclassified. 5 of these were labelled as neutral or Can’t determine by annotators, 2 were labeled pro-aggression, and 1 was labelled pro-peace. Thus, this evaluation supports our conclusion in §5, that unclassified tweets primarily contain non-polarized language, and we have no reason to believe that our method fails to classify tweets of either polarity more often.

11 Additional Data Analysis

Sample inferred pro-aggression tweets by BJP members: ““Pakistan will pay heavy price for #Pulwama attack. Free hand given to our forces, Terrorists will pay a heavy price, It’s time to unite for our nation.” - PM Sri @narendramodi, I’am sure our retribution will make more noise than #JihadiTerror #CRPF #KashmirTerrorAttack”
and “#IndiaWantsRevenge — We need to give a befitting reply to Pakistan, we will strike back. But people who support the separatists and maoists and even terrorists, the country does not need them”

Network Type Pro-India Pro-Pakistan Pro-Aggression Pro-Peace
Retweet 3.8 2.8 4.1 1.3
Mention 5 3.4 4 1.6
Reply 0.14 0.11 0.13 0.062
All Communication 5 3.4 5.1 1.6
Table 7: Network densities of different derived networks. All values are ()
Figure 4: Number of Tweets and Retweets by different politicians.