Topical differences between Chinese language Twitter and Sina Weibo

12/22/2015 ∙ by Qian Zhang, et al. ∙ Northeastern University 0

Sina Weibo, China's most popular microblogging platform, is currently used by over 500M users and is considered to be a proxy of Chinese social life. In this study, we contrast the discussions occurring on Sina Weibo and on Chinese language Twitter in order to observe two different strands of Chinese culture: people within China who use Sina Weibo with its government imposed restrictions and those outside that are free to speak completely anonymously. We first propose a simple ad-hoc algorithm to identify topics of Tweets and Weibo. Different from previous works on micro-message topic detection, our algorithm considers topics of the same contents but with different #tags. Our algorithm can also detect topics for Tweets and Weibos without any #tags. Using a large corpus of Weibo and Chinese language tweets, covering the period from January 1 to December 31, 2012, we obtain a list of topics using clustered #tags that we can then use to compare the two platforms. Surprisingly, we find that there are no common entries among the Top 100 most popular topics. Furthermore, only 9.2% of tweets correspond to the Top 1000 topics on Sina Weibo platform, and conversely only 4.4% of weibos were found to discuss the most popular Twitter topics. Our results reveal significant differences in social attention on the two platforms, with most popular topics on Sina Weibo relating to entertainment while most tweets corresponded to cultural or political contents that is practically non existent in Sina Weibo.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

Introduction

China is known for its rich internal Internet ecosystem where Chinese alternatives to most foreign Internet services flourish. This is due not only to cultural differences that prevent foreign websites from gaining a large market share, but also due to stringent government controls that sometimes prevent foreign Internet companies from selling their services or that outright block access to them.

Sina Weibo, as China’s most popular microblogging platform, is perhaps the most visible face of China’s own internal version of the Internet. It is currently used by over users and, similarly to its foreign counterpart Twitter [1] that is widely considered to be a proxy for its users social life and interests [2, 3, 4, 5], it has recently started to draw the attention of researchers everywhere [6, 7, 8, 9].

Sina Weibos origins date back to but it wasn’t until that it rose to prominence. Since July 2009, Twitter has been blocked in China [9], leaving national alternatives such as Sina Weibo as the only alternative. In March 2012, Weibo started requiring its users to associate their profile with their true identity [6] while still giving users the option to display whichever screenname they wished.

The previous works on topic detection on microblogs are usually designed for pre-selected specific topics [10, 11] or only for short-messages with #tags [12]. However, the majority of Tweets are not # tagged [10], and there is few work focusing on automatic topic detection for microblogs. We first propose a simple ad-hoc algorithm to identify topics on microblogs without pre-selection, and cluster those microblogs without #tag into the detected topics. More importantly, without the assumption that a #tag represents a unique topic, our algorithm merges microblogs of the same contents but with different #tags.

Past studies on Chinese microblogging platforms [7, 9] mainly focused on censorship and analyzed deleting practices on microblogs containing censored key words. Others compared the user behaviors, texture features of posts and temporal dynamics of re-posting [1] and an artificially selected categorical events [13] on Sina Weibo and Twitter. There is little research on comparing the collective attention of Chinese microbloggers in a large scale. Here, we take a first step in this direction by proposing an algorithm to model and compare Sina Weibo and Twitter. By contrasting the discussions occurring on these two platforms, we can observe two different versions of chinese culture: people inside China ( of geotagged Weibos are within China) and those outside (

of geotagged Tweets are located outside China). Despite China’s growing global relevance, and due to the complexity of its language, the number of people outside China learning Chinese as a second language is still very small. A recent study estimates that, in

, just over students in American universities were taking Chinese language classes, compared with over studying Spanish [14], indicating that people who Tweet in Chinese outside of China are likely either Chinese expats or from Chinese heritage. While some analyses have been performed on geographically distributed populations speaking the same language  [15], this combination of technically equivalent services serving populations with a similar cultural background that are isolated from each other is unique and provides us with the perfect opportunity to study the cultural differences in the virtual world between Chinese speakers inside and outside China.

In summary, we perform a topical comparison of both Twitter and Sina Weibo. Our results reveal significant differences in social attention distribution across both platforms, with the most popular topics on Sina Weibo relating to entertainment while the most topics in Twitter corresponded to cultural or political contents.

Data Description

We use the dataset of Sina Weibo from Open Weiboscope Data Access [7, 8]. The dataset contains million Weibo posts (Weibo for short) collected over the full course of . The Twitter dataset used in this study was extracted from the raw Gardenhose feed [16], an unbiased sample of of the entire Twitter dataset that provides a statistically significant real time view of all Twitter account activity [17]. To identify Tweets and Weibo in Chinese language, we perform language detection using the “Chromium Compact Language Detector” [18]. See [19] for further details. This way, we collected Million Tweets and Million Weibo in both simplified and traditional Chinese language covering the entire year of . The Sina Weibo dataset also include microblogs which are not accessible to the public, either censored or self deleted. Following [7], we consider weibos deleted by the censorship (with message “permission denied” from API). In total, we considered deleted weibos for our study.

Clustering microblogs into topics

Topic modeling on micro-messages is still challenging due to its inherent sparseness [12] and noise [11]. In this study, we take microblogs with #tags as potential known topics. There are tweets and

weibos with #tags. We build a vocabulary vector space on each microblogging platform with words of high-frequency (high TF-IDF score), and cluster similar #tags into a specific topic. For instance, for the Top

#tags on Sina Weibo and Twitter, we merge them into and topics respectively. For the rest of microblogs without #tags, we assign them to topics that are closest to them in the vocabulary vector space. To reduce statistical fluctuations we restrict our study to the Top topics in each platform. In total,

weibos are classified into popular topics on Weibo and

of tweets discuss popular topics on Twitter. In the remaining part of this section, we briefly describe our algorithm of clustering microblogs into topics.

Preprocessing. We first filter microblogs by removing the words representing short URLs and mentioning other users (“@username”). Filtered microblogs in traditional Chinese are then converted to simplified Chinese with the python-jianfan library [20]. Chinese word segmentation is performed using Jieba [21] and part-of-speech tagging (POS) is performed following [11]. This way each microblog is represented as a set of words tagged as noun, name, location, organization, time, place word, position word or verb.

Vector representation. We merge all microblogs with the same #tag as a document , and calculate its TF-IDF (term frequency–inverse document frequency). For each , we exclude the words with length less than 2 since a single character word in Chinese can be noisy and under-representative, and choose the first words with highest frequency and their TF-IDF weights , and its vocabulary vector can be written as , and where is the number of #tags we select.

Clustering.

Since several similar #tags likely refer to the same topic, we further cluster #tags into topics using hierarchical clustering. In Figure

1, we show the dendrogram for the Top #tags on Sina Weibo platform, calculated using cosine distances in the embedding vector space . Interestingly, most clades are simplicifolious, indicating that distribution of words for each #tag is substantially different from the distribution in others. We observe similar dendrogram for Top #tags on Twitter (figure not shown). Thus, we apply a modified divisive clustering method (Algorithm 1), where we iteratively divide the largest cluster into a small cluster and a large one, until the size of the small cluster is .

Input: The hierarchical cluster
Output: The set of topics
initialization;
;
while  do
       Partition into a larger cluster and a small cluster ;
       ;
       ;
      
Algorithm 1 Merging #tags into topics
Figure 1: Cluster Dendrogram. Cluster dendrogram for Top #tags on Sina Weibo contains a number of simplicifolious clades, indicating most of topics labelled by #tags are distant from each other.

After merging # tags into topics, each topic in vocabulary vector space now is defined as , and . is now the first words with highest frequency in a topic and their TF-IDF weights . The centroids of the final clusters are taken to represent topics in the vector space of each platform. To classify the remaining microblogs on one platform, we measure the cosine distance between the centroid of a topic , and each microblog , . If is smaller than a threshold , we consider the microblog is discussing the topic , shown in Algorithm 2.

Input: The set of topics and a set of microblogs
foreach topic in  do
       computing the centroid in the vocabulary vector space where ;
       label with its # tags clustered from Algorithm 1;
      
foreach microblog in  do
       if  contains # tag  then
             Assign into the topic with # tag ;
            
      else
             set , where ;
             let be the topic for ;
             let be the minimum distance from to any centroid of clustered topic ;
             foreach topic in  do
                   computing the distance ;
                   if  then
                         ;
                         ;
                        
                  
            if  is not None then
                   Assign the microblog into the topic ;
                  
            else
                   Tag the microblog to be ‘unknown topic’ ;
                  
            
      
Algorithm 2 Clustering micrblogs into topics

To determine the threshold , we measure the distribution of distance between a centroid of a topic and microblogs inside that topic. About about of tweets and of weibos have distances less than to their topical centroid. Meanwhile, if we measure distances from a microblog to centroids of other topics, on average only about of microblogs outside a topical centroid have distances less than . Therefore, we use as our threshold.

Results and Discussion

Our analysis aims to compare topical spaces in Chinese language on different microblogging systems. With identified centroids in the vocabulary vector space defined in the last section, we first calculate the distance between the centroids of the Top topics on the two platforms. We define as the cosine distance in the vocabulary vector space between the centroid of topic on Twitter and topic on Sina Weibo. Figure 2

-A shows the cumulative distribution function of distance

for pairs of topical centroids. Surprisingly, only pairs of topics have distance less than . In Figure 2-B, we show the distance between Top topics on Weibo and Top topics on Twitter. Surprisingly, the distance between of pairs of Top topics on the two platforms is larger than , indicating that microbloggers in each platform have significantly different conversation topics and interests.

Figure 2: Distance between popular topics on Twitter and Sina Weibo. (A) The cumulative distribution function of the distance as the cosine distance between the centroid of topic in Top topics on Twitter and topic in Top topics on Sina Weibo. (B) The distance matrix between the centroid of topic in Top topics on Twitter and topic in Top topics on Sina Weibo. The distance refers two exactly similar topics, and indicates that two topics are completely different.

In Table 1, we provide the Top topics in Chinese language on Sina Weibo and Twitter to illustrate the differences. On Sina Weibo, of entire datasets can be classified into top 10 topics ( for the Top ); while on Twitter over all tweets are categorized into top 10 topics ( for Top ). The microblogs on Sina Weibo focus on entertainment (singers, actors and games) and advertising. In contrast, on Twitter, there is no commercial advertisement appearing, and the last two topics are about games. The others are all corresponding to political contents.

Topic Topic
Rank Sina Weibo in English % Twitter in English %
1 三国来了 an online game 0.51 陈光诚
Chen Guangcheng
(a Chinese civil rights activist)
3.56
2 林峰
Raymond Lam
(a singer from Hongkong)
0.39 乌坎 Wukan protest 2.56
3 晚安/早安 good morning/night 0.38 Freetibet Free Tibet 1.62
4 微博客户端 Sina Weibo app 0.36 李旺阳
Li Wangyang (a Chinese
dissident labor rights activist)
0.09
5 搞笑 joke 0.07 温云 @wenyunch 0.97
6 美图秀秀
Meitu (an iOS/Android
app to edit pictures)
0.04 抗暴 Tibetan Uprising Day 0.88
7 有奖转发 re-posting to win a prize 0.04 达赖喇嘛 Dalai Lama 0.68
8 WeicoLomo
An iOS/Android app for
Weibo to record video
0.04 钓鱼岛
Uotsuri Jima
Diaoyu Dao / Diaoyutai
0.66
9 韩庚
Han Geng
(a Chinese singer and actor)
0.03 ipadgame iPad game 0.61
10 新版微博 new version of Sina Weibo 0.02 武士朝代 an Andorid game 0.48
Table 1: Top 10 Topics on Sina Weibo and Chinese language Twitter in .

In the previous section, we have classified weibos and tweets into the topical space in their own vocabulary vector space. For an unclassified weibo or tweet, we calculate its distance to centroids on both platforms, and assign it to the closest topic. Interestingly, we find there are only of tweets correspond to the Top topics on Sina Weibo platform, and only of weibos were discussing the most popular topics on Twitter. Chinese microbloggers speaking the same languages on two platforms share a few social attentions.

We further investigate deleted weibos that were likely censored [7] by checking if they belong to topics which appear on Twitter. In total, deleted weibos can be classified into the Top topics on Twitter. We re-rank the topics in accordance with the frequency of deleted weibos. The Kendall rank correlation coefficient between the top topics for all tweets and for deleted weibos is , with -value . In Table 2, we list Top topics for deleted weibos on Twitter’s vector space. Compared with popular topics on Sina Weibo, the deleted weibos are significanlty more likely to discuss political issues.

rank topic on Twitter in English
1 fb Facebook
2 乌坎 Wukan protest
3 np Now Playing
4 hitbag -
5 AutoShare -
6 GFW Great Firewall
7 HK71 Hong Kong 1 July march
8 bot -
9 A片 Adult movie
10 JapanLife -
Table 2: Top 10 topics in deleted weibos on Twitter’s vector space .

Conclusion

The social attention of online users from the same cultural backgrounds but living in different countries might be different due to the changes of social environments. In this study, we take the first steps toward understanding such differences.

Sina Weibo is used almost exclusively within China while most Chinese language use of Twitter occurs almost exclusively outside Chinese borders. By comparing the most popular topics in these two platforms we can, for the first time, observe how the interests of two populations, with similar cultural backgrounds, differ. Surprisingly, we find that there is very little overlap between the two attention profiles. Weibo users speak mostly about popular culture and games while Twitter users focus mostly on political issues.

The reasons behind this divergence are hard to discern but can likely be attributed to one of two factors: lack of interest for political topics within China or a high degree of self-censorship that prevents Chinese from discussing politics in public. A small indication towards this second hypothesis is the list of topics seen in deleted Weibos (see Table 2) that have higher political content. It is worth to remark that our algorithm of detecting topics still depends on # tags, and some of such # tags may not necessarily be a social topic but likely represent some commercial web/mobile applications. Manual annotations could be included in the future work to improve the topic detection results. Another key datapoint we are missing to fully clarify this question is the number of people who use foreign VPN services as a way of being able to reach Twitter where the discussion is more politically centered. An analysis of this interesting factor will be the subject of future study. The proposed methodology in this paper can be easily applied to any other languages across different online conversation platforms if data are available.

Another possibility worth considering when comparing user behavior across two different platforms are the technical differences between the platforms are not to be excluded. However, the similarity between the two platforms likely minimizes this effect. Indeed, it would be difficult to argue that Twitter is, on technical grounds, any more or less suitable to discussion of the topics listed on the right side of Table than Sina Weibo or vice-versa. A final possibility is simply that different "cultural norms" have emerged in the two platforms [22] with the Sina Weibo community naturally becoming much more focused on pop-culture and entertainment and Twitter becoming more political.

References

  • [1] Q. Gao, F. Abel, G. Houben, and Y. Yu. A comparative study of users’ microblogging behavior on Sina Weibo and Twitter. In User modeling, adaptation, and personalization, pages 88–101. Springer, 2012.
  • [2] D. Bamman, J. Eisenstein, and T. Schnoebelen. Gender identity and lexical variation in social media. J. of Sociolinguistics, 18:135, 2014.
  • [3] O. Phelan, K. McCarthy, and B. Smyth. Using Twitter to recommend real-time topical news. In RecSys’09, page 385, 2009.
  • [4] F. Ciulla, D. Mocanu, A. Baronchelli, B. Gonçalves, N. Perra, and A. Vespignani. Beating the news using social media: the case study of american idol. EPJ Datascience, 1:8, 2012.
  • [5] P.T. Metaxas, E. Mustafaraj, and D. Gayo-Avello. How (not) to predict elections. In IEEE Third Inernational Conference on Social Computing (SocialCom), page 165, 2011.
  • [6] A. Rauchfleisch and Mike S. Schäfer. Multiple public spheres of Weibo: a typology of forms and potentials of online public spheres in China. Information, Communication & Society, 18:139, 2014.
  • [7] K. Fu, C. Chan, and M. Chau. Assessing censorship on microblogs in China: discriminatory keyword analysis and the real-name registration policy. Internet Computing, IEEE, 17(3):42–50, 2013.
  • [8] K. Fu and M. Chau. Reality check for the Chinese microblog space: a random sampling approach. PloS one, 8(3):e58356, 2013.
  • [9] D. Bamman, B. O’Connor, and N. Smith. Censorship and deletion practices in Chinese social media. First Monday, 17(3), 2012.
  • [10] Swit Phuvipadawat and Tsuyoshi Murata. Breaking news detection and tracking in twitter. In Web Intelligence and Intelligent Agent Technology (WI-IAT), 2010 IEEE/WIC/ACM International Conference on, volume 3, pages 120–123. IEEE, 2010.
  • [11] G. Li, K. Meng, and J. Xie. An improved topic detection method for chinese microblog based on incremental clustering. Journal of Software, 8(9):2313–2320, 2013.
  • [12] O. Tsur, A. Littman, and A. Rappoport. Efficient clustering of short messages into general domains. In ICWSM, 2013.
  • [13] Xin Shuai, Xiaozhong Liu, Tian Xia, Yuqing Wu, and Chun Guo. Comparing the pulses of categorical hot events in twitter and weibo. In Proceedings of the 25th ACM conference on Hypertext and social media, pages 126–135. ACM, 2014.
  • [14] N. Furman, D. Goldberg, and N. Lusin. Enrollments in Languages Other Than Engish in United States Institutions of Higher Education, Fall 2009. Technical report, Modern Language Association, 2010.
  • [15] B. Gonçalves and D. Sánchez. Crowdsourcing dialect characteriation through twitter. PLoS One, 9:E112074, 2014.
  • [16] J. Ratkiewicz, M. Conover, M. Meiss, B. Gonçalves, S. Patil, A. Flammini, and F. Menczer. Truthy: mapping the spread of astroturf in microblog streams. In WWW, pages 249–252. ACM, 2011.
  • [17] Guide to the Twitter API Part 3 of 3: An Overview of Twitters Streaming API. http://blog.gnip.com/ tag/gardenhose/, 2014.
  • [18] M.M. Candless. Chromium Compact Language Detector. http://code.google.com/p/chromium-compact-language-detector/, 2012.
  • [19] D. Mocanu, A. Baronchelli, N. Perra, B. Gonçalves, Q. Zhang, and A. Vespignani. The Twitter of Babel: Mapping world languages through microblogging platforms. PloS one, 8(4):e61981, 2013.
  • [20] Jianfan. https://code.google.com/p/python-jianfan/, 2013.
  • [21] Jieba: Chinese word segmentation module. https://github.com/fxsjy/jieba, 2014.
  • [22] Tao Zhou. Understanding online community user participation: a social influence perspective. Internet Research, 21(1):67–81, 2011.