Mining the online infosphere: A survey

01/02/2021 ∙ by Sayantan Adak, et al. ∙ 0

The evolution of AI-based system and applications had pervaded everyday life to make decisions that have momentous impact on individuals and society. With the staggering growth of online data, often termed as the Online Infosphere it has become paramount to monitor the infosphere to ensure social good as the AI-based decisions are severely dependent on it. The goal of this survey is to provide a comprehensive review of some of the most important research areas related to infosphere, focusing on the technical challenges and potential solutions. The survey also outlines some of the important future directions. We begin by discussions focused on the collaborative systems that have emerged within the infosphere with a special thrust on Wikipedia. In the follow up we demonstrate how the infosphere has been instrumental in the growth of scientific citations and collaborations thus fueling interdisciplinary research. Finally, we illustrate the issues related to the governance of the infosphere such as the tackling of the (a) rising hateful and abusive behavior and (b) bias and discrimination in different online platforms and news reporting.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Online infosphere111https://en.wikipedia.org/wiki/Infosphere is the term corresponding to the Internet becoming a virtual parallel world formed from billions of networks of artificial life at different scales ranging from tiny pieces of software to massive AI tools running a factory or driving a car. The motivations for this are diverse, seeking to both help mankind and harm it.

In this article, we shall attempt to portray some of the areas that are increasingly gaining importance in research related to the evolution of this infosphere. In particular, we would begin with infosphere as a collaborative platform, Wikipedia being the prime point of discussion. As a next step we would discuss how the infosphere has influenced the evolution of scientific citations and collaborations. Finally, we shall outline the emerging research interest in the governance of this infosphere to eradicate discrimination, bias, abuse and hate.

1.1 Infosphere as a collaborative platform

The infosphere hosts numerous collaborative platforms including question answering sites, folksonomies, microblogging sites and above all encyclopedias. In this survey we shall focus on Wikipedia which is one of the largest online collaborative encyclopedia. We shall primarily discuss two of the most important aspects of Wikipedia – (a) the quality of an article and its indicators and (b) the collaboration dynamics of Wikipedia editors who constitute the backbone of this massive initiative. Under the first topic we shall identify the different features of an article like its language, structure and stability as well as their quality [150, 53, 120]. We shall further summarise attempts that have been made to automatically predict the quality of an article [52, 91]. Within the second topic we shall briefly describe various issues related to the community of editors including anomalies, vandalism and edit wars[71, 135]. Finally, we shall talk about ways to enabling retention of editors on the platform [54, 104, 132].

1.2 Infosphere shaping scientific citations and collaborations

Citations play a crucial role in shaping the evolution of a scientific discipline. With an exponential growth of research publications in various disciplines it has become very important for researchers and scientists to grasp different concepts within a short period of time. We would explore how the infosphere has influenced the growth and interaction of different scientific disciplines over the period of last few years by investigating several different aspects of citation networks. Our survey includes – (a) a detailed account of how the basic sciences and the computer sciences have interacted with each other over the years resulting in an interdisciplinary research landscape [58, 103],(b) the temporal dynamics of citations [129], (c) ways for assessment of article quality, and finally (d) a brief account of anomalous citation flows.

1.3 Governance of the infosphere

The stupendous growth of the infosphere has resulted in the emergence of various online communities that have massively started infusing bias, discrimination, hatred and abuse often resulting in violence in the offline world. In this segment, we shall primarily focus our discussion on the following topics – (a) analysis, spread, detection and mitigation of online hate speech and (b) biases that manifest across news media and in traditional recommendation systems. Within the first topic we shall motivate the need to tackle online hate speech by citing some of the adverse consequences of the same. In particular, we shall see how unmoderated hate speech spreads in a social network [93, 94], what are the challenges to automatically detect online hate speech [49, 2] and the possible techniques to combat this problem [95, 96]

. Within the second topic we shall discuss two important forms of biases. The first one corresponds to political biases that manifest due to the massive production of unverified (and in many cases false) news generated in the form of news/blog/tweets etc. We shall also discuss the difficulties faced by modern machine learning techniques in preventing the infusion of such biases. The second one narrates the idea of formation of

filter bubbles [107] in traditional recommendation systems followed by a discussion on the need to systematically audit such systems [28, 27].

2 Wikipedia as a collaborative platform

Wikipedia models a hypertext collaborative platform along with an open and free knowledge base catering a large variety of information ranging from history, arts, culture, politics to science, technology and many more fields. Being one of the most widely viewed sites (within top ten) in the world since 2007, it spans over 208 languages with a copious amount of articles in each edition. For example, the English-language Wikipedia, the largest in volume, contains more than 6 million articles as of February, 2020. Owing to the collaborative nature and an open-access policy announced by Wikipedia as “anyone can edit”, a number of challenges have cropped up in maintaining the veracity-quality balance of the content. Although Wikipedia has enforced several rules and strict administrative policies to protect the encyclopedia from malicious activities, a lack of authoritative vigilance prohibits its trustworthiness in academics. In contrast, Wikipedia’s structured, complete and detailed evolution history receives increasing attention of the research community in discovering automated solution (e.g. bots, software, API etc.) to meet the goal of quality management.

2.1 Quality

The elementary purpose of Wikipedia is free, unbiased, accurate information curation. To achieve this objective Wikimedia foundation which is the governing body of the platform, has developed labyrinth of guidelines which editors are expected to follow so that highest encyclopedic standards are maintained. These guidelines also enhance accessibility of the Wikipedia articles to a broad community of netizens. We enumerate these guidelines into three categories as discussed below.

2.1.1 Article language

Expressions describing a subject should be neutral. Promotion bearing words such as renowned, visionary, iconic, virtuoso etc. should not be used. Subject importance should be demonstrated using facts and attribution222https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Words_to_watch Prose should have active voice. Jargon needs to be elaborated or substantiated with reference. Any effort to propagate myth or contentious content should be curtailed. An example for this is addition of prefix pseudo or suffix -gate which encourages the reader to assume that the subject is factitious or scandalous respectively. Euphemisms (e.g., passed away, collateral damage and cliches (e.g., lion’s share, tip of the iceberg) disallows presentation of prose directly and hence is restricted. Any unnecessary emphasis in the form of italics, quotations etc. is discouraged. For a complete list of details of content guidelines for the English Wikipedia we refer the reader to the Wikipeida manual of style333https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style.

2.1.2 Article structure

These guidelines include proper formatting of the Wikipedia article in terms of section headings, infobox, article name, section organisation etc. The lead section should not be of arbitrary length. The following sections should not be exorbitant in size and bigger sections should be broken into coherent smaller sections. Another requirement is proper positioning of the images with captions and references. In order to alleviate manual labor in improving article structure there have been some automated approaches leveraging advances in machine learning techniques [63].

2.1.3 Article stability

These guidelines denote stability of the article, i.e., the respective article should not be subject of frequent edit wars. There should not be abusive language exchange among editors and discussions toward improving article quality should organically reach consensus. This is the most difficult objective in collaborative content creation and generally the onus lies in the hands of senior level editors and moderators for smooth conflict arbitration.

2.1.4 Peer review framework

Although Wikipedia has grown significantly in terms of volume and veracity over the last decade, the quality of articles is not uniform [138]. The quality of Wikipedia articles is monitored through a rating system where each article is assigned one of several class indicators. Some of the major article categories are FA, GA, B, C, Start and Stub. Most complete and dependable content is annotated by an FA (aka featured article) tag while the lowest quality content is annotated with a Stub tag. The intention behind this elaborate scheme is to notify editors regarding current state of the article and extent of effort needed for escalating to encyclopedic standards444wiki/Wikipedia:WikiProjectWikipedia/Assessment. The editors are expected to rigorously follow the aforementioned guidelines. As has been evident from the guidelines, they are circuitous and often require experience for implementation. Such strict policy adherence have also been sometimes a barrier for onboarding of new editors on Wikipedia which has led to the decline of newcomers over the past decade [132, 55]. Since it is nontrivial to discern qualifying differences between articles manually, it has given rise to the emergence of automated techniques using machine learning models.

2.1.5 Computational methods for quality prediction

Automatic article assessment is one of the key research agendas of the Wikimedia foundation555www.mediawiki.org/wiki/ORES. One of the preliminary approaches [56] seeking to solve this problem extracted structural features such as presence of infobox, references, level 2 headings etc. as indicators of the article quality. [26]

proposed the first application of deep neural networks into quality assessment task where they employed distributional representation of documents 

[80] without using manual features. The authors in [123]

introduce a hybrid approach, where textual content of the Wikipedia articles are encoded using a BILSTM model. The hidden representation captured by the sequence model is further augmented with handcrafted features and the concatenated feature vector is used for final classification.  

[151] is an edit history based approach where every version of an article is represented by dimensional handcrafted features. Hence, an article with versions will be represented by matrix. This length sequence is passed through a stacked LSTM for final representation used in classification.  [124] proposed a multimodal information fusion approach where embeddings obtained from both article text as well as html rendering of the article webpage is used for final classification.  [52] proposed the first approach which incorporates information from three modes for quality assessment, i.e., article text, article image and article talk page. [52] obtains improvement over [125] approach and achieves the SOTA result. A complementary direction of exploration has been put forward by [84, 31] where correlation between article quality and structural properties of co-editor network and editor-article network has been exploited. An orthogonal direction of research looks into edit level quality prediction which is a fine-grained approach toward article content management [120].

2.2 Collaboration among editors

The workhorse behind the success story of Wikipedia is the large pool of its voluntary editors; an encouragement toward global collaboration influences people to contribute on almost all wikipages. These group of people maintain Wikipedia pages behind the scenes which includes creating new pages, adding facts and graphics, citing references, keeping the wording and formatting appropriate etc. to lead the articles to the highest level of quality. The achievement of any open collaborative project is hinged on the continued and active participation of its collaborators, and hence, Wikipedia needs to manage its voluntarily contributing editor community carefully. In the days of extreme socio-cultural polarization, algorithmically crafted filter bubbles and fake information represented as facts, editors are highly motivated to contribute to the largest non-biased knowledge sharing platform although their works are not financially compensated most of the times [88]. In these lines there have been several works [145, 104], which attempt to understand the dynamics of interaction behaviours of the community in sustaining the health of Wikipedia.

2.2.1 Anomaly

While investigating the editing behaviours of editors in general context, researchers have found out a taxonomy of semantic intentions behind the edits, and conflicts and controversy are inherent components of the classification. Wikipedia owes its success for several reasons and openness is one of those pillars. Sometimes, the very openness misguides editors to violate Wikipedia’s strict guidelines of the neutral-point of view (NPOV), and their disruptive edits cause various kinds of anomalies. We describe the two dominant disputes, produced by the damaging edits as follows.

Vandalism: With the freedom of editing anything by anyone, Wikipedia has to struggle in stopping the malicious practice of contaminating articles by bad faith edits intentionally. The popular pages like famous celebrities, controversial topics etc. become the frequent targets of vandalism where vandals try to mislead the readers by addition, deletion or modification, which can be termed as hoax. Wikipedia has enforced several strict policies such as blocking and banning Vandals (registered / unregistered editors), patrolling recent changes by adding watch-lists, protecting articles (ex, semi-protected pages) from new editors, random IP addresses etc. In addition to the administrative decisions, bots 666https://en.wikipedia.org/wiki/User:ClueBot_NG are employed to detect and revert the vandalism automatically and finally warn the editors without human intervention. Researchers have proposed various automated ways [131, 71], i.e., the state-of-the-art techniques based on machine learning [75, 133] and deep neural methods [92, 135] in preventing Wikipedia from vandalism.

Edit war: Apart from the intended malpractice of vandalism, editors often engage themselves in disagreement which further influence them to override each other’s contribution instead of dispute resolution. Any such actions violating the three-revert rule777https://en.wikipedia.org/wiki/Wikipedia:Edit_warring is coined as edit warring in Wikipedia and it promotes a toxic environment in the community. Ultimately in the long-term, the integrity of the encyclopedia will be affected significantly by the damaging effects of edit wars [117]. Although Wikipedia encourages editors to be bold, in contrast a constant refusal to get the point is also not entertained.

2.2.2 Retention of editors

Historically, Wikipedia managed the numbers of its volunteers quite successfully; however, experts [54] note that it is at the danger of sharp decline of its active editors due to the lack of the socialization effort. Editors may choose to leave the platform for personal reasons as well as for their disagreement/conflict with their fellow editors. The damage is happening in both ways - when new editors fail to inherit the rules and policies they easily become upset and leave eventually. Experienced editors, on the other hand, can get discouraged because of the continuous upgradation of policies to retain newcomers, or even for the nuisances by the newbies. Two way effort are being taken to combat with this problem – researchers are coming up with various approaches (see [101, 102, 147] and the refereces therein) while Wikipedia itself is running several wikiprojects888https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Editor_Retention999https://en.wikipedia.org/wiki/Wikipedia:Expert_retention to proactively retain its contributors.

Future directions: Due to the enormous volume of data publicly available from various multilingual wikiprojects, several interesting future directions can be explored. One of the directions is combating repeat abusers who add malicious content annonymously after being blocked through sockpuppetry [90] or collusion. A large volume of effort has been invested in understanding editor behaviour; however, similar large scale exploration need to be done on understanding readers. A promising start in this direction includes the following  [65, 115]. However, further work needs to explore the interplay between editors and readers and how these two stakeholders can forge a partnership in mutually beneficial fashion. We refer to the mediawiki research index 101010https://meta.wikimedia.org/wiki/Research:Index for a comprehensive take on this emerging research scope.

3 Recent trends in citation dynamics

Research on citation network have always remained essential in solving various problems such as predicting emerging topics, early citation prediction, modelling evolving citation networks. Citation network is a directed graph where nodes could be authors/papers/journals and edges are the citation flows (weighted/unweighted) from one node to another node. Using citation networks, one can predict which field/topic could be the ‘most attractive ones to work on’ in the immediate future years. The research dynamics in various fields over the years can be analyzed with the help of the underlying citation network. Various studies uncover the chances of the manifestation of certain new fields/sub-fields by investigating the citation flows from papers of one field to the papers of another field over the time. In recent years, citation count prediction task plays an important role in fund allocations and rewards. Researchers are also interested in building models for automatic citation recommendation while drafting an article. Apart from these, some anomalous practices in exchanging citations have been exposed in the late ’90s. Now, such malpractices are becoming more common among the researchers/journals (mostly low ranked). In the rest of this section, we shall discuss each of the above issues in details.

3.1 Interdisciplinary research in terms of citation interactions

Various research questions such as “which field will collaborate with which field in future?”, “Which field will receive more citations from recently published papers?”, etc., can be addressed with the help of the underlying citation networks among the articles. Nowadays, research is performed by combining the ideas from multiple disciplines. In [58] the authors have analyzed the interdisciplinarity among the two basic science fields – Mathematics and Physics and one fast growing field – Computer Science. Further they observe how the citation from papers of one discipline flows to the papers of another discipline over the years. They observe that in initial years huge amount of citation flows from Physics to Mathematics and vice versa. Over the years, Computer Science started gaining citation from Mathematics. In the recent years, both the basic science fields tend to massively cite papers from Computer Science. They observe how popularity of some topics decreases over the time. They found that the Computer Science papers mostly cites the quantum physics sub-field for long time span. In late ’90s, Physics mostly cites information theory papers of Computer Science but in recent times it mostly cites papers from machine learning and social & information networks domain.

Further, interdisciplinarity has been studied in different fields including biology [103], mathematics [103], cognitive science [9, 76], social science [109], humanities [109]. Various studies  [7, 121, 9] have attempted to propose novel metrics to measure the degree of interdisciplinarity based on researchers’ scientific impact, collaborator’s knowledge, publication history, etc. In addition, metric for measuring interdisciplinarity of an article has been proposed [9] where authors’ research area, publications in different domains have been used to define the metric. A study [126] has been carried out to analyze the interdisciplinarity nature of highly cited articles from Thomson Reuters’ databases published in between 1900-2012 years.

3.2 Analyzing and modelling the citation dynamics

Several studies have been carried out in the past to model the temporal dynamics of citation networks. In order to model the temporal dynamics of citation networks, researchers traditionally used preferential attachment [1] and copying based [74] models.

In [137], the authors investigated the citation behavior of older papers in various fields. They concluded that older articles receive more citations over the years. It is observed that in 2013, 36% of citations flowed toward the at least ten years old papers. However, a re-investigation of this study showed that the observations are only partly true since the authors did not take into account the accelerating volume of publications over time. In order to tackle the tug-of-war between obsolescence and entrenchment, recently, in [129], the authors proposed a complex model based on the idea of relay-linking where the older article relays a citation to a recently published article. This model has very less number of parameters and fits with the real data much better than the traditional models. Yet another novel citation growth model called RefOrCite [106] have been proposed recently where the authors allow copying from the references (out-edges) and citations (in-edges) of an article (as opposed to only references in the traditional setup). It is observed that RefOrCite model fits well with real compared to the previous models.

Citation count prediction

Predicting future impact of scientific articles is important for making decision in fund allocation (by funding agencies), recruitment etc. There are various works [82, 143, 149, 83]

that have been carried out in the past to automatically estimate the citation count of scientific articles. In this article, we shall mainly focus on the recent literature. In 2015, the authors in 

[82] proposed Trend-based Citation Count Prediction (T-CCP) model where the model first would first learn the type of the citation trends of the articles and then predict the citation count for that trend. All the articles were categorized into five citation trend categories based on the “burst” time (“burst time” is the time when the paper gets maximum citations) – early burst, middle burst, late burst, multi bursts, and no bursts. Two types of features have been used – (a) publication related features like author centric features (i.e., h-index, number of papers published, citation count, number of collaborators), publication venue (average citation count, impact factor etc.) and (b) reinforcement features which are the graph based features (i.e., PageRank, HITS etc.) calculated from weighted citation network among authors. In their model, they mainly use SVR and SVM (LibLinear) for citation count prediction and classification task respectively. In paper [128], the authors found that the knowledge gathered from citation context within the article could help to predict future citation count. Number of occurrences of the citations for a paper within the article and the average number of words in citation context, have been derived from citation context knowledge. Further they categorized the articles into six citation profiles (PeakInit, PeakMul, PeakLate, MonDec, MonIncr, Oth) and found that the above two citation context based features are able to nicely distinguish these six categories. In [127], the authors observed that the long term citation of an article depends on the citations it receives in the early years (within one or two years from its publication date). The authors who cite an article in its early years are called early citer. Early citers based on whether they are influential or not affect the long term citation count of the article. In most cases, influential authors negatively affect the long term citation of an article. In [143], the authors have proposed a novel point process method to predict the citations of individual articles. In their approach they tried to capture two properties – the “rich gets richer” effect and the recency effect. The authors in [149] used four factors – intrinsic quality (citation count) of a paper, aging effect, Mathew effect and recency effect to derive a model called long term individual level citation count prediction (LT-CCP). In this model they mainly use RNN with LSTM units. It is observed that LT-CPP model achieves better performance than existing models. Authors in [83]

proposed a neural model for predicting citation count with the help of peer review text. They mainly learn two deep features – (a) the abstract-review match mechanism (in order to learn the abstract aware review representation) and (b) the cross review match from peer review text.

3.3 Citation recommendation

Often new researchers face difficulties in finding appropriate published research papers while exploring the domain literature and citing published papers. Citation recommendation is a technique that recommends appropriate published articles for the given text/sentence. The sentences present around the reference (placeholder) are called context sentences. Citation recommendation task can be divided into two parts – (i) local citation recommendation, and (ii) global citation recommendation. In case of local citation recommendation, only the context sentences are used. In case of global citation recommendation, the whole article is used as input and the system outputs a list of published papers as output. In cite [10] the authors proposed a model for the global citation recommendation task where they embedded the textual information (i.e., the title and the abstract) of the candidate citations in a vector space and considered the nearest neighbors as the candidate citations for the target document. Further, re-ranking of the candidate citations was done. They used DBLP (50K articles having an average citation of 5 per article) and PubMed (45K articles with average citation of 17 per article) datasets and also introduced a new dataset OpenCorpus (7 million articles) in the paper. They showed that their model achieved state-of-the-art performance without using metadata (authors, publication venues, keyphrases). In paper [64]

, the authors proposed a deep learning model (consists of context encoder and citation encoder) and used a dataset 

[59] for context aware citation recommendation. Pre-trained BERT [34] model has been used in order to learn the embedding of the context sentences. GCN has been employed to learn the citation graph embedding from the paper-paper citation graph. They mainly revised two existing datasets – AAN and FullTextPeerRead (revised version of PeerRead). They showed that their model performed three times better than the SOTA approaches (CACR etc.). The authors in [110]

proposed a novel method – ConvCN – based on the citation knowledge graph embedding.

3.4 Detection of anomalous citation flows

Various anomalous citation patterns have been found to emerge over the years. Various ways of maliciously increasing one’s citation are through self-citations, citation stacking among journals, and citation cartel. Nowadays, authors are more concerned about their position in academia, publication pressure etc. and this leads to most of them adopting unfair means to increase their citation. Citation cartel is one of the anomalous citation patterns which was first reported in late ’90s111111https://science.sciencemag.org/content/286/5437/53. Citation cartel is formed by a group of authors/editors/journals where they cite each other heavily for mutual benefit. The relationship in citation cartel could be author-author, editor-author, journal-journal etc. There are a few cases found where the journal’s impact factor increases rapidly due to this anomalous behavior. Cell Transplantation121212https://www.cognizantcommunication.com/journal-titles/cell-transplantation is a medical journal whose impact factor rapidly increased between 2006 and 2010 (3.48 to 6.20). After investigation carried out by JCR publisher, it was found that one review article published in this journal Medical Science Monitor131313http://www.medscimonit.com/ cited almost 91% papers published in Cell Transplantation from the time bucket 2008–2009. It was found that the impact factor of the journal Cell Transplantation was calculated based on this time bucket141414https://scholarlykitchen.sspnet.org/2012/04/10/emergence-of-a-citation-cartel/. Surprisingly, the authors (three out of four) are from the editorial board of this journal. In cite [40] the authors tried to detect citation cartels. They defined a citation cartel as a group of authors citing each other excessively than they do with other authors’ works in the same domain. They observed that there could be multiple reasons like academic pressure, “publish or perish” concept in academia, fear of losing job, scientific competition etc. behind establishing such citation cartels. It was observed that such unfair means are mostly adopted by low ranked researchers [41]. In their work, they prepared a multilayer graph where they include paper-paper citation network (directed graph), authors’ collaboration network and authors’ citation networks (weighted directed graph). Finally, citation cartel has been captured from the authors’ citation network. Cartels have been discovered by using Resource Description Framework (RDF) and RDF query language and some threshold has been declared to identify the existence of citation cartel among authors. The authors in [72] proposed a novel algorithm – Citation Donors and REcipients (CIDRE) to detect the citation cartel among the journals that cite each other disproportionately to increase the impact factor of the journal. CIDRE algorithm first distinguishes between the normal and malicious citation exchange with the help of few parameters. These parameters are similarity in research areas, citation inflow and outflow. A weighted citation network among 48K journals was constructed from the dataset collected from MAG151515https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/. With the help of the algorithm, more than half of the malicious journals were detected (those were actually suspended by Thomson Reuters) in the same year. In addition, CIDRE algorithm detected few malicious journal groups in 2019 whose journals received 30% of its in-flow citation from the journals in the same group. Such anomalous citations help to grow the impact factor of the journals over the years. In [62], the authors studied how malicious journals are increasing in the Indian research community and avoiding proper rules and regulations. The analysis has been carried out on Indian publishing group OMICS (considered as predatory by the research community). Surprisingly they observed that such malicious journals share very similar characteristics with various reputed journals.

Future directions: In order to gather more citations, malpractices among the journals are rapidly increasing. More research is required to build a mechanism which can automatically predict those (predatory) journals (depending on the topics of the journal). In case of citation recommendation, there is a need for improving the recommendation system such that the system is able to recommend papers that are conceptually similar or exhibit conflicting claims [42]. Also, prioritizing the citation recommendation would be another help to maintain the page limit given by many conferences [42].

4 Governance

As noted in the introduction, this section is laid out into two major parts. The former part centers around the spread, automatic detection and containment of hate speech. The latter part deals with bias in media outlets and online recommendation platforms.

4.1 Hate speech

4.1.1 Spread

The Internet is one of the greatest innovations of mankind which has brought together people from every race, religion, and nationality. Social media sites such as Twitter and Facebook have connected billions of people161616https://techcrunch.com/2018/07/25/facebook-2-5-billion-people and allowed them to share their ideas and opinions instantly. That being said, there are several ill consequences as well such as online harassment, trolling, cyber-bullying, and hate speech.

The rise of hate speech: Hate speech has recently received a lot of research attention with several works that focus on detecting hate speech in online social media [29, 33, 4, 119, 73]. Even though several government and social media sites are trying to curb all forms of hate speech, it is still plaguing our society. With hate crimes increasing in several states171717http://www.aaiusa.org/unprecedented_increase_expected_in_upcoming_fbi_hate_crime_report, there is an urgent need to have a better understanding of how the users spread hateful posts in online social media. Companies like Facebook have been accused of instigating anti-Muslim mob violence in Sri Lanka that left three people dead181818https://www.theguardian.com/world/2018/mar/14/facebook-accused-by-sri-lanka-of-failing-to-control-hate-speech and a United Nations report blamed them for playing a leading role in the possible genocide of the Rohingya community in Myanmar by spreading hate speech191919https://www.reuters.com/investigates/special-report/myanmar-facebook-hate. In response to the UN report, Facebook later banned several accounts belonging to Myanmar military officials202020https://www.reuters.com/article/us-myanmar-facebook/facebook-bans-myanmar-army-chief-others-in-unprecedented-move-idUSKCN1LC0R7 for spreading hate speech. In the recent Pittsburgh synagogue shooting212121 https://en.wikipedia.org/wiki/Pittsburgh_synagogue_shooting, the sole suspect, Robert Gregory Bowers, maintained an account (@onedingo) on GabLABEL:https://gab.com/ and posted his final message before the shooting222222https://www.independent.co.uk/news/world/americas/pittsburgh-synagogue-shooter-gab-robert-bowers-final-posts-online-comments-a8605721.html. Inspection of his Gab account shows months of anti-semitic and racist posts that were endorsed by a lot of users on Gab.

Understanding the spread of hate speech: We perform the first study which looks into the diffusion dynamics of the posts by hateful users in Gab [93]. We choose Gab for all our analysis. This choice is primarily motivated by the nature of Gab, which allows users to post content that may be hateful in nature without any fear of repercussion. This provides an unique opportunity to study how the hateful content would spread in the online medium, if there were no restrictions. To this end, we crawl the Gab platform and acquire 21M posts by 341K users over a period of 20 months (October, 2016 to June, 2018). Our analysis reveals that the posts by hateful users tend to spread faster, farther, and wider as compared to normal users. We find that the hate users in our dataset (which constitute 0.67% of the total number of users) are very densely connected and are responsible for 26.80% of posts generated in Gab.

We also study the temporal effect of hate speech on the users and the platform as well [94]

. To understand the temporal characteristics, we needed data from consecutive time points in Gab. As a first step, using a heuristic 

[98], we generate successive graphs which capture the different time snapshots of Gab at one month intervals. Then, using the DeGroot model [32], we assign a hate intensity score to every user in the temporal snapshot and categorize them based on their degrees of hate. We then perform several linguistic and network studies on these users across the different time snapshots. We find that the amount of hate speech in Gab is consistently increasing. This is true for the new users joining as well. We further find that the recently joining new users take much less time to turn hateful as compared to those that joined at earlier time periods. In addition, the fraction of users becoming hateful is increasing as well. Also, we find that the language used by the community as a whole is becoming more correlated with that of the hateful users as compared to the non-hateful ones. The hateful users also seem to be playing a pivotal role from the network point of view.

4.1.2 Detection

Due to the massive scale of online social media, methods that automatically detect hate speech are required. In this section, we explore a few of the methods that try to automatically detect hate speech. Some of the approaches utilize Keyword-based techniques, machine learning models, etc. Perceiving the right features for a classification problem can be one of the challenging tasks when using machine learning. Though surface-level features, such as bag of words, uni-grams, larger n-grams, etc. 

[18, 136] have been used for this problem, since hate speech detection is usually applied on small pieces of text, one may face a data sparsity problem. Lately, neural network based distributed word/paragraph representations, also referred to as word embeddings/paragraph embeddings [36] have been proposed. Using large (unlabelled) text corpus, for each word or for a paragraph a vector representation is induced [81] that can eventually be used as classification features, replacing binary features indicating the presence or frequency of particular words.

Hate speech detection is a task that cannot always be solved by using only lexicon based features/ word embedding. For instance,

‘6 Million Wasn’t Enough‘ may not be regarded as some form of hate speech when observed in isolation. However, when the context is given that the utterance is directed toward Jewish people who were killed in the holocaust by white supremacists and Neo Nazis232323https://www.vice.com/en/article/y3z9aj/a-national-guard-twitch-streamer-said-6-million-wasnt-enough-on-stream, one could infer that this is a hate speech against Jews. The above example shows us whether a message is hateful or not can be highly dependent on world knowledge. In [46], the authors annotated context dependant hate speech and showed that incorporating context information improved the overall performance of the model.

Apart from world knowledge, meta-information (i.e., background information about the user of a post, number of posts by a user, geographical origin) can be used as a feature to improve the hate speech detection task. Since the data commonly comes from the online social media platform, variety of meta-information about the post can be collected while crawling the data. A user who is known to post mostly hateful content, may do so in the future. Existing research [93] has found that, high number of hateful messages are generated from less number of users. It has been also observed that men are more likely to spread hate speech than women [139]. Also, the number of profane words in the post history of user has been used as a feature for hate speech classification task [24].

Nowadays the number of posts which consists of images, audios, and video content are getting shared more in Social media platforms. In [49], the authors have explored the textual and visual information of images for the hate speech detection task.

Most of the methods that have been explored earlier were supervised and heavily dependant on the annotated data. Off-the-shelf classifiers such as logistic regression, support vector machines have been extensively used. Recently, deep neural models are being extensively used for the classification task. In 

[2], the authors explored several models such as CNN-GRU, BERT, mBERT for the classification task in 9 languages and observed that with increasing training data, the classifier performance increased. Another key observation they made was that the relative performance seemed to vary depending on the language and the model. While better models for hate speech detection are continually being developed, little study has been done to understand the bias and interpretability aspect of hate speech. In [97], the authors release a benchmark dataset which could be used to study this. In their work, they show that the models that perform very well in classification do not score high on explainability metrics.

4.1.3 Counter speech

After detection of hate speech, we need proper mitigation strategies to stop it from becoming viral. The current methods largely depend on blocking or suspending the users, deleting the tweets etc. This is performed mostly by moderators which is a tedious task for them given the information rate. Many companies like Facebook have started to automate this process but both these methods have the risk of violating free speech. This work [69] identifies various pitfalls with respect to trust, fairness and bias of these algorithms. A more promising direction could be to counter with speech which are popularly known as counter speech. Specifically, counter speech is a direct non-hostile response/comment that counters the hateful or harmful speech [116]. The idea that ‘more speech’ is a remedy for dangerous speech has been familiar in liberal democratic thought at least since the U.S. Supreme Court Justice Louis Brandeis declared it in 1927. There are several initiatives with the aim of using counter speech to tackle hate speech. For example, UNESCO released a study [44] titled ‘Countering Online Hate Speech’, to help countries deal with this problem.

The frameworks for mitigating hate speech using counter speech involves two school of thoughts. One of them will be to develop fully automatic counter speech generation system, which can output contextually relevant counter speech given a hate speech. Since generating contextual replies to a text is still a nascent area in natural language processing, generating counter speech is expected to be further difficult for AI systems due to the variety of socio-political variables present in them. Hence, a more practical approach could be to find a task force of moderators who can suitably edit system generated counter speech for large scale use.

One of the earliest computational studies attempted to identify hate and counter users on Twitter and further observe how they interact in online social media [95]. The authors created a small dataset of hate-counter reply pairs and observed how different communities responded differently to the hate speech targeting them. The paper further tried to build a machine learning model to classify a user as hate or counter. A followup study [96] on YouTube comments was conducted further to understand how different communities attempted to respond to hate speech. Taking the YouTube videos that contain hateful content toward three target communities: Jews, African-American (Blacks) and LGBT, the authors collected user comments to create a dataset which contained counter speech. The dataset is rich since it not only has the counter/non-counter binary classes but also a detailed fine-grained classification of the counter class as described in [142] with a slight modification to the ‘Tone’ Category. The authors observed that the LGBT community usually responded to hate speech via “humour” whereas the Jew community used messages with a “positive tone”. The work further adds a classification model which can identify a text as counter speech and its type. The authors in [111]

generated a large scale dataset having hate and their counter replies. They further used this dataset for counter speech generation using seq2seq models and variational autoencoders. One of the limitations of this paper was that the counter speech data annotated through crowd-sourcing were very generic in nature. Hence, a later work 

[21] took help from experts from and NGO to curate a dataset of counter speech toward Islamophobic statements from the social media. While this dataset provides diverse and to the point reply to hate speech, it largely depends on the experts availability. These challenges were compiled in [134], where the authors showed how data collection and counter speech generation is dependant on the assistance from the experts. The former paper also highlighted the weakness of the generation models with around 10% of the automatic responses being proper response to the given hate speech. This reinstates the fact that current generation systems are not capable of understanding the hidden nuances required to generate proper counter speech.

Another important question that lurks around in the research community is the “effect of counter speech”. While in the case of banning or suspension the effect, i.e., removal of tweet/user is visible, the effect of counter speech is rather subjective in nature. In a recent work [47], the authors used a classifier to identify  100,000 hate-counter speech pairs and found reduction of hate speech due to the organised counter speech. While whether this solely was caused by the counter speech or some other reason is still an open question to the research community.

Overall the counter speech research shows promise but has several unanswered questions about its data collection, execution and effect. Nevertheless, fighting hate speech in this way has some benefits: it is faster, more flexible and responsive, capable of dealing with extremism from anywhere and in any language and it does not form a barrier against the principle of free and open public space for debate. We hope gradually by using counter speech, it will be slowly possible to move toward an online world where differences could exist but not divisions.

Future directions:

Increased polarization seems to be spreading hate speech more. Most of the current models have been developed for English language. There is a need for larger and better hate speech datasets for other languages as well. Using transfer learning for improving the task is another direction. Zero or few shot learning would allow models to be able to build models for low resource languages. Finally, an orthogonal but very interesting direction is to understand the complexity of the annotation task itself. Due to the subjective nature of the task, the perception of hate speech is different for people belonging to different demographics. Another important direction is to integrate the detection and counter systems to build an end-to-end framework for effective hate speech detection and countering mechamism, as shown in figure 

1.

Figure 1: An overview of the hate speech framework.

4.2 Bias and fairness

As outlined in the introduction, in this section we shall talk about two types of important biases (out of many more that dwells in the online world) – bias in news reporting and bias in online recommendation systems.

4.2.1 Media bias

As we hit 53.6% Internet penetration worldwide, compounded by an exponential growth in the number of social media users in the developing countries fuelled by cheap data-rates and smartphone-based-accessibility242424https://economictimes.indiatimes.com/tech/internet/internet-users-in-india-to-reach-627-million-in-2019-report/articleshow/68288868.cms?from=mdr, the news media continues to play a significant role in shaping political discourse and influencing national priorities. Of late, while on one hand, it has become easier to produce news without adequate references, on the other hand, most newsreaders share news without any verification252525https://www.washingtonpost.com/news/the-intersect/wp/2016/06/16/six-in-10-of-you-will-share-this-link-without-reading-it-according-to-a-new-and-depressing-study/ [43]. In many instances, even the mainstream media houses have been accused of copying and distributing news from other media houses with little or no verification 262626http://archiwum.thenews.pl/1/10/Artykul/280476. While being informed about news from sources other than direct correspondents is a common practice, distribution of that news without verification indeed is a worrying trend. In many cases, this has led to fake news propagation by the most reputed media moguls. So, quantifying media bias and defining the abstract idea of bias in this context is an important area of research.

Genesis: In Manufacturing Consent [60], 1988, Edward Herman & Noam Chomsky saw news media as the propagandist which will find ways to propagate the “filtered” message of the rich and powerful to the ordinary masses. Sooner or later in any system, they hypothesize that the news medium will get concentrated in the hands of a few people of power and money and will get manipulated either by ownership or by filtering out news not beneficial for the people in power. Following this model, researchers have hypothesized different kinds of biases like there have been numerous studies examining bias in media especially in the US and the European context. While the term “bias” still remains abstract, some studies have put efforts to make a distinction between the computational sense of bias and the journalistic sense of the same making it more scientifically definable and quantifiable. Journalistic and linguistic studies mostly discuss selection/coverage bias, confirmation/statement bias [78, 86, 105, 118] and psychological/cognitive biases [15, 112]. Recently, a lot of work is being done where the researchers are interested to formulate a computational basis for investigating bias. Some works are focused on specific kinds of bias, such as gender [11, 89, 152], and race [20]. Politics, in particular, is a widely studied and discussed topic. Researchers seek to find ideological political bias of users in social networks [22, 66, 141], news media [5, 12, 77, 79, 113] and user comments [114, 148]. D’Alessio and Allen [25] list three kinds of media bias to be the most widely studied: Coverage/visibility bias [37], gatekeeping bias/selectivity [61] or selection bias [51] (sometimes referred to as agenda bias [37]) and statement bias/tonality bias/presentation bias [37, 51].

Document level bias in reporting: A news article may choose to cover some aspects of one news and filter other aspects to bias the sentiments of the readers toward/against a specific political party or interest group. Researchers have annotated such sentiment leanings of news articles at the article level [70, 12] or the sentence level [85] building document sentiment prediction models based on the annotated data. Following previous researches on sentiment prediction of documents [70, 19, 16], dominated by BERT [35] based methods, Longformer [8] is shown to be the best choice in the prediction of document level bias.

Media bias is topic & demographics dependent: While machine learning models can achieve high accuracy in the task of bias detection, the models are not immune to dataset variations. Inferring media bias is shown to be highly subjective and dependent on the political beliefs of the annotators [45]. Also, the demographic shift plays a key role. The authors in [23] showed that simple language variations in the data due to change in point of view (a cross-country dataset) can have a devastating effect on the media bias classifier. In [45], the authors showed that the common belief [70] that the bias of the source media house is the bias of any news article originated from it, does not hold well. Also, media houses tend to show different political bias both by magnitude and polarity for different topics (e.g., ABC News and CNN show heavy liberal bias on Gun issues but exhibit conservative bias on immigration issues) [23].

Source level bias: Predicting the topical/political bias of individual news outlets is as critical to media profiling as determining factuality. With the advent of user-generated content and exponential rise of digital media, scaling the process of determining media-bias and factuality of reporting of the media houses has become more and more important as everybody who shares an article or screenshot of the same of any source is a news provider now. While measuring the factuality or bias of each news media is a hard task and requires world-knowledge, the prediction of aggregate factuality or bias of a news media house is relatively straight-forward. Political bias and factuality of reporting have a linguistic aspect (what was written) along with a social context (who read it). So, the authors in [6] crawled relevant data from Twitter, Facebook, YouTube & Wikipedia and studied the impact of different metadata extracted from these sources while classifying the media sources. The evaluation results showed that what was written matters most, and that putting all information sources together yields huge improvements over the current state-of-the-art. On the other hand, in  [113], the authors studied the demographics of the US population interested in a media source with a specific political inclination, using the Facebook AdSense tool, to understand the leanings of the audience of the news media houses and reports high accuracy by doing just that. Along with political bias, they were also able to identify the demographic biases in the consumer population of any media house in a zero-shot setting (i.e., using no training data). A demo of their application can be found here272727https://twitter-app.mpi-sws.org/media-bias-monitor/. TIMME [144] supervises a special form of GCNs to identify the political bias of each Twitter user on annotated data and is able to identify the geographic distribution of Twitter users with particular bias which correlates well with the voting pattern of American citizens. They were able to use the same algorithm to identify the bias of each news media house by gathering their Twitter data.

Future directions: News media is widely cited as the fourth pillar of democracy. While the health of a democratic institution depends heavily on fair coverage of the institution by the media houses, studies on how computational media bias is related to the health of democracies are lacking. Again, most of the studies in media bias are concentrated on two-party systems and is done on American and European demographics for the English language media while other democracies also face the same problem deserving similar attention. Further research is needed to understand the challenges faced in a multi-party system for other languages in different demographics. Also, event space in media changes very rapidly. So, research in an online learning setup is needed to further enhance the media bias prediction accuracy over time.

4.2.2 Fairness in recommendation

The digital platform is full of choices. To help users make intelligent choices, different information filtering systems are deployed in online platforms. Recommendation systems (RSs) are one such omnipresent module. From consumers’ perspective, RSs help in finding useful contents; while from platforms’ and producers’ perspectives RSs bring revenue and profit [50, 122]. Given the multi-stakeholder setup and importance of RSs in the livelihood of many of the stakeholders, recommendation fairness is of utmost importance.

Filter bubble and evolution of fairness in RSs: Traditionally, RSs like other information filtering systems are keyed to relevance [87, 130]. However, over dependency on relevance has led to differential services to different users or different user groups. To describe these effects succinctly, Eli Pariser coined a term ‘filter bubble’ [107]. The filter bubble problem is a concern that personalization technologies, including RSs, narrow and bias the topics of information provided to people while they do not notice these facts. To account for these effects, the field of fairness in recommendation first evolved as ‘Information Neutrality in RSs’. Focusing on customer fairness (information neutrality toward customers) Kamishima et. al. [67, 68] tried to solve the unfairness issue in RSs by adding a regularization term that enforces demographic parity282828 can be an instantiation of such demographic parity based regularizers [146].. Such objectives penalized the differences among the average predicted ratings of user groups (based on sensitive attributes, e.g., gender). However, demographic parity is only appropriate when preferences are unrelated to the sensitive features. In tasks such as recommendation, user preferences are indeed influenced by sensitive features such as gender, race, and age [17, 30]. Taking a leaf out of the progresses in fairness literature in supervised machine learning [57, 99], Yao et. al. [146]

put forward fairness notions to bridge the gap. They formulated different customer fairness metric by taking a leaf out of the evolution of fairness in supervised learning 

[57] and showed their effectiveness in improving customer fairness in recommendation [146]. Following their footsteps a number of works focused on ‘group fairness’ in personalized recommendations [153, 38] where first they quantified biases due to recommendation algorithms toward socially salient groups and proposed methodologies to mitigate such biases. However, a major drawback of many of these works was their negligence toward one of the major stakeholder in RSs, i.e., the producer of items/services. This led to a second school of thought when Burke et. al. [13, 14] first advocated for fairness toward both customers and providers in a recommendation framework. Considering RSs as a two-sided affair many nuanced algorithms came into existence considering fairness toward both customers and producers and thus taking a giant step toward a fair marketplace [100, 108, 48].

Figure 2: The figure shows the genesis of notions of fairness in recommendation frameworks. In early 2010, it started with the notion of ‘neutrality in recommendation and by the end of the decade it has evolved into multi-sided fairness toward building a fair marketplace.

Auditing RSs: While the fairness community seems to have covered different forms of biases, there is a lack of understanding of the existing online recommendation systems and biases thereof. Understanding of these systems are especially important today due to the emergence of different private label products (and in-house products) in e-commerce (and OTT) platforms [140, 39, 3]. A private label product is often produced and sold under the retailer’s brand name, providing enough monetary incentive to the platforms to be discriminative against several other products (or producers) on the platform. Note that no third party (3P) regulator can quantify such biases because of the lack of access to the exact underlying algorithms and the exact user-item interaction details. To enable such 3P audits, in one of our works, we presented a novel network-based technique that enabled us to extract important parameters for auditing RSs by considering them as black-boxes [28]. With detailed analysis on three different existing online RSs, we first proposed ways to quantify their induced diversity and extent of information segregation [28]. The usefulness of such a framework is manifold: (a) it sheds light on how recommendations are formed between items based on different item-centric properties, (b) it can be used as a tool for quantifying and auditing for different consumer-focused metrics e.g., relevance, diversity, information-segregation etc., (c) it can be used to quantify different biases toward the associated producers of different items/services, and (d) finally, it also can be used as a tool for mitigating the existing biases by re-wiring few of the recommendation edges. In another follow-up work, we analyzed the sponsored recommendation on e-commerce marketplaces and showed how they are utilized as a trojan-horse to improve sales of products having special relationship with the marketplace (e.g., private label products) [27]. Given that sponsored recommendations potentially have delayed impact on their organic counterparts (by affecting the sales and visibility of products), such studies open up a completely new avenue of research.

Future directions: While the fairness community seems to have covered different forms of biases in recommendation frameworks, it has overlooked the special relationships that may exist between the digital marketplace and a subset of stakeholders, and the biases thereof. Hence, studies of unfairness discovery and mitigation considering the special relationships of platforms remain an under-explored avenue of research till date. The introduction of sponsored search and recommendations complicates the scenario even further. Policies that allow sponsored results to deviate from organic results; while adhering to fairness of marketplace can be another interesting broad direction for further research.

5 Conclusion

In this survey we have presented a critical rundown on the evolution of the online infosphere by depicting some of the research areas that are becoming very crucial at current times. We started our discussion with a view of the infosphere as a collaborative platform, with a dedicated focus on Wikipedia. Wikipedia, the freely available and one of the largest knowledge base, containing a wide variety of information has been a primary focus of an extensive research so far. In this survey we have presented a detailed account of the works on article quality monitoring, editor behaviour and their retention and malicious activities like vandalism.

In the next section we have detailed the growth of the citations and collaborations within and across various scientific disciplines that have their roots in the infosphere. In fact, this has resulted in the birth of many new interdisciplinary landscapes. We also discussed how machine learning algorithms can be used to predict future citations as well as for recommending citations. Finally we touched upon various issues related to anomalous citation flows and their behaviour.

Finally, we summarised the research drives for patrolling the infosphere to suppress the rising volume of harmful content. The discussion started with analyzing the concept of hate speech and its growth over the past few years through the online social media platforms and the adverse impact, thereby, on both the online and the real world. We have shown the massive efforts the research community have put forward in detection and mitigation of such hateful behaviour. However, a lot of issues still remain as open problems and need immediate attention. We observed dense connectivity in the hateful users network by crawling Gab platform and analyzing their data and found that a significant amount of posts are generated by these hateful users in social media platforms. Additionally, we studied the temporal effect of hate speech on the users by using the Gab data and found the increasing rate of hateful users in the social network. After skimming through the recent literature we have pointed out that by incorporating knowledge based context information for a given post improves the overall performance of hatespeech detection rather than by analyzing only the textual information. Apart from that we have observed that by adding the user information of a given post can be further analyzed to improve the hatespeech detection task. The last segment of the discussion dealt with the bias and discrimination that are becoming pervasive across different online environments like recommendation and news media platforms.

To conclude, the above mentioned research aspects related to the online infosphere have attracted a lot of attention across the board including scientific communities, industry stakeholders and policy-makers. This paper discusses the technical challenges and possible solutions in a direction that utilizes the immense power of AI for solving real world problems but also considers the societal implications of these solutions. As a final note, we could see that the problems related to the anomalies in scientific collaborations and citations, hate speech detection and mitigation in social media and the bias and unfairness in news media and recommendation systems have a lot of open ends thus enabling exciting opportunities for future research.

References

  • [1] Albert, R., and Barabási, A.-L. Statistical mechanics of complex networks. Rev. Mod. Phys. 74 (Jan 2002), 47–97.
  • [2] Aluru, S. S., Mathew, B., Saha, P., and Mukherjee, A. Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465 (2020).
  • [3] Amazon. Online platforms and market power, part 2: Innovation and entrepreneurship. https://docs.house.gov/meetings/JU/JU05/20190716/109793/HHRG-116-JU05-20190716-SD038.pdf, 2019.
  • [4] Badjatiya, P., Gupta, S., Gupta, M., and Varma, V. Deep learning for hate speech detection in tweets. WWW, pp. 759–760.
  • [5] Baly, R., Karadzhov, G., Alexandrov, D., Glass, J., and Nakov, P. Predicting factuality of reporting and bias of news media sources. arXiv preprint arXiv:1810.01765 (2018).
  • [6] Baly, R., Karadzhov, G., An, J., Kwak, H., Dinkov, Y., Ali, A., Glass, J., and Nakov, P. What was written vs. who read it: News media profiling using text analysis and social media context. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Online, July 2020), Association for Computational Linguistics, pp. 3364–3374.
  • [7] Barthel, R., and Seidl, R. Interdisciplinary collaboration between natural and social sciences – status and trends exemplified in groundwater research. PLOS ONE 12, 1 (01 2017), 1–27.
  • [8] Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer, 2020.
  • [9] Bergmann, T., Dale, R., Sattari, N., Heit, E., and Bhat, H. S. The interdisciplinarity of collaborations in cognitive science. Cognitive science 41, 5 (2017), 1412–1418.
  • [10] Bhagavatula, C., Feldman, S., Power, R., and Ammar, W. Content-based citation recommendation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (New Orleans, Louisiana, June 2018), Association for Computational Linguistics, pp. 238–251.
  • [11] Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., and Kalai, A. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems (USA, 2016), NIPS’16, Curran Associates Inc., pp. 4356–4364.
  • [12] Budak, C., Goel, S., and Rao, J. M. Fair and balanced? quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80, S1 (2016), 250–271.
  • [13] Burke, R. Multisided fairness for recommendation. arXiv preprint arXiv:1707.00093 (2017).
  • [14] Burke, R., Sonboli, N., and Ordonez-Gauger, A. Balanced neighborhoods for multi-sided fairness in recommendation. In ACM FAccT (2018).
  • [15] Caliskan, A., Bryson, J. J., and Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186.
  • [16] Chakraborty, S., Goyal, P., and Mukherjee, A.

    Aspect-based sentiment analysis of scientific reviews.

    In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (New York, NY, USA, 2020), JCDL ’20, Association for Computing Machinery, p. 207–216.
  • [17] Chausson, O. Who watches what?: assessing the impact of gender and personality on film preferences. Paper published online on the MyPersonality project website http://mypersonality. org/wiki/doku. php (2010).
  • [18] Chen, Y., Zhou, Y., Zhu, S., and Xu, H. Detecting offensive language in social media to protect adolescent online safety. pp. 71–80.
  • [19] Choi, G., Oh, S., and Kim, H. Improving document-level sentiment classification using importance of sentences. Entropy 22, 12 (2020), 1336.
  • [20] Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153–163.
  • [21] Chung, Y.-L., Kuzmenko, E., Tekiroglu, S. S., and Guerini, M. CONAN - COunter NArratives through nichesourcing: a multilingual dataset of responses to fight online hate speech. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Florence, Italy, July 2019), Association for Computational Linguistics, pp. 2819–2829.
  • [22] Conover, M. D., Gonçalves, B., Ratkiewicz, J., Flammini, A., and Menczer, F. Predicting the political alignment of twitter users. In 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing (2011), IEEE, pp. 192–199.
  • [23] Cremisini, A., Aguilar, D., and Finlayson, M. A Challenging Dataset for Bias Detection: The Case of the Crisis in the Ukraine. 06 2019, pp. 173–183.
  • [24] Dadvar, M., Trieschnigg, D., Ordelman, R., and de Jong, F. Improving cyberbullying detection with user context. pp. pp 693–696.
  • [25] D’Alessio, D., and Allen, M. Media bias in presidential elections: A meta-analysis. Journal of communication 50, 4 (2000), 133–156.
  • [26] Dang, Q. V., and Ignat, C.-L. Quality assessment of wikipedia articles without feature engineering. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries (2016), pp. 27–30.
  • [27] Dash, A., Chakraborty, A., Ghosh, S., Mukherjee, A., and Gummadi, K. P. When the umpire is also a player: Bias in private label product recommendations on e-commerce marketplaces. In ACM FAccT (2021).
  • [28] Dash, A., Mukherjee, A., and Ghosh, S. A network-centric framework for auditing recommendation systems. In IEEE INFOCOM (2019).
  • [29] Davidson, T., Warmsley, D., Macy, M., and Weber, I. Automated hate speech detection and the problem of offensive language.
  • [30] Daymont, T. N., and Andrisani, P. J. Job preferences, college major, and the gender gap in earnings. Journal of Human Resources (1984).
  • [31] de La Robertie, B., Pitarch, Y., and Teste, O. Measuring article quality in wikipedia using the collaboration network. In 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) (2015), IEEE, pp. 464–471.
  • [32] DeGroot, M. H. Reaching a consensus. Journal of the American Statistical Association 69, 345 (1974), 118–121.
  • [33] Del Vigna, F., Cimino, A., Dell’Orletta, F., Petrocchi, M., and Tesconi, M. Hate me, hate me not: Hate speech detection on facebook.
  • [34] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (Minneapolis, Minnesota, June 2019), Association for Computational Linguistics, pp. 4171–4186.
  • [35] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (Minneapolis, Minnesota, June 2019), Association for Computational Linguistics, pp. 4171–4186.
  • [36] Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., and Bhamidipati, N. Hate speech detection with comment embeddings. pp. 29–30.
  • [37] Eberl, J.-M., Boomgaarden, H. G., and Wagner, M. One bias fits all? three types of media bias and their effects on party preferences. Communication Research 44, 8 (2017), 1125–1148.
  • [38] Edizel, B., Bonchi, F., Hajian, S., Panisson, A., and Tassa, T. Fairecsys: mitigating algorithmic bias in recommender systems.

    Springer International Journal of Data Science and Analytics

    (2019).
  • [39] Faherty, E., Huang, K., and Land, R. The amazon monopoly: Is amazon’s private label business the tipping point? Munich Personal RePEc Archive (2017).
  • [40] Fister, I., Fister, I., and Perc, M. Toward the discovery of citation cartels in citation networks. Frontiers in Physics 4 (2016), 49.
  • [41] Fister, I., Mlakar, U., and Brest, J. A new population-based nature-inspired algorithm every month : Is the current era coming to the end ? In StuCoSReC: Proceedings of the 2016 3rd Student Computer Science Research Conference (2016).
  • [42] Färber, M., and Jatowt, A. Citation recommendation: approaches and datasets. International Journal on Digital Libraries 21, 4 (Aug 2020), 375–405.
  • [43] Gabielkov, M., Ramachandran, A., Chaintreau, A., and Legout, A. Social Clicks: What and Who Gets Read on Twitter? In IFIP Perf. (Antibes Juan-les-Pins, France, June 2016).
  • [44] Gagliardone, I., Gal, D., Alves, T., and Martinez, G. Countering online hate speech. UNESCO Publishing, 2015.
  • [45] Ganguly, S., Kulshrestha, J., An, J., and Kwak, H. Empirical evaluation of three common assumptions in building political media bias datasets. Proceedings of the International AAAI Conference on Web and Social Media 14, 1 (May 2020), 939–943.
  • [46] Gao, L., and Huang, R. Detecting online hate speech using context aware models. pp. 260–266.
  • [47] Garland, J., Ghazi-Zahedi, K., Young, J.-G., Hébert-Dufresne, L., and Galesic, M. Countering hate on social media: Large scale classification of hate and counter speech. arXiv preprint arXiv:2006.01974 (2020).
  • [48] Geyik, S. C., Ambler, S., and Kenthapadi, K. Fairness-aware ranking in search & recommendation systems with application to linkedin talent search.
  • [49] Gomez, R., Gibert, J., Gomez, L., and Karatzas, D. Exploring hate speech detection in multimodal publications. pp. 1459–1467.
  • [50] Gomez-Uribe, C. A., and Hunt, N. The netflix recommender system: Algorithms, business value, and innovation. ACM TMIS 6, 4 (2016).
  • [51] Groeling, T. Media bias by the numbers: Challenges and opportunities in the empirical study of partisan news. Annual Review of Political Science 16, 1 (2013), 129–151.
  • [52] Guda, B. P. R., Seelaboyina, S. B., Sarkar, S., and Mukherjee, A. Nwqm: A neural quality assessment framework for wikipedia. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020), pp. 8396–8406.
  • [53] Halfaker, A., and Geiger, R. S. Ores: Lowering barriers with participatory machine learning in wikipedia. arXiv preprint arXiv:1909.05189 (2019).
  • [54] Halfaker, A., Geiger, R. S., Morgan, J. T., and Riedl, J. The rise and decline of an open collaboration system: How wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist 57, 5 (2013), 664–688.
  • [55] Halfaker, A., Kittur, A., and Riedl, J. Don’t bite the newbies: how reverts affect the quantity and quality of wikipedia work. In Proceedings of the 7th international symposium on wikis and open collaboration (2011), pp. 163–172.
  • [56] Halfaker, A., and Taraborelli, D. Artificial intelligence service “ores” gives wikipedians x-ray specs to see through bad edits. Wikimedia Blog (2015).
  • [57] Hardt, M., Price, E., Srebro, N., et al. Equality of opportunity in supervised learning. In NIPS (2016).
  • [58] Hazra, R., Singh, M., Goyal, P., Adhikari, B., and Mukherjee, A. The rise and rise of interdisciplinary research: Understanding the interaction dynamics of three major fields - physics, mathematics and computer science. In Digital Libraries at the Crossroads of Digital Information for the Future - 21st International Conference on Asia-Pacific Digital Libraries, ICADL 2019, Kuala Lumpur, Malaysia, November 4-7, 2019, Proceedings (2019), pp. 71–77.
  • [59] He, Q., Pei, J., Kifer, D., Mitra, P., and Giles, L. Context-aware citation recommendation. In Proceedings of the 19th International Conference on World Wide Web (New York, NY, USA, 2010), WWW ’10, Association for Computing Machinery, p. 421–430.
  • [60] Herman, E. S., and Chomsky, N. Manufacturing consent: The political economy of the mass media. Pantheon Books, 1988.
  • [61] Hofstetter, C. R., and Buss, T. F. Bias in television news coverage of political events: A methodological analysis. Journal of Broadcasting 22, 4 (1978), 517–530.
  • [62] Jain, N., and Singh, M. The evolving ecosystem of predatory journals: a case study in indian perspective. In International Conference on Asian Digital Libraries (2019), Springer, pp. 78–92.
  • [63] Jana, A., Kanojiya, P., Goyal, P., and Mukherjee, A. Wikiref: Wikilinks as a route to recommending appropriate references for scientific wikipedia pages. arXiv preprint arXiv:1806.04092 (2018).
  • [64] Jeong, C., Jang, S., Shin, H., Park, E., and Choi, S. A context-aware citation recommendation model with bert and graph convolutional networks, 2019.
  • [65] Johnson, I., Lemmerich, F., Sáez-Trumper, D., West, R., Strohmaier, M., and Zia, L. Global gender differences in wikipedia readership. arXiv preprint arXiv:2007.10403 (2020).
  • [66] Johnson, K., and Goldwasser, D. Identifying stance by analyzing political discourse on twitter. In Proceedings of the First Workshop on NLP and Computational Social Science (2016), pp. 66–75.
  • [67] Kamishima, T., Akaho, S., Asoh, H., and Sakuma, J. Enhancement of the neutrality in recommendation. In Decisions@ RecSys (2012).
  • [68] Kamishima, T., Akaho, S., and Sakuma, J. Fairness-aware learning through regularization approach. In IEEE ICDMW (2011).
  • [69] Katzenbach, C., Binns, R., and Gorwa, R. Algorithmic content moderation: Technical and political challenges in the automation of platform governance. Big Data and Society 7, 1 (2020).
  • [70] Kiesel, J., Mestre, M., Shukla, R., Vincent, E., Adineh, P., Corney, D., Stein, B., and Potthast, M. SemEval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation (Minneapolis, Minnesota, USA, June 2019), Association for Computational Linguistics, pp. 829–839.
  • [71] Kiesel, J., Potthast, M., Hagen, M., and Stein, B. Spatio-temporal analysis of reverted wikipedia edits. In ICWSM (2017).
  • [72] Kojaku, S., Livan, G., and Masuda, N. Detecting citation cartels in journal networks, 2020.
  • [73] Kshirsagar, R., Cukuvac, T., McKeown, K., and McGregor, S. Predictive embeddings for hate speech detection on twitter. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2) (2018), pp. 26–32.
  • [74] Kumar, R., Raghavan, P., Rajagopalan, S., Sivakumar, D., Tomkins, A., and Upfal, E. Stochastic models for the web graph. In Proceedings of the 41st Annual Symposium on Foundations of Computer Science (USA, 2000), FOCS ’00, IEEE Computer Society, p. 57.
  • [75] Kumar, S., Spezzano, F., and Subrahmanian, V. Vews: A wikipedia vandal early warning system. ArXiv abs/1507.01272 (2015).
  • [76] Kwon, S., Solomon, G. E. A., Youtie, J., and Porter, A. L. A measure of knowledge flow between specific fields: Implications of interdisciplinarity for impact and funding. PLOS ONE 12, 10 (10 2017), 1–16.
  • [77] Laver, M., Benoit, K., and Garry, J. Extracting policy positions from political texts using words as data. American political science review 97, 2 (2003), 311–331.
  • [78] Lazaridou, K., Krestel, R., and Naumann, F. Identifying media bias by analyzing reported speech. In 2017 IEEE International Conference on Data Mining (ICDM) (2017), IEEE, pp. 943–948.
  • [79] Le, H. T. T., Shafiq, Z., and Srinivasan, P. Scalable news slant measurement using twitter. In Eleventh International AAAI Conference on Web and Social Media (2017).
  • [80] Le, Q., and Mikolov, T. Distributed representations of sentences and documents. In International conference on machine learning (2014), pp. 1188–1196.
  • [81] Le, Q., and Mikolov, T. Distributed representations of sentences and documents. 31st International Conference on Machine Learning, ICML 2014 4 (05 2014).
  • [82] Li, C.-T., Lin, Y.-J., Yan, R., and Yeh, M.-Y. Trend-based citation count prediction for research articles. In Advances in Knowledge Discovery and Data Mining (Cham, 2015), T. Cao, E.-P. Lim, Z.-H. Zhou, T.-B. Ho, D. Cheung, and H. Motoda, Eds., Springer International Publishing, pp. 659–671.
  • [83] Li, S., Zhao, W. X., Yin, E. J., and Wen, J.-R. A neural citation count prediction model based on peer review text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Hong Kong, China, Nov. 2019), Association for Computational Linguistics, pp. 4914–4924.
  • [84] Li, X., Tang, J., Wang, T., Luo, Z., and De Rijke, M. Automatically assessing wikipedia article quality by exploiting article–editor networks. In European Conference on Information Retrieval (2015), Springer, pp. 574–580.
  • [85] Lim, S., Jatowt, A., and Masatoshi, Y. Creating a dataset for fine-grained bias detection in news articles. Forum on Data Engineering and Information Management 12 (May 2020), 1–35.
  • [86] Lin, Y.-R., Bagrow, J. P., and Lazer, D. More voices than ever? quantifying media bias in networks. In Fifth International AAAI Conference on Weblogs and Social Media (2011).
  • [87] Linden, G., Smith, B., and York, J. Amazon. com recommendations: Item-to-item collaborative filtering. IEEE Internet computing (2003).
  • [88] Littlejohn, A., and Hood, N. Becoming an online editor: Perceived roles and responsibilities of wikipedia editors. Information Research 23, 1 (2018).
  • [89] Madaan, N., Mehta, S., Agrawaal, T., Malhotra, V., Aggarwal, A., Gupta, Y., and Saxena, M. Analyze, detect and remove gender stereotyping from bollywood movies. In Conference on Fairness, Accountability and Transparency (2018), pp. 92–105.
  • [90] Maity, S. K., Chakraborty, A., Goyal, P., and Mukherjee, A. Detection of sockpuppets in social media. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (2017), pp. 243–246.
  • [91] Marrese-Taylor, E., Loyola, P., and Matsuo, Y. An edit-centric approach for wikipedia article quality assessment. arXiv preprint arXiv:1909.08880 (2019).
  • [92] Martinez-Rico, J. R., Martínez-Romo, J., and Araujo, L. Can deep learning techniques improve classification performance of vandalism detection in wikipedia? Eng. Appl. Artif. Intell. 78 (2019), 248–259.
  • [93] Mathew, B., Dutt, R., Goyal, P., and Mukherjee, A. Spread of hate speech in online social media. pp. 173–182.
  • [94] Mathew, B., Illendula, A., Saha, P., Sarkar, S., Goyal, P., and Mukherjee, A. Hate begets hate: A temporal study of hate speech. Proc. ACM Hum.-Comput. Interact. 4, CSCW2 (Oct. 2020).
  • [95] Mathew, B., Kumar, N., Goyal, P., and Mukherjee, A. Interaction dynamics between hate and counter users on twitter. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD. 2020, pp. 116–124.
  • [96] Mathew, B., Saha, P., Tharad, H., Rajgaria, S., Singhania, P., Maity, S. K., Goyal, P., and Mukherjee, A. Thou shalt not hate: Countering online hate speech. In Proceedings of the International AAAI Conference on Web and Social Media (2019), vol. 13, pp. 369–380.
  • [97] Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., and Mukherjee, A. Hatexplain: A benchmark dataset for explainable hate speech detection. arXiv preprint arXiv:2012.10289 (2020).
  • [98] Meeder, B., Karrer, B., Sayedi, A., Ravi, R., Borgs, C., and Chayes, J. We know who you followed last summer: inferring social link creation times in twitter. In Proceedings of the 20th international conference on World wide web (2011), pp. 517–526.
  • [99] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019).
  • [100] Mehrotra, R., McInerney, J., Bouchard, H., Lalmas, M., and Diaz, F. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. In ACM CIKM (2018).
  • [101] Morgan, J., Bouterse, S., Walls, H., and Stierch, S. Tea and sympathy: crafting positive new user experiences on wikipedia. In CSCW ’13 (2013).
  • [102] Morgan, J., and Halfaker, A. Evaluating the impact of the wikipedia teahouse on newcomer socialization and retention. Proceedings of the 14th International Symposium on Open Collaboration (2018).
  • [103] Morillo, F., Bordons, M., and Gómez, I. Interdisciplinarity in science: A tentative typology of disciplines and research areas. J. Am. Soc. Inf. Sci. Technol. 54, 13 (Nov. 2003), 1237–1249.
  • [104] Murić, G., Abeliuk, A., Lerman, K., and Ferrara, E. Collaboration drives individual productivity. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–24.
  • [105] Nickerson, R. S. Confirmation bias: A ubiquitous phenomenon in many guises. Review of general psychology 2, 2 (1998), 175–220.
  • [106] Pandey, P. K., Singh, M., Goyal, P., Mukherjee, A., and Chakrabarti, S. Analysis of reference and citation copying in evolving bibliographic networks. Journal of Informetrics 14, 1 (2020), 101003.
  • [107] Pariser, E. The filter bubble: What the Internet is hiding from you. Penguin UK, 2011.
  • [108] Patro, G. K., Biswas, A., Ganguly, N., Gummadi, K. P., and Chakraborty, A. Fairrec: Two-sided fairness for personalized recommendations in two-sided platforms. In WWW (2020).
  • [109] Pedersen, D. B. Integrating social sciences and humanities in interdisciplinary research. Palgrave Communications 2 (2016), 16036.
  • [110] Pornprasit, C., Liu, X., Kertkeidkachorn, N., Kim, K.-S., Noraset, T., and Tuarob, S. Convcn: A cnn-based citation network embedding algorithm towards citation recommendation. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (New York, NY, USA, 2020), JCDL ’20, Association for Computing Machinery, p. 433–436.
  • [111] Qian, J., Bethke, A., Liu, Y., Belding, E., and Wang, W. Y. A benchmark dataset for learning to intervene in online hate speech. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Hong Kong, China, Nov. 2019), Association for Computational Linguistics, pp. 4755–4764.
  • [112] Recasens, M., Danescu-Niculescu-Mizil, C., and Jurafsky, D. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2013), pp. 1650–1659.
  • [113] Ribeiro, F. N., Henrique, L., Benevenuto, F., Chakraborty, A., Kulshrestha, J., Babaei, M., and Gummadi, K. P. Media bias monitor: Quantifying biases of social media news outlets at large-scale. In Twelfth International AAAI Conference on Web and Social Media (2018).
  • [114] Ribeiro, M. H., Calais, P. H., Almeida, V. A., and Meira Jr, W. ” everything i disagree with is# fakenews”: Correlating political polarization and spread of misinformation. arXiv preprint arXiv:1706.05924 (2017).
  • [115] Ribeiro, M. H., Gligorić, K., Peyrard, M., Lemmerich, F., Strohmaier, M., and West, R. Sudden attention shifts on wikipedia following covid-19 mobility restrictions. arXiv preprint arXiv:2005.08505 (2020).
  • [116] Richards, R. D., and Calvert, C. Counterspeech 2000: A new look at the old remedy for bad speech. BYU L. Rev. (2000), 553.
  • [117] Ruprechter, T., Santos, T., and Helic, D. Relating wikipedia article quality to edit behavior and link structure. Applied Network Science 5 (2020), 1–20.
  • [118] Saez-Trumper, D., Castillo, C., and Lalmas, M. Social media news communities: Gatekeeping, coverage, and statement bias. In Proceedings of the 22Nd ACM International Conference on Information & Knowledge Management (New York, NY, USA, 2013), CIKM ’13, ACM, pp. 1679–1684.
  • [119] Saha, P., Mathew, B., Goyal, P., and Mukherjee, A. Hateminers: Detecting hate speech against women. arXiv preprint arXiv:1812.06700 (2018).
  • [120] Sarkar, S., Reddy, B. P., Sikdar, S., and Mukherjee, A. Stre: Self attentive edit quality prediction in wikipedia. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), pp. 3962–3972.
  • [121] Sayama, H., and Akaishi, J. Characterizing interdisciplinarity of researchers and research topics using web search engines. PLOS ONE 7, 6 (06 2012), 1–9.
  • [122] Sharma, A., Hofman, J. M., and Watts, D. J. Estimating the causal impact of recommendation systems from observational data. In ACM EC (2015).
  • [123] Shen, A., Qi, J., and Baldwin, T. A hybrid model for quality assessment of wikipedia articles. In Proceedings of the Australasian Language Technology Association Workshop 2017 (2017), pp. 43–52.
  • [124] Shen, A., Salehi, B., Baldwin, T., and Qi, J. A joint model for multimodal document quality assessment. In 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL) (2019), IEEE, pp. 107–110.
  • [125] Shen, A., Salehi, B., Qi, J., and Baldwin, T. Feature-guided neural model training for supervised document representation learning. In Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association (2019), pp. 47–51.
  • [126] Shiji, C., Yves, G., Clément, A., and Vincent, L. Interdisciplinarity patterns of highly‐cited papers: A cross‐disciplinary analysis. Proceedings of the American Society for Information Science and Technology 51, 1 (2015), 1–4.
  • [127] Singh, M., Jaiswal, A., Shree, P., Pal, A., Mukherjee, A., and Goyal, P. Understanding the impact of early citers on long-term scientific impact. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL) (2017), pp. 1–10.
  • [128] Singh, M., Patidar, V., Kumar, S., Chakraborty, T., Mukherjee, A., and Goyal, P. The role of citation context in predicting long-term citation profiles: An experimental study based on a massive bibliographic text dataset. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (New York, NY, USA, 2015), CIKM ’15, Association for Computing Machinery, p. 1271–1280.
  • [129] Singh, M., Sarkar, R., Goyal, P., Mukherjee, A., and Chakrabarti, S. Relay-linking models for prominence and obsolescence in evolving networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA, 2017), KDD ’17, Association for Computing Machinery, p. 1077–1086.
  • [130] Smith, B., and Linden, G. Two decades of recommender systems at amazon. com. IEEE internet computing 21, 3 (2017).
  • [131] Spezzano, F., Suyehira, K., and Gundala, L. A. Detecting pages to protect in wikipedia across multiple languages. Social Network Analysis and Mining 9 (2019), 1–16.
  • [132] Steinmacher, I., Conte, T., Gerosa, M. A., and Redmiles, D.

    Social barriers faced by newcomers placing their first contribution in open source software projects.

    In Proceedings of the 18th ACM conference on Computer supported cooperative work & social computing (2015), pp. 1379–1392.
  • [133] Susuri, A., Hamiti, M., and Dika, A. Machine learning based detection of vandalism in wikipedia across languages. 2016 5th Mediterranean Conference on Embedded Computing (MECO) (2016), 446–451.
  • [134] Tekiroglu, S. S., Chung, Y.-L., and Guerini, M. Generating counter narratives against online hate speech: Data and strategies. arXiv preprint arXiv:2004.04216 (2020).
  • [135] Tran, K., and Christen, P. Cross-language learning from bots and users to detect vandalism on wikipedia. IEEE Transactions on Knowledge and Data Engineering 27 (2015), 673–685.
  • [136] Van Hee, C., Lefever, E., Verhoeven, B., Mennes, J., Desmet, B., Pauw, G., Daelemans, W., and Hoste, V. Detection and fine-grained classification of cyberbullying events.
  • [137] Verstak, A., Acharya, A., Suzuki, H., Henderson, S., Iakhiaev, M., Lin, C. C. Y., and Shetty, N. On the shoulders of giants: The growing impact of older articles, 2014.
  • [138] Warncke-Wang, M., Ayukaev, V. R., Hecht, B., and Terveen, L. G. The success and failure of quality improvement projects in peer production communities. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (2015), pp. 743–756.
  • [139] Waseem, Z., and Hovy, D. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. pp. 88–93.
  • [140] Wired. Amazon and netflix look to their own shows as the key to world domination. https://www.wired.com/2016/12/amazon-netflix-look-shows-key-world-domination/, 2016.
  • [141] Wong, F. M. F., Tan, C. W., Sen, S., and Chiang, M. Quantifying political leaning from tweets, retweets, and retweeters. IEEE transactions on knowledge and data engineering 28, 8 (2016), 2158–2172.
  • [142] Wright, L., Ruths, D., Dillon, K. P., Saleem, H. M., and Benesch, S. Vectors for counterspeech on Twitter. In Proceedings of the First Workshop on Abusive Language Online (Vancouver, BC, Canada, Aug. 2017), Association for Computational Linguistics, pp. 57–62.
  • [143] Xiao, S., Yan, J., Li, C., Jin, B., Wang, X., Yang, X., Chu, S. M., and Zhu, H. On modeling and predicting individual paper citation count over time. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (2016), IJCAI’16, AAAI Press, p. 2676–2682.
  • [144] Xiao, Z., Song, W., Xu, H., Ren, Z., and Sun, Y. Timme: Twitter ideology-detection via multi-task multi-relational embedding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA, 2020), KDD ’20, Association for Computing Machinery, p. 2258–2268.
  • [145] Yang, D., Halfaker, A., Kraut, R. E., and Hovy, E. H. Who did what: Editor role identification in wikipedia. In ICWSM (2016), pp. 446–455.
  • [146] Yao, S., and Huang, B. Beyond parity: Fairness objectives for collaborative filtering. In NeurIPS (2017).
  • [147] Yazdanian, R., Zia, L., Morgan, J., Mansurov, B., and West, R. Eliciting new wikipedia users’ interests via automatically mined questionnaires: For a warm welcome, not a cold start. In Proceedings of the International AAAI Conference on Web and Social Media (2019), vol. 13, pp. 537–547.
  • [148] Yigit-Sert, S., Altingovde, I. S., and Ulusoy, O. Towards detecting media bias by utilizing user comments. In Proceedings of the 8th ACM Conference on Web Science (New York, NY, USA, 2016), WebSci ’16, ACM, pp. 374–375.
  • [149] Yuan, S., Tang, J., Zhang, Y., Wang, Y., and Xiao, T. Modeling and predicting citation count via recurrent neural network with long short-term memory. ArXiv abs/1811.02129 (2018).
  • [150] Zhang, H., Ren, Y., and Kraut, R. E. Mining and predicting temporal patterns in the quality evolution of wikipedia articles. In HICSS (2020), pp. 1–10.
  • [151] Zhang, S., Hu, Z., Zhang, C., and Yu, K. History-based article quality assessment on wikipedia. In 2018 IEEE International Conference on Big Data and Smart Computing (BigComp) (2018), IEEE, pp. 1–8.
  • [152] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K.-W. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457 (2017).
  • [153] Zhu, Z., Hu, X., and Caverlee, J.

    Fairness-aware tensor-based recommendation.

    In ACM CIKM (2018).