Ranking systems are at the core of many online services, including search engines, recommender systems, or news feeds in social media. Recent research suggests that the underlying ranking algorithms may impact society, playing an active role in the spread of misinformation (Vosoughi et al., 2018), political polarization (Dandekar et al., 2013), or trustworthiness (Diaz, 2008). They might also reinforce existing judgment biases (Baeza-Yates, 2018).
Rankings systematically affect the information people access about products, services, events, or ideas, because users are more likely to click on top-ranked items (Joachims et al., 2017; Wang et al., 2018; Mens et al., 2018). When items are ranked based on popularity, this leads to a self-reinforcing dynamics according to which popular items become increasingly more popular (Salganik et al., 2006).
In this paper, we identify a surprising effect of popularity-based rankings. Consider a setting with two distinct classes of news sources that differ in their political orientations, e.g., left-leaning or right-leaning. We show that, under a fairly broad set of conditions, the total share of web traffic (proportion of clicks) attracted by a given class of news sources decreases with the number of news sources in that class. We call this phenomenon the ‘few-get-richer’ effect. For example, if there are 20 news sources, the total number of clicks on left-leaning sources will be larger when there are just 3 of these sources than when there are 17 of them.
Intuition suggests that popular items should be more relevant and trustworthy than unpopular ones. Yet, extensive research indicates that popularity is often not very informative about quality, especially in settings characterized by ‘rich-get-richer’ dynamics (sometimes called the ‘Matthew effect’) (Merton, 1968; Salganik et al., 2006), or information cascades (Banerjee, 1992; Bikhchandani et al., 1992). In these settings, the randomness inherent to the dynamics of the system implies that items that become the most popular are not always those with the best quality. The ‘few-get-richer’ effect adds to research on the ‘rich-get-richer’ dynamics by showing that popularity-based rankings do not only create ‘noise’ in the ranking, but can also lead to a systematic ranking bias: when there are two distinct classes of items, items from the smaller class become better ranked than similar items from the larger class.
The few-get-richer effect emerges in settings characterized by two design features. The first feature consists in the ranking of items in terms of popularity (i.e., items with more clicks are higher ranked). The second feature is a partition of the available items in two (or more) distinct classes.
We make two reasonable behavioral assumptions. The first assumption is users’ tendency to click on top-ranked items. The second assumption is that users have heterogeneous preferences for the item classes. Some users have a preference for items of a particular class, while others have a preference for items of other classes. Still other users are indifferent to the item class.
Returning to our news search example, suppose there are few left-leaning and many right-leaning news sources. We assume there are three types of users: left-leaning, right-leaning, and indifferent. The heterogeneous preference assumption means that left-leaning users are more likely to click on left-leaning news sources, right-leaning users are more likely to click on right-leaning news sources, and indifferent users click exclusively based on rank. Even if the left-leaning news sources are unpopular, left-leaning individuals will seek them out. Because there are few such news sources, the clicks of these left-leaning individuals will be concentrated on a few news sources, and these sources will tend to ‘shoot up to the top’. Once a news source has gotten close to the top, it will attract not only the clicks from the left-leaning individuals, but also the clicks of indifferent users, simply because of the rich-get-richer dynamics. This is the few-get-richer effect.
2. Related Work
Our results contribute to the understanding of the limitations of recommender systems (Herlocker et al., 2004; McNee et al., 2006; Mens et al., 2018; Ciampaglia et al., 2018; Kremer et al., 2014; Che and Hörner, 2018), with direct applications to the design of fair, transparent and efficient ranking systems (Castillo, 2019; Biega et al., 2018; Zehlike et al., 2017), as well as methods to reduce the spread of misinformation (Kim et al., 2018; Vosoughi et al., 2018) or uncivil behavior (Muddiman and Stroud, 2017; Cheng et al., 2017).
An extensive literature on modeling (click) user behavior is weakly related to our work (Zhang et al., 2011; Chapelle and Zhang, 2009; Dupret and Piwowarski, 2008; Richardson et al., 2007). Closer to our work, several papers have proposed models of the dynamics of interactions between individual searches and ranking algorithms, e.g., for understanding the feedback loop between ranking system and user queries (Demange, 2014), explaining the observed mitigation of search engines’ popularity bias (Fortunato et al., 2006), or the competition of memes using limited attention (Weng et al., 2012). The paper closest to ours is (Germano and Sobbrio, 2018), which also obtains a few-get-richer effect in a model where individuals get multiple signals and where (news) items are ranked via a probabilistic popularity-based ranking. Besides being simpler, our model works with a discrete and deterministic ranking of the websites rather than a continuous and probabilistic one. Among other things, this allows for a tighter connection with the experiment of Section 5.
3. The Model
We present a stylized model of a search environment where individuals use a search engine to look for information on a binary issue. At the center of the model is the ranking algorithm, which ranks and directs individuals to the different websites, based on the popularity of individuals’ choices.
3.1. Model of the Search Environment
There are items, each of which belongs to exactly one of two classes. For example, the items can be news articles and the class of an article can be whether the source is known to be left-leaning or right-leaning. A different, visual, example can comprise images about animals, some of which are cats while the others are dogs. In general, each of the items is characterized by a binary signal that defines its class; let denote the set and number of items of class , .
There is a single popularity-based ranking algorithm which, starting with a given initial ranking , ranks all the items in according to the number of clicks received. Let denote the ranking seen by individual , where denotes the rank of item observed by individual .
There are individuals, each of which is characterized by a parameter , which can be of one of three types. The three types are denoted by Type , Type and Type , and their proportions in the population are , and , respectively. Type 0 (resp. Type 1) individuals have a preference for clicking on items of class (resp. class ). Type individuals are indifferent between clicking on items of class or of class absent ranking. We represent the set of possible types by , where
denotes the probability that an agent of Type(i.e., with ) clicks on an item of class absent ranking. Our assumptions about the preferences of the individuals in the three types imply , and .
We summarize these preferences in terms of propensities with which individual with clicks on item , defined by:
For simplicity, in this section, we consider the extreme and symmetric case, where , and .
3.2. Model of Stochastic Choices
Individuals enter one after the other and observe the ranked list of items, and based on the ranking and on their preference for the signals given by , they click on one of the items according to a probabilistic choice function obtained as follows. We use the function to weigh the propensities , whereby calibrates an individual’s search cost or attention bias, so that an item ranked exactly one position higher has times as much probability of being clicked. The probability individual clicks on website is given by:
Thus, individual observes and clicks on a website according to . His click gets recorded by the ranking algorithm, which updates its ranking to . This affects the ranking of the websites observed by the next individual, who clicks on a website according to and so on. We denote the total clicking probability on items of class () by .
Overall, this defines a search environment with parameters . We assume and so that individuals are subject to an attention bias and are heterogeneous in the sense that there are nonzero shares of Types 0, 1 and 2. For simplicity, in this section, we assume .
3.3. The Few-Get-Richer Effect
The following result shows how few items of a given class can attract more traffic than many more items of the same class taken together. In particular, it implies that if there is just one item of a given class, then it will attract more traffic by itself than such items taken together in a corresponding environment where there are such items.
Proposition 1 ().
Fix two popularity-based search environments and that differ only in the number of items of class 1 ( and respectively). Suppose , then there exists such that, for any , the total clicking probability () by individual on an item in in environment is strictly greater than the total clicking probability () by individual on an item in in environment , provided is sufficiently small.
The proof is in three steps. First, we characterize a limit ranking () of the process defined by Eq. (2) and show it constitutes a (stable) limit. Second, we show it is the unique such limit ranking. Finally, we compute total traffic on all items in at the limit and show it is over half of total traffic when , and hence greater than total traffic on all items in for .
Step 1. Consider items and a ranking defined by , for . Then, by popularity ranking, is a (stable) limit ranking of the process if and only if at :
holds for the expected individual . Suppose is such that the items are ranked in two blocks of items of the same class where, if , the first ranked items are all in and the remaining (bottom ranked) ones are all in ; and symmetrically if , the bottom ranked items are all in and the remaining (top ranked) ones are items in . To see that these constitute limit rankings, suppose , and consider the ranking , where the first items are ranked on top. In this case, the corresponding clicking probabilities for (i.e., ) are given by:
and for (i.e., ) satisfy:
Clearly, holds within the classes and , that is, for and for Hence it suffices to show that . This is easily checked for the expected individual (whose value of is drawn from according to the probabilities, respectively, and ).
Step 2: To see that the above ranking constitutes a unique limit, we show that no other ranking satisfies Eq. (3) and that, for any other ranking, whenever an item is in the less numerous class (say when ), it will always get strictly more clicks in expectation than the item of the other more numerous class ranked just above. Assume , then it can be checked that the two strongest constraints to be satisfied are the ones comparing the clicking probability on the lowest-ranked item of class when it is ranked in the th position while all remaining items are ranked in the first positions, and when it is ranked in position while all remaining items are ranked in the first positions. Both are easily seen to be satisfied whenever and . By continuity they continue to hold for sufficiently small . The case , which implies and which has the items ranked on top holds by symmetry.
Step 3. It suffices to show that whenever , then the first ranked items in always get strictly more that half the share of the total clicks. Since the share of traffic on the individual items is given by the probabilities defined above, it suffices to show that for any , . Given our assumptions on and , this is easily checked for the expected individual (using the fact that, for any , ). It also implies that when all items in will (be bottom-ranked) and will obtain less than half the total traffic.
When the share of items in is close to one half (), then there may be multiple limit rankings and the above proof no longer applies. We believe that the few-get-richer result may still go through in these cases, but it is necessary to evaluate the likelihood of the different limit rankings and guarantee that limit rankings are more likely to give classes with fewer items a higher probability of being higher ranked. This goes beyond the scope of this paper.
Figure 1 shows the proportion of clicks on the items () as a function of for the (expected) limit ranking and for different values of , assuming and . In this symmetric case, when has few items (minority case ) it obtains a larger proportion of clicks than when has many items (majority case ).
(right plot). Error bars indicate confidence intervals for 100 different random realizations.
4. Simulations of the Model
We now analyze through simulations the presence of the few-get-richer effect in different settings. Our focus is on analyzing the click-through rate (CTR), defined as the the ratio of the probability of clicking on an item in to the total number of clicks (which we fix at users), as a function of for different settings. We mainly consider stochastic choices with , which means that non-indifferent users do not exclusively click on one of the two classes. We assume that the ranking is proportional to the number of clicks that the item received at time . As before, we consider an initial ranking with the items at the bottom. In this case, however, the dynamics occur at a faster time-scale, allowing us to characterize the transient during which the minority class may reach the top of the ranking. This makes our analysis dependent on the initial conditions. For an item initially at position , we analyze a uniform initialization, with all items having one click.
4.1. Dependence on
We first consider the dependence on the ranking effect, parametrized by . Larger values of correspond to a stronger relative ranking effect compared to the propensities. Here, we consider a symmetric case with .
Figure 2 (left) shows the CTR for different values of . The resulting CTR is almost symmetric. We observe a monotonic decrease as a function of in all cases, showing, in particular, that the minority class always receives more clicks than the majority one.
We can differentiate between three cases, corresponding to small, intermediate and large values of , respectively. In general, larger values of lead to a relatively larger CTR for the minority class, due to relatively larger probability of clicking on top ranked items. For small values of , the CTR is the smallest, but even here the effect as a function of is particularly pronounced for small (and large) . For intermediate values of , the flat region for intermediate values of disappears and the CTR decreases monotonically with constant slope, indicating a decrease independent of . Finally, for larger values of , the CTR is largest, and the effect as a function of is smallest for small (and large) . In this extreme case, for , the single item at the bottom quickly reaches the top and attracts almost all the traffic, leaving almost no traffic to the remaining items of the other class.
From these simulations, we can conclude that the few-get-richer effect is robust to varying .
4.2. Dependence on
We now analyze the CTR as a function of the proportion of indifferent users . For this, we choose and vary the proportion of indifferent users while keeping . To better analyze this dependence, we consider extreme preferences, so that only indifferent users can really click on both items, that is, we assume .
Figure 2(middle) shows the results. We observe that, in this case, the few-get-richer effect is more pronounced as increases. Having a larger proportion of indifferent users results in relatively more clicks per item, and hence a larger amplification of ranking effect, which is a key ingredient for the few-get-richer effect to emerge. With less extreme preferences, e.g., , the effect in is still present, but less pronounced (data not shown).
We conclude that the indifferent users play a key role in amplifying the effect of the ranking, and that in general, having a larger proportion of them contributes importantly towards the few-get-richer effect.
4.3. Dependence on the ratio
So far we have considered cases where the distribution of user types was symmetric, . In practice this need not be the case since, for example, the minority class might also be preferred by a minority of users. We analyze the effect as a function of the log-ratio .
Figure 2(right) shows the results for different values of . As expected, the few-get-richer effect is more pronounced when there is a larger relative proportion of users that prefer the minority item. To see this, compare the blue line, where (there are three times more users who prefer the ‘minority’ item), and the purple line, where . We see that the effect is still present for this choice of parameters, even when there are seven times more users who prefer the majority item. Here, the proportion of indifferent users is set to . Consistent with the results of the previous subsection, if we increase and keep the same ratios, the effect becomes more pronounced.
We conclude that the effect is also robust to different ratios of proportions .
4.4. Ranking evolution
Finally, we illustrate the typical behavior of the ranking evolution for different values of , assuming our usual baseline parameter values for , and and uniform initialization. Figure 3 shows the results, which confirm the idea that minority items tend to move towards the top.
5. Online Experiment with Human Participants
To test the predictions of the model, we executed an online experiment in which participants clicked on one out of 20 possible options (pictures). The options belonged to two classes, and , just as in the model. As in the simulations reported above, we measure the effect of the popularity-based ranking of options as the total number of clicks attracted by items of type . To this end, we created 8 independent conditions. In conditions D1 to D4, the ordering of the options changed dynamically as a function of the number of clicks received by each option, with the most popular option (in terms of cumulative number of clicks) at the top of the screen and the least popular option at the bottom of the screen. The total number of options was 20 in all conditions. The conditions differed in terms of the number of options in and , as summarized in Table 1. The number of options in was 17, 12, 8, or 3 in conditions D1, D2, D3 and D4, respectively.
These four ‘dynamic’ conditions were matched to four ‘static’ conditions with the same sets of options and . In the static conditions, the ranking of options did not change over time and was set to the initial ranking in the matching dynamic condition. We denote these static conditions by S1, S2, S3, and S4.
Comparisons of matched pairs of conditions allows us to establish the causal effect of the ranking algorithm on the total traffic attracted by each option. Comparisons of the 4 dynamic conditions allows us to test the prediction that the total traffic attracted by options in decreases with the number of options in .
To keep the experimental setup as simple as possible, the options in our experiment were not news sources, but pictures of dogs and cats. In order to activate their preferences for cat or dog pictures, participants were first asked whether they were a ‘cat person’, a ‘dog person’, or ‘neither a cat person nor a dog person.’ Answers to this identity question allowed us to compute the proportion of participants in each of the 3 types discussed in the model section. Then participants were shown a set of 20 pictures in a vertical list. In this setup, is the set of cat pictures (initially at the top of the screen) and is the set of dog pictures (initially at the bottom). The initial popularity of all pictures was set to 0.
5.1. Methodological Details
We recruited 786 participants on Amazon Mechanical Turk. It was administered via a Qualtrics survey embedded in the Amazon Mechanical Turk webpage as an iframe. After signing up for the task, participants read the informed-consent form. Then they were randomly assigned to one of 8 conditions. They first answered a question about their type: “Are you more of a cat person or a dog person?” with three possible choices “I am a cat person” / “I am neither a cat person nor a dog person” / “I am a dog person.” On the next screen they were shown 20 buttons with “Please click on a photo from the following list of photos of cats and dogs and rate it according to your liking.” The buttons were displayed in a vertical list ranked in terms of popularity. Participants could initially see 3 to 4 buttons and had to scroll down to access the other buttons. After clicking a button, participants were shown the corresponding picture and gave it a rating of 1 to 5 stars. The rating task was presented as a reason to ask participants to select an item according to their preferences. The collected ratings are not discussed because our interest is only in the clicking behavior of the participant. Participants were paid $0.15 for their time.
|# Cats ()||3||8||12||17|
|# Dogs (||17||12||8||3|
|Dynamic Button Ordering|
|# participants||Cat person||34||30||24||29|
|Static Button Ordering|
|# participants||Cat person||34||30||25||33|
5.2.1. Participant Types
30% indicated they were ‘Cat persons’ (), 55% ‘Dog persons’ (), 15% neither ().
5.2.2. Total Traffic Attracted by (Dog pictures)
Dog pictures were initially ranked at the bottom the screen. First we discuss the effect of popularity based ranking of options on the total traffic attracted by Dog pictures. In the 4 dynamic conditions, Dog pictures attracted substantially more traffic than in the corresponding static condition (compare the two rows ‘Experiment’ in the top and bottom panels of Table 1). In other words, ordering the options in terms of popularity had a systematic effect on the share of traffic attracted by options that started at the bottom of the choice screen.
Unsurprisingly, in the 4 static conditions, the total traffic attracted by Dog pictures was lower than 50%. The set of Dog pictures attracted more traffic when there were relatively more Dog pictures. For example, with 17 Dog pictures, these attracted 44% of the traffic. By contrast, with 3 Dog pictures, these attracted 27% of the traffic.
The pattern is completely different in the dynamic conditions. First, in all 4 dynamic conditions, the total traffic attracted by Dog pictures was higher than 50%. The most important finding is that the total traffic attracted by Dog pictures was larger with just 3 Dog pictures (and 17 Cat pictures) than with 17 Dog pictures (and 3 Cat pictures)! Similarly, the total traffic attracted by Dog pictures was larger with 8 Dog pictures (and 12 Cat pictures) than with 12 Dog pictures (and 8 Cat pictures). These results are consistent with the predictions of our model. The total traffic attracted by Dog pictures did not decrease monotonically with the size of the set of Dog pictures (it is larger with 8 Dog pictures than with 3 Dog pictures). This pattern is seemingly inconsistent with the predictions of the simulations (Fig. 2). Yet, this lack of monotonicity can be explained by the fact that the conditions were not perfectly balanced with respect to the distribution of identity types of the participants. In condition D3 (with 8 Dog pictures) there were substantially more Dog lovers (64/99) than in condition D4 (with 3 Dog pictures (56/101). This is a chance event resulting from the fact that we elicited the identity type of the participants after the random assignment into conditions, thus we could not balance types within conditions.
5.2.3. Estimated Parameters of the Choice Model
We estimated the parameters of the stochastic choice model (eq.2) using MLE on the whole dataset. Our parameter estimates are , for Cat persons ( in the model), for Dog persons ( in the model). Cat and Dog persons had strong tendencies to choose pictures from the class consistent with their identity type.
We used these estimated parameters to simulate choices and the dynamics of picture ranks in the 8 experimental conditions. Simulations reported in rows ‘Sim1’ in Table 1 indicate a close match between the simulated data and the actual traffic proportions attracted by Dog pictures (both in static and dynamics conditions).
We also used the estimated parameters to simulate what would happen if there were 100 participants in each condition and if the distributions of identity types were the same in all conditions (‘Sim2’ rows). We find a decreasing monotonous relation between the number of Dog pictures and the share of traffic attracted by Dog pictures. This pattern is consistent with the qualitative prediction of our model.
Overall, the results indicate that the experimental setting falls within the boundary condition of application of our theory. They provide a proof-of-concept that popularity-based ordering of option can lead to the counter-intuitive phenomenon that when there are fewer options of one class, the total share of traffic attracted by this class of options becomes larger.
6. Discussion & Conclusion
The few-get-richer effect can have both positive and negative effects on the quality of the information people obtain from search results. On the positive side, when there are few relevant items, the few-get-richer effect may help them become top-ranked, making them more accessible. At the same time, the few-get-richer effect can contribute to the spread of misinformation. It may help few irrelevant or ‘fake news’ items become top-ranked, especially if there is a strong preference for such items and only few websites report them.
Our analyses highlight a potential unintended effect of regulations that ‘ban’ particular ‘alternative’ news sources known for spreading ‘fake news.’ When a sizeable proportion of users have a preference for identifiable ‘alternative’ news sources, the removal of some of these news sources, might lead to an increase in the total traffic attracted by the remaining ‘alternative’ news sources. This could result in more overall exposure to ‘fake news’!
To neutralize the few-get-richer effect, our theory suggests that it may be advisable to keep track of the number of items in each class when incorporating clicks in search engine algorithm. Ideally, the ranking algorithm should use the popularity of the different items in a way that is neutral to the number of items in each class.
The few-get-richer effect also has implications for the design of recommender systems. The learning efficiency of these systems is impeded by the presentation bias problem: items shown to the user can get clicks whereas items not shown get no clicks. The recommender system thus cannot learn about the relevance of the latter items. A popular solution to this challenge is the explore-and-exploit approach, in which some items from a minority class are randomly inserted in the search results (Agarwal et al., 2009). This purportedly increases the amount of exploration (clicks on the minority class), and thus increases the learning opportunities of the system at the cost of slightly hurting the user experience (McNee et al., 2006; Herlocker et al., 2004). The few-get-richer effect suggests precisely the opposite. Adding more of those items might reduce rather than increase the total amount of exploration.
F. Germano acknowledges financial support from the Spanish Ministry of Economy and Competitiveness (grant ECO2017-89240-P). G. Le Mens benefited from financial support from Ramon y Cajal Fellowship RYC-2014-15035 and Grant #AEI/FEDER UE-PSI2016-75353 from the Spanish MINECO, and ERC Consolidator Grant #772268 from the European Commission. V. Gómez is funded by the Ramon y Cajal program RYC-2015-18878 (AEI/MINECO/FSE,UE) and the María de Maeztu Units of Excellence Programme (MDM-2015-0502). The research leading to these results has received funding from “La Caixa” Banking Foundation.
- Agarwal et al. (2009) Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. 2009. Explore/exploit schemes for web content optimization. In Data Mining, 2009. ICDM’09. Ninth IEEE International Conference on. IEEE, 1–10.
- Baeza-Yates (2018) Ricardo Baeza-Yates. 2018. Bias on the Web. Commun. ACM 61, 6 (May 2018), 54–61.
- Banerjee (1992) Abhijit V Banerjee. 1992. A simple model of herd behavior. The quarterly journal of economics 107, 3 (1992), 797–817.
- Biega et al. (2018) Asia J. Biega, Krishna P. Gummadi, and Gerhard Weikum. 2018. Equity of Attention: Amortizing Individual Fairness in Rankings. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR ’18). ACM, New York, NY, USA, 405–414.
- Bikhchandani et al. (1992) Sushil Bikhchandani, David Hirshleifer, and Ivo Welch. 1992. A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of political Economy 100, 5 (1992), 992–1026.
- Castillo (2019) Carlos Castillo. 2019. Fairness and Transparency in Ranking. SIGIR Forum 52, 2 (Jan. 2019), 64–71.
Olivier Chapelle and Ya
A dynamic bayesian network click model for web search ranking. InProceedings of the 18th international conference on World wide web. ACM, 1–10.
- Che and Hörner (2018) Yeon-Koo Che and Johannes Hörner. 2018. Recommender Systems as Mechanisms for Social Learning. Quarterly Journal of Economics 133, 2 (2018), 875–921.
- Cheng et al. (2017) Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017. Anyone can become a troll: Causes of trolling behavior in online discussions. In CSCW: proceedings of the Conference on Computer-Supported Cooperative Work, Vol. 2017. NIH Public Access, 1217.
- Ciampaglia et al. (2018) Giovanni Luca Ciampaglia, Azadeh Nematzadeh, Filippo Menczer, and Alessandro Flammini. 2018. How algorithmic popularity bias hinders or promotes quality. Scientific Reports 8, 1 (2018), 15951–.
- Dandekar et al. (2013) Pranav Dandekar, Ashish Goel, and David T. Lee. 2013. Biased assimilation, homophily, and the dynamics of polarization. Proceedings of the National Academy of Sciences 110, 15 (2013), 5791–5796.
- Demange (2014) Gabrielle Demange. 2014. Collective attention and ranking methods. Journal of Dynamics & Games 1, 2164-6066 (2014), 17.
- Diaz (2008) Alejandro Diaz. 2008. Through the Google goggles: Sociopolitical bias in search engine design. In Web search. Springer, 11–34.
- Dupret and Piwowarski (2008) Georges E Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations.. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 331–338.
- Fortunato et al. (2006) Santo Fortunato, Alessandro Flammini, Filippo Menczer, and Alessandro Vespignani. 2006. Topical interests and the mitigation of search engine bias. Proceedings of the National Academy of Sciences 103, 34 (2006), 12684–12689.
- Germano and Sobbrio (2018) Fabrizio Germano and Francesco Sobbrio. 2018. Opinion Dynamics via Search Engines (and other Algorithmic Gatekeepers). ArXiv e-prints (Oct. 2018). arXiv:1810.06973
- Herlocker et al. (2004) Jonathan L Herlocker, Joseph A Konstan, Loren G Terveen, and John T Riedl. 2004. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS) 22, 1 (2004), 5–53.
- Joachims et al. (2017) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2017. Accurately interpreting clickthrough data as implicit feedback. In ACM SIGIR Forum, Vol. 51. Acm, 4–11.
- Kim et al. (2018) Jooyeon Kim, Behzad Tabibian, Alice Oh, Bernhard Schölkopf, and Manuel Gomez-Rodriguez. 2018. Leveraging the crowd to detect and reduce the spread of fake news and misinformation. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 324–332.
- Kremer et al. (2014) Ilan Kremer, Yishay Mansour, and Motty Perry. 2014. Implementing the “Wisdom of the Crowd”. Journal of Political Economy 122, 5 (2014), 988–1012.
- McNee et al. (2006) Sean M McNee, John Riedl, and Joseph A Konstan. 2006. Being accurate is not enough: how accuracy metrics have hurt recommender systems. In CHI’06 extended abstracts on Human factors in computing systems. ACM, 1097–1101.
- Mens et al. (2018) Gaël Le Mens, Balázs Kovács, Judith Avrahami, and Yaakov Kareev. 2018. How Endogenous Crowd Formation Undermines the Wisdom of the Crowd in Online Ratings. Psychological Science 29, 9 (2018), 1475–1490.
- Merton (1968) Robert K Merton. 1968. The Matthew effect in science: The reward and communication systems of science are considered. Science 159, 3810 (1968), 56–63.
- Muddiman and Stroud (2017) Ashley Muddiman and Natalie Jomini Stroud. 2017. News values, cognitive biases, and partisan incivility in comment sections. Journal of communication 67, 4 (2017), 586–609.
- Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th international conference on World Wide Web. ACM, 521–530.
- Salganik et al. (2006) Matthew J Salganik, Peter Sheridan Dodds, and Duncan J Watts. 2006. Experimental study of inequality and unpredictability in an artificial cultural market. science 311, 5762 (2006), 854–856.
- Vosoughi et al. (2018) Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science 359, 6380 (2018), 1146–1151.
- Wang et al. (2018) Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 610–618.
- Weng et al. (2012) Lillian Weng, Alessandro Flammini, Alessandro Vespignani, and Fillipo Menczer. 2012. Competition among memes in a world with limited attention. Scientific reports 2 (2012), 335.
- Zehlike et al. (2017) Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed, and Ricardo Baeza-Yates. 2017. Fa* ir: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 1569–1578.
- Zhang et al. (2011) Yuchen Zhang, Weizhu Chen, Dong Wang, and Qiang Yang. 2011. User-click modeling for understanding and predicting search-behavior. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1388–1396.