A Re-visit of the Popularity Baseline in Recommender Systems

by   Yitong Ji, et al.

Popularity is often included in experimental evaluation to provide a reference performance for a recommendation task. To understand how popularity baseline is defined and evaluated, we sample 12 papers from top-tier conferences including KDD, WWW, SIGIR, and RecSys, and 6 open source toolkits. We note that the widely adopted MostPop baseline simply ranks items based on the number of interactions in the training data. We argue that the current evaluation of popularity (i) does not reflect the popular items at the time when a user interacts with the system, and (ii) may recommend items released after a user's last interaction with the system. On the widely used MovieLens dataset, we show that the performance of popularity could be significantly improved by 70 when a user interacts with the system. We further show that, on MovieLens dataset, the users having lower tendencies on movies tend to follow the crowd and rate more popular movies. Movie lovers who rate a large number of movies, rate movies based on their own preferences and interests. Through this study, we call for a re-visit of the popularity baseline in recommender system to better reflect its effectiveness.


page 1

page 2

page 3

page 4


The Unfairness of Popularity Bias in Recommendation

Recommender systems are known to suffer from the popularity bias problem...

The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study

Research has shown that recommender systems are typically biased towards...

Quantifying and Mitigating Popularity Bias in Conversational Recommender Systems

Conversational recommender systems (CRS) have shown great success in acc...

From Counter-intuitive Observations to a Fresh Look at Recommender System

Recently, a few papers report counter-intuitive observations made from e...

The few-get-richer: a surprising consequence of popularity-based rankings

Ranking algorithms play a crucial role in online platforms ranging from ...

Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

Recommendation algorithms are susceptible to popularity bias: a tendency...

Investigating the Robustness of Sequential Recommender Systems Against Training Data Perturbations: an Empirical Study

Sequential Recommender Systems (SRSs) have been widely used to model use...

Please sign up or login with your details

Forgot password? Click here to reset