User Validation of Recommendation Serendipity Metrics

by   Li Chen, et al.

Though it has been recognized that recommending serendipitous (i.e., surprising and relevant) items can be helpful for increasing users' satisfaction and behavioral intention, how to measure serendipity in the offline environment is still an open issue. In recent years, a number of metrics have been proposed, but most of them were based on researchers' assumptions due to the serendipity's subjective nature. In order to validate these metrics' actual performance, we collected over 10,000 users' real feedback data and compared with the metrics' results. It turns out the user profile based metrics, especially content-based ones, perform better than those based on item popularity, in terms of estimating the unexpectedness facet of recommendations. Moreover, the full metrics, which involve the unexpectedness component, relevance, timeliness, and user curiosity, can more accurately indicate the recommendation's serendipity degree, relative to those that just involve some of them. The application of these metrics to several recommender algorithms further consolidates their practical usage, because the comparison results are consistent with those from user evaluation. Thus, this work is constructive for filling the gap between offline measurement and user study on recommendation serendipity.



page 1

page 2

page 3

page 4


Estimating Error and Bias in Offline Evaluation Results

Offline evaluations of recommender systems attempt to estimate users' sa...

Connecting User and Item Perspectives in Popularity Debiasing for Collaborative Recommendation

Recommender systems learn from historical data that is often non-uniform...

Evaluation Metrics for Item Recommendation under Sampling

The task of item recommendation requires ranking a large catalogue of it...

Assessing Fashion Recommendations: A Multifaceted Offline Evaluation Approach

Fashion is a unique domain for developing recommender systems (RS). Pers...

A new system-wide diversity measure for recommendations with efficient algorithms

Recommender systems often operate on item catalogs clustered by genres, ...

A Next Basket Recommendation Reality Check

The goal of a next basket recommendation (NBR) system is to recommend it...

From 3-DoF to 6-DoF: New Metrics to Analyse Users Behaviour in Immersive Applications

This work focuses on enabling user-centric immersive systems, in which e...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.