Estimating Error and Bias in Offline Evaluation Results

by   Mucun Tian, et al.

Offline evaluations of recommender systems attempt to estimate users' satisfaction with recommendations using static data from prior user interactions. These evaluations provide researchers and developers with first approximations of the likely performance of a new system and help weed out bad ideas before presenting them to users. However, offline evaluation cannot accurately assess novel, relevant recommendations, because the most novel items were previously unknown to the user, so they are missing from the historical data and cannot be judged as relevant. We present a simulation study to estimate the error that such missing data causes in commonly-used evaluation metrics in order to assess its prevalence and impact. We find that missing data in the rating or observation process causes the evaluation protocol to systematically mis-estimate metric values, and in some cases erroneously determine that a popularity-based recommender outperforms even a perfect personalized recommender. Substantial breakthroughs in recommendation quality, therefore, will be difficult to assess with existing offline techniques.



There are no comments yet.


page 1

page 2

page 3

page 4


User Validation of Recommendation Serendipity Metrics

Though it has been recognized that recommending serendipitous (i.e., sur...

DUM: Diversity-Weighted Utility Maximization for Recommendations

The need for diversification of recommendation lists manifests in a numb...

PURS: Personalized Unexpected Recommender System for Improving User Satisfaction

Classical recommender system methods typically face the filter bubble pr...

Assessing Fashion Recommendations: A Multifaceted Offline Evaluation Approach

Fashion is a unique domain for developing recommender systems (RS). Pers...

Using offline metrics and user behavior analysis to combine multiple systems for music recommendation

There are many offline metrics that can be used as a reference for evalu...

An Evaluation Framework for Interactive Recommender System

Traditional recommender systems present a relatively static list of reco...

Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce

In this paper, we present our work towards comparing on-line and off-lin...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.