Estimating Error and Bias in Offline Evaluation Results

01/26/2020
by   Mucun Tian, et al.
0

Offline evaluations of recommender systems attempt to estimate users' satisfaction with recommendations using static data from prior user interactions. These evaluations provide researchers and developers with first approximations of the likely performance of a new system and help weed out bad ideas before presenting them to users. However, offline evaluation cannot accurately assess novel, relevant recommendations, because the most novel items were previously unknown to the user, so they are missing from the historical data and cannot be judged as relevant. We present a simulation study to estimate the error that such missing data causes in commonly-used evaluation metrics in order to assess its prevalence and impact. We find that missing data in the rating or observation process causes the evaluation protocol to systematically mis-estimate metric values, and in some cases erroneously determine that a popularity-based recommender outperforms even a perfect personalized recommender. Substantial breakthroughs in recommendation quality, therefore, will be difficult to assess with existing offline techniques.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/27/2019

User Validation of Recommendation Serendipity Metrics

Though it has been recognized that recommending serendipitous (i.e., sur...
11/13/2014

DUM: Diversity-Weighted Utility Maximization for Recommendations

The need for diversification of recommendation lists manifests in a numb...
06/05/2021

PURS: Personalized Unexpected Recommender System for Improving User Satisfaction

Classical recommender system methods typically face the filter bubble pr...
09/05/2019

Assessing Fashion Recommendations: A Multifaceted Offline Evaluation Approach

Fashion is a unique domain for developing recommender systems (RS). Pers...
01/08/2019

Using offline metrics and user behavior analysis to combine multiple systems for music recommendation

There are many offline metrics that can be used as a reference for evalu...
04/16/2019

An Evaluation Framework for Interactive Recommender System

Traditional recommender systems present a relatively static list of reco...
09/10/2018

Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce

In this paper, we present our work towards comparing on-line and off-lin...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.