Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce

09/10/2018
by   Ladislav Peska, et al.
0

In this paper, we present our work towards comparing on-line and off-line evaluation metrics in the context of small e-commerce recommender systems. Recommending on small e-commerce enterprises are rather challenging due to the lower volume of interactions and low user loyalty, rarely extending beyond a single session. On the other hand, we usually have to deal with lower volumes of objects, which are easier to discover by users through various browsing/searching GUIs. The main goal of this paper is to determine applicability of off-line evaluation metrics in learning true usability of recommender systems (evaluated on-line in A/B testing). In total 800 variants of recommending algorithms were evaluated off-line w.r.t. 18 metrics covering rating-based, ranking-based, novelty and diversity evaluation. The off-line results were afterwards compared with on-line evaluation of 12 selected recommender variants. Off-line results shown a great variance in performance w.r.t. different metrics with the Pareto front covering 68% of the approaches. On-line metrics correlates positively with ranking-based metrics (AUC, MRR, nDCG), while too high values of diversity and novelty had a negative impact on the on-line results. We further train two regressors to predict on-line results based on the off-line metrics and estimate performance of recommenders not evaluated in A/B testing directly.

READ FULL TEXT

page 5

page 6

research
12/06/2022

Pareto Pairwise Ranking for Fairness Enhancement of Recommender Systems

Learning to rank is an effective recommendation approach since its intro...
research
06/29/2019

One Size Does Not Fit All: Modeling Users' Personal Curiosity in Recommender Systems

Today's recommender systems are criticized for recommending items that a...
research
08/28/2017

It's Time to Consider "Time" when Evaluating Recommender-System Algorithms [Proposal]

In this position paper, we question the current practice of calculating ...
research
09/12/2023

Distributionally-Informed Recommender System Evaluation

Current practice for evaluating recommender systems typically focuses on...
research
03/13/2022

Exploring Customer Price Preference and Product Profit Role in Recommender Systems

Most of the research in the recommender systems domain is focused on the...
research
06/08/2018

Conversational Recommender System

A personalized conversational sales agent could have much commercial pot...
research
10/15/2019

How to eliminate detour behaviors in E-hailing: On-line detection and Pricing regulation

With the fast development of information and communication technology (I...

Please sign up or login with your details

Forgot password? Click here to reset