Online Evaluation Methods for the Causal Effect of Recommendations

07/14/2021
by   Masahiro Sato, et al.
0

Evaluating the causal effect of recommendations is an important objective because the causal effect on user interactions can directly leads to an increase in sales and user engagement. To select an optimal recommendation model, it is common to conduct A/B testing to compare model performance. However, A/B testing of causal effects requires a large number of users, making such experiments costly and risky. We therefore propose the first interleaving methods that can efficiently compare recommendation models in terms of causal effects. In contrast to conventional interleaving methods, we measure the outcomes of both items on an interleaved list and items not on the interleaved list, since the causal effect is the difference between outcomes with and without recommendations. To ensure that the evaluations are unbiased, we either select items with equal probability or weight the outcomes using inverse propensity scores. We then verify the unbiasedness and efficiency of online evaluation methods through simulated online experiments. The results indicate that our proposed methods are unbiased and that they have superior efficiency to A/B testing.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/11/2020

Unbiased Learning for the Causal Effect of Recommendation

Increasing users' positive interactions, such as purchases or clicks, is...
research
04/10/2019

Causal Embeddings for Recommendation: An Extended Abstract

Recommendations are commonly used to modify user's natural behavior, for...
research
05/28/2021

CausCF: Causal Collaborative Filtering for RecommendationEffect Estimation

To improve user experience and profits of corporations, modern industria...
research
12/17/2020

Causality-Aware Neighborhood Methods for Recommender Systems

The business objectives of recommenders, such as increasing sales, are a...
research
06/01/2023

Causal Estimation of User Learning in Personalized Systems

In online platforms, the impact of a treatment on an observed outcome ma...
research
10/15/2022

Fair Effect Attribution in Parallel Online Experiments

A/B tests serve the purpose of reliably identifying the effect of change...
research
06/01/2022

In the Eye of the Beholder: Robust Prediction with Causal User Modeling

Accurately predicting the relevance of items to users is crucial to the ...

Please sign up or login with your details

Forgot password? Click here to reset