DeepAI AI Chat
Log In Sign Up

Inverse Propensity Score based offline estimator for deterministic ranking lists using position bias

08/31/2022
by   Nick Wood, et al.
0

In this work, we present a novel way of computing IPS using a position-bias model for deterministic logging policies. This technique significantly widens the policies on which OPE can be used. We validate this technique using two different experiments on industry-scale data. The OPE results are clearly strongly correlated with the online results, with some constant bias. The estimator requires the examination model to be a reasonably accurate approximation of real user behaviour.

READ FULL TEXT

page 1

page 2

page 3

page 4

08/24/2020

When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank

Besides position bias, which has been well-studied, trust bias is anothe...
11/05/2018

Intervention Harvesting for Context-Dependent Examination-Bias Estimation

Accurate estimates of examination bias are crucial for unbiased learning...
02/03/2022

Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model

In real-world recommender systems and search engines, optimizing ranking...
01/18/2021

Mitigating the Position Bias of Transformer Models in Passage Re-Ranking

Supervised machine learning models and their evaluation strongly depends...
09/17/2019

Ranking metrics on non-shuffled traffic

Ranking metrics are a family of metrics largely used to evaluate recomme...