Estimating Position Bias without Intrusive Interventions

12/12/2018
by   Aman Agarwal, et al.
0

Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal. While it was recently shown how counterfactual learning-to-rank (LTR) approaches Joachims/etal/17a can provably overcome presentation bias when observation propensities are known, it remains to show how to effectively estimate these propensities. In this paper, we propose the first method for producing consistent propensity estimates without manual relevance judgments, disruptive interventions, or restrictive relevance modeling assumptions. First, we show how to harvest a specific type of intervention data from historic feedback logs of multiple different ranking functions, and show that this data is sufficient for consistent propensity estimation in the position-based model. Second, we propose a new extremum estimator that makes effective use of this data. In an empirical evaluation, we find that the new estimator provides superior propensity estimates in two real-world systems -- Arxiv Full-text Search and Google Drive Search. Beyond these two points, we find that the method is robust to a wide range of settings in simulation studies.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/09/2018

Consistent Position Bias Estimation without Online Interventions for Learning-to-Rank

Presentation bias is one of the key challenges when learning from implic...
research
11/05/2018

Intervention Harvesting for Context-Dependent Examination-Bias Estimation

Accurate estimates of examination bias are crucial for unbiased learning...
research
12/08/2020

Unifying Online and Counterfactual Learning to Rank

Optimizing ranking systems based on user interactions is a well-studied ...
research
08/16/2022

Approximated Doubly Robust Search Relevance Estimation

Extracting query-document relevance from the sparse, biased clickthrough...
research
07/19/2020

Counterfactual Learning to Rank using Heterogeneous Treatment Effect Estimation

Learning-to-Rank (LTR) models trained from implicit feedback (e.g. click...
research
05/23/2023

Counterfactual Augmentation for Multimodal Learning Under Presentation Bias

In real-world machine learning systems, labels are often derived from us...
research
04/30/2018

Counterfactual Learning-to-Rank for Additive Metrics and Deep Models

Implicit feedback (e.g., clicks, dwell times) is an attractive source of...

Please sign up or login with your details

Forgot password? Click here to reset