DeepAI
Log In Sign Up

Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank

08/20/2020
by   Tao Yang, et al.
0

Leveraging biased click data for optimizing learning to rank systems has been a popular approach in information retrieval. Because click data is often noisy and biased, a variety of methods have been proposed to construct unbiased learning to rank (ULTR) algorithms for the learning of unbiased ranking models. Among them, automatic unbiased learning to rank (AutoULTR) algorithms that jointly learn user bias models (i.e., propensity models) with unbiased rankers have received a lot of attention due to their superior performance and low deployment cost in practice. Despite their differences in theories and algorithm design, existing studies on ULTR usually use uni-variate ranking functions to score each document or result independently. On the other hand, recent advances in context-aware learning-to-rank models have shown that multivariate scoring functions, which read multiple documents together and predict their ranking scores jointly, are more powerful than uni-variate ranking functions in ranking tasks with human-annotated relevance labels. Whether such superior performance would hold in ULTR with noisy data, however, is mostly unknown. In this paper, we investigate existing multivariate scoring functions and AutoULTR algorithms in theory and prove that permutation invariance is a crucial factor that determines whether a context-aware learning-to-rank model could be applied to existing AutoULTR framework. Our experiments with synthetic clicks on two large-scale benchmark datasets show that AutoULTR models with permutation-invariant multivariate scoring functions significantly outperform those with uni-variate scoring functions and permutation-variant multivariate scoring functions.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/12/2019

SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval

In learning-to-rank for information retrieval, a ranking model is automa...
05/18/2020

Unbiased Learning to Rank via Propensity Ratio Scoring

Implicit feedback, such as user clicks, is a major source of supervision...
11/22/2019

Responsible Scoring Mechanisms Through Function Sampling

Human decision-makers often receive assistance from data-driven algorith...
12/17/2020

Binomial Tails for Community Analysis

An important task of community discovery in networks is assessing signif...
10/21/2019

Self-Attentive Document Interaction Networks for Permutation Equivariant Ranking

How to leverage cross-document interactions to improve ranking performan...
06/03/2022

Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank

Unbiased learning to rank (ULTR) aims to train an unbiased ranking model...
09/11/2017

A Short Note on Proximity-based Scoring of Documents with Multiple Fields

The BM25 ranking function is one of the most well known query relevance ...

Code Repositories

CIKM_2020_Multivariate_AutoULTR

This is the repository for reproduce our CIKM 2020 paper, Analysis of Multivariate Scoring Functions for AutomaticUnbiased Learning to Rank


view repo