Log In Sign Up

Estimation of Fair Ranking Metrics with Incomplete Judgments

by   Omer Kirnap, et al.

There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identified using protected attributes such as gender or ethnicity. To date, these metrics typically assume the availability and completeness of protected attribute labels of items. However, the protected attributes of individuals are rarely present, limiting the application of fair ranking metrics in large scale systems. In order to address this problem, we propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items. We evaluate our approach using both simulated and real world data. Our experimental results demonstrate that our method can estimate this family of fair ranking metrics and provides a robust, reliable alternative to exhaustive or random data annotation.


page 1

page 2

page 3

page 4


Comparing Fair Ranking Metrics

Ranking is a fundamental aspect of recommender systems. However, ranked ...

Fair Ranking with Noisy Protected Attributes

The fair-ranking problem, which asks to rank a given set of items to max...

Mitigating Bias in Set Selection with Noisy Protected Attributes

Subset selection algorithms are ubiquitous in AI-driven applications, in...

MANI-Rank: Multiple Attribute and Intersectional Group Fairness for Consensus Ranking

Combining the preferences of many rankers into one single consensus rank...

A Ranking Approach to Fair Classification

Algorithmic decision systems are increasingly used in areas such as hiri...

Reducing Disparate Exposure in Ranking: A Learning To Rank Approach

In this paper we consider a ranking problem in which we would like to or...

Non-deterministic Behavior of Ranking-based Metrics when Evaluating Embeddings

Embedding data into vector spaces is a very popular strategy of pattern ...