Evaluating Fairness in Argument Retrieval

Existing commercial search engines often struggle to represent different perspectives of a search query. Argument retrieval systems address this limitation of search engines and provide both positive (PRO) and negative (CON) perspectives about a user's information need on a controversial topic (e.g., climate change). The effectiveness of such argument retrieval systems is typically evaluated based on topical relevance and argument quality, without taking into account the often differing number of documents shown for the argument stances (PRO or CON). Therefore, systems may retrieve relevant passages, but with a biased exposure of arguments. In this work, we analyze a range of non-stochastic fairness-aware ranking and diversity metrics to evaluate the extent to which argument stances are fairly exposed in argument retrieval systems. Using the official runs of the argument retrieval task Touché at CLEF 2020, as well as synthetic data to control the amount and order of argument stances in the rankings, we show that systems with the best effectiveness in terms of topical relevance are not necessarily the most fair or the most diverse in terms of argument stance. The relationships we found between (un)fairness and diversity metrics shed light on how to evaluate group fairness – in addition to topical relevance – in argument retrieval settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/16/2021

FAIR: Fairness-Aware Information Retrieval Evaluation

With the emerging needs of creating fairness-aware solutions for search ...
research
11/04/2020

Diversity Aware Relevance Learning for Argument Search

In this work, we focus on the problem of retrieving relevant arguments f...
research
05/06/2023

Fairness in Image Search: A Study of Occupational Stereotyping in Image Retrieval and its Debiasing

Multi-modal search engines have experienced significant growth and wides...
research
10/27/2020

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics

The way pages are ranked in search results influences whether the users ...
research
05/16/2022

The Diversity of Argument-Making in the Wild: from Assumptions and Definitions to Causation and Anecdote in Reddit's "Change My View"

What kinds of arguments do people make, and what effect do they have on ...
research
05/03/2022

Assessing Confidence with Assurance 2.0

An assurance case is intended to provide justifiable confidence in the t...
research
05/09/2023

The Role of Relevance in Fair Ranking

Online platforms mediate access to opportunity: relevance-based rankings...

Please sign up or login with your details

Forgot password? Click here to reset