Algorithmic Fairness Datasets: the Story so Far

02/03/2022
by   Alessandro Fabris, et al.
0

Data-driven algorithms are being studied and deployed in diverse domains to support critical decisions, directly impacting on people's well-being. As a result, a growing community of algorithmic fairness researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of the risks and opportunities of automated decision-making for different populations. Algorithmic fairness progress hinges on data, which can be used appropriately only if adequately documented. Unfortunately, the algorithmic fairness community, as a whole, suffers from a collective data documentation debt caused by a lack of information on specific resources (opacity) and scatteredness of available information (sparsity). In this work, we survey over two hundred datasets employed in algorithmic fairness research, producing standardized and searchable documentation for each of them, along with in-depth documentation for the three most popular fairness datasets, namely Adult, COMPAS and German Credit. These documentation efforts support multiple contributions. Firstly, we summarize the merits and limitations of popular algorithmic fairness datasets, questioning their suitability as general-purpose fairness benchmarks. Secondly, we document hundreds of available alternatives, annotating their domain and supported fairness tasks, to assist dataset users in task-oriented and domain-oriented search. Finally, we analyze these resources from the perspective of five important data curation topics: anonymization, consent, inclusivity, labeling of sensitive attributes, and transparency. We discuss different approaches and levels of attention to these topics, making them tangible, and distill them into a set of best practices for the curation of novel datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/15/2019

Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges

Machine learning algorithms are now frequently used in sensitive context...
research
10/01/2021

A survey on datasets for fairness-aware machine learning

As decision-making increasingly relies on machine learning and (big) dat...
research
06/10/2021

It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks

Risk assessment instrument (RAI) datasets, particularly ProPublica's COM...
research
05/17/2019

Contrastive Fairness in Machine Learning

We present contrastive fairness, a new direction in causal inference app...
research
09/04/2022

FairSNA: Algorithmic Fairness in Social Network Analysis

In recent years, designing fairness-aware methods has received much atte...
research
10/27/2017

The Algorithmic-Autoregulation (AA) Methodology and Software: a collective focus on self-transparency

There are numerous efforts to achieve a lightweight and systematic accou...
research
04/13/2021

Fairness in Rankings and Recommendations: An Overview

We increasingly depend on a variety of data-driven algorithmic systems t...

Please sign up or login with your details

Forgot password? Click here to reset