Systematic Evaluation of Predictive Fairness

10/17/2022
by   Xudong Han, et al.
0

Mitigating bias in training on biased datasets is an important open problem. Several techniques have been proposed, however the typical evaluation regime is very limited, considering very narrow data conditions. For instance, the effect of target class imbalance and stereotyping is under-studied. To address this gap, we examine the performance of various debiasing methods across multiple tasks, spanning binary classification (Twitter sentiment), multi-class classification (profession prediction), and regression (valence prediction). Through extensive experimentation, we find that data conditions have a strong influence on relative model performance, and that general conclusions cannot be drawn about method efficacy when evaluating only on standard datasets, as is current practice in fairness research.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/06/2023

Can Domain Adaptation Improve Accuracy and Fairness of Skin Lesion Classification?

Deep learning-based diagnostic system has demonstrated potential in clas...
research
10/08/2021

Measure Twice, Cut Once: Quantifying Bias and Fairness in Deep Neural Networks

Algorithmic bias is of increasing concern, both to the research communit...
research
09/21/2021

Fairness-aware Class Imbalanced Learning

Class imbalance is a common challenge in many NLP tasks, and has clear c...
research
04/11/2023

Learning Optimal Fair Scoring Systems for Multi-Class Classification

Machine Learning models are increasingly used for decision making, in pa...
research
07/06/2023

Through the Fairness Lens: Experimental Analysis and Evaluation of Entity Matching

Entity matching (EM) is a challenging problem studied by different commu...
research
06/14/2019

Binary Classification using Pairs of Minimum Spanning Trees or N-ary Trees

One-class classifiers are trained with target class only samples. Intuit...
research
09/14/2021

Predicting Loss Risks for B2B Tendering Processes

Sellers and executives who maintain a bidding pipeline of sales engageme...

Please sign up or login with your details

Forgot password? Click here to reset