An Empirical Comparison of Bias Reduction Methods on Real-World Problems in High-Stakes Policy Settings

05/13/2021
by   Hemank Lamba, et al.
17

Applications of machine learning (ML) to high-stakes policy settings – such as education, criminal justice, healthcare, and social service delivery – have grown rapidly in recent years, sparking important conversations about how to ensure fair outcomes from these systems. The machine learning research community has responded to this challenge with a wide array of proposed fairness-enhancing strategies for ML models, but despite the large number of methods that have been developed, little empirical work exists evaluating these methods in real-world settings. Here, we seek to fill this research gap by investigating the performance of several methods that operate at different points in the ML pipeline across four real-world public policy and social good problems. Across these problems, we find a wide degree of variability and inconsistency in the ability of many of these methods to improve model fairness, but post-processing by choosing group-specific score thresholds consistently removes disparities, with important implications for both the ML research community and practitioners deploying machine learning to inform consequential policy decisions.

READ FULL TEXT
12/05/2020

Machine learning for public policy: Do we need to sacrifice accuracy to make models fair?

Growing applications of machine learning in policy settings have raised ...
02/17/2022

Does the End Justify the Means? On the Moral Justification of Fairness-Aware Machine Learning

Despite an abundance of fairness-aware machine learning (fair-ml) algori...
06/06/2022

A Human-Centric Take on Model Monitoring

Predictive models are increasingly used to make various consequential de...
09/16/2022

FairGBM: Gradient Boosting with Fairness Constraints

Machine Learning (ML) algorithms based on gradient boosted decision tree...
12/17/2019

How Personal is Machine Learning Personalization?

Though used extensively, the concept and process of machine learning (ML...
10/27/2020

Explainable Machine Learning for Public Policy: Use Cases, Gaps, and Research Directions

In Machine Learning (ML) models used for supporting decisions in high-st...
06/10/2020

Fair Data Integration

The use of machine learning (ML) in high-stakes societal decisions has e...