Investigating Human + Machine Complementarity for Recidivism Predictions

08/28/2018
by   Sarah Tan, et al.
0

When might human input help (or not) when assessing risk in fairness-related domains? Dressel and Farid asked Mechanical Turk workers to evaluate a subset of individuals in the ProPublica COMPAS data set for risk of recidivism, and concluded that COMPAS predictions were no more accurate or fair than predictions made by humans. We delve deeper into this claim in this paper. We construct a Human Risk Score based on the predictions made by multiple Mechanical Turk workers on the same individual, study the agreement and disagreement between COMPAS and Human Scores on subgroups of individuals, and construct hybrid Human+AI models to predict recidivism. Our key finding is that on this data set, human and COMPAS decision making differed, but not in ways that could be leveraged to significantly improve ground truth prediction. We present the results of our analyses and suggestions for how machine and human input may have complementary strengths to address challenges in the fairness domain.

READ FULL TEXT
research
11/29/2017

Paradoxes in Fair Computer-Aided Decision Making

Computer-aided decision making, where some classifier (e.g., an algorith...
research
04/21/2022

People are not coins. Morally distinct types of predictions necessitate different fairness constraints

A recent paper (Hedden 2021) has argued that most of the group fairness ...
research
11/30/2020

Towards Auditability for Fairness in Deep Learning

Group fairness metrics can detect when a deep learning model behaves dif...
research
11/03/2022

Making Machine Learning Datasets and Models FAIR for HPC: A Methodology and Case Study

The FAIR Guiding Principles aim to improve the findability, accessibilit...
research
01/16/2020

Fairness and Decision-making in Collaborative Shift Scheduling Systems

The strains associated with shift work decrease healthcare workers' well...
research
06/22/2021

Machine learning for risk assessment in gender-based crime

Gender-based crime is one of the most concerning scourges of contemporar...

Please sign up or login with your details

Forgot password? Click here to reset