[Re] Badder Seeds: Reproducing the Evaluation of Lexical Methods for Bias Measurement

06/03/2022
by   Jille van der Togt, et al.
0

Combating bias in NLP requires bias measurement. Bias measurement is almost always achieved by using lexicons of seed terms, i.e. sets of words specifying stereotypes or dimensions of interest. This reproducibility study focuses on the original authors' main claim that the rationale for the construction of these lexicons needs thorough checking before usage, as the seeds used for bias measurement can themselves exhibit biases. The study aims to evaluate the reproducibility of the quantitative and qualitative results presented in the paper and the conclusions drawn thereof. We reproduce most of the results supporting the original authors' general claim: seed sets often suffer from biases that affect their performance as a baseline for bias metrics. Generally, our results mirror the original paper's. They are slightly different on select occasions, but not in ways that undermine the paper's general intent to show the fragility of seed sets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/12/2023

With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector

This work presents our efforts to reproduce the results of the human eva...
research
11/24/2022

Undesirable biases in NLP: Averting a crisis of measurement

As Natural Language Processing (NLP) technology rapidly develops and spr...
research
11/30/2019

[Re] Learning to Learn By Self-Critique

This work is a reproducibility study of the paper of Antoniou and Storke...
research
12/08/2021

Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study

Screening or assessing studies is critical to the quality and outcomes o...
research
04/12/2022

Quantified Reproducibility Assessment of NLP Results

This paper describes and tests a method for carrying out quantified repr...
research
05/14/2021

RC2020 Report: Learning De-biased Representations with Biased Representations

As part of the ML Reproducibility Challenge 2020, we investigated the IC...
research
07/19/2023

Bias in Internet Measurement Platforms

Network operators and researchers frequently use Internet measurement pl...

Please sign up or login with your details

Forgot password? Click here to reset