Towards WinoQueer: Developing a Benchmark for Anti-Queer Bias in Large Language Models

06/23/2022
by   Virginia K. Felkner, et al.
0

This paper presents exploratory work on whether and to what extent biases against queer and trans people are encoded in large language models (LLMs) such as BERT. We also propose a method for reducing these biases in downstream tasks: finetuning the models on data written by and/or about queer people. To measure anti-queer bias, we introduce a new benchmark dataset, WinoQueer, modeled after other bias-detection benchmarks but addressing homophobic and transphobic biases. We found that BERT shows significant homophobic bias, but this bias can be mostly mitigated by finetuning BERT on a natural language corpus written by members of the LGBTQ+ community.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/26/2023

WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models

We present WinoQueer: a benchmark specifically designed to measure wheth...
research
01/14/2021

Persistent Anti-Muslim Bias in Large Language Models

It has been observed that large-scale language models capture undesirabl...
research
06/27/2023

Gender Bias in BERT – Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task

Pretrained language models are publicly available and constantly finetun...
research
09/15/2023

Casteist but Not Racist? Quantifying Disparities in Large Language Model Bias between India and the West

Large Language Models (LLMs), now used daily by millions of users, can e...
research
05/22/2023

Keeping Up with the Language Models: Robustness-Bias Interplay in NLI Data and Models

Auditing unwanted social bias in language models (LMs) is inherently har...
research
09/08/2021

Sustainable Modular Debiasing of Language Models

Unfair stereotypical biases (e.g., gender, racial, or religious biases) ...
research
03/22/2022

Suum Cuique: Studying Bias in Taboo Detection with a Community Perspective

Prior research has discussed and illustrated the need to consider lingui...

Please sign up or login with your details

Forgot password? Click here to reset