On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations

03/25/2022
by   Yang Trista Cao, et al.
0

Multiple metrics have been introduced to measure fairness in various natural language processing tasks. These metrics can be roughly categorized into two categories: 1) extrinsic metrics for evaluating fairness in downstream applications and 2) intrinsic metrics for estimating fairness in upstream contextualized language representation models. In this paper, we conduct an extensive correlation study between intrinsic and extrinsic metrics across bias notions using 19 contextualized language models. We find that intrinsic and extrinsic metrics do not necessarily correlate in their original setting, even when correcting for metric misalignments, noise in evaluation datasets, and confounding factors such as experiment configuration for extrinsic metrics.

READ FULL TEXT
research
08/20/2023

A Survey on Fairness in Large Language Models

Large language models (LLMs) have shown powerful performance and develop...
research
12/14/2021

Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models

An increasing awareness of biased patterns in natural language processin...
research
01/30/2023

How Far Can It Go?: On Intrinsic Gender Bias Mitigation for Text Classification

To mitigate gender bias in contextualized language models, different int...
research
11/16/2022

Holistic Evaluation of Language Models

Language models (LMs) are becoming the foundation for almost all major l...
research
05/04/2022

fairlib: A Unified Framework for Assessing and Improving Classification Fairness

This paper presents fairlib, an open-source framework for assessing and ...
research
05/18/2021

Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Twitter uses machine learning to crop images, where crops are centered a...
research
09/17/2023

Do Large GPT Models Discover Moral Dimensions in Language Representations? A Topological Study Of Sentence Embeddings

As Large Language Models are deployed within Artificial Intelligence sys...

Please sign up or login with your details

Forgot password? Click here to reset