Evaluating Gender Bias in Machine Translation

06/03/2019
by   Gabriel Stanovsky, et al.
0

We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT). Our approach uses two recent coreference resolution datasets composed of English sentences which cast participants into non-stereotypical gender roles (e.g., "The doctor asked the nurse to help her in the operation"). We devise an automatic gender bias evaluation method for eight target languages with grammatical gender, based on morphological analysis (e.g., the use of female inflection for the word "doctor"). Our analyses show that four popular industrial MT systems and two recent state-of-the-art academic MT models are significantly prone to gender-biased translation errors for all tested target languages. Our data and code are made publicly available.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/24/2021

Extending Challenge Sets to Uncover Gender Bias in Machine Translation: Impact of Stereotypical Verbs and Adjectives

Human gender bias is reflected in language and text production. Because ...
research
11/02/2022

MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

As generic machine translation (MT) quality has improved, the need for t...
research
10/12/2020

Gender Coreference and Bias Evaluation at WMT 2020

Gender bias in machine translation can manifest when choosing gender inf...
research
10/27/2020

Evaluating Gender Bias in Speech Translation

The scientific community is more and more aware of the necessity to embr...
research
09/08/2021

Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation

Recent works have found evidence of gender bias in models of machine tra...
research
04/13/2021

Gender Bias in Machine Translation

Machine translation (MT) technology has facilitated our daily tasks by p...
research
05/28/2019

On Measuring Gender Bias in Translation of Gender-neutral Pronouns

Ethics regarding social bias has recently thrown striking issues in natu...

Please sign up or login with your details

Forgot password? Click here to reset