Evaluating Inclusivity, Equity, and Accessibility of NLP Technology: A Case Study for Indian Languages

05/25/2022
by   Simran Khanuja, et al.
0

In order for NLP technology to be widely applicable and useful, it needs to be inclusive of users across the world's languages, equitable, i.e., not unduly biased towards any particular language, and accessible to users, particularly in low-resource settings where compute constraints are common. In this paper, we propose an evaluation paradigm that assesses NLP technologies across all three dimensions, hence quantifying the diversity of users they can serve. While inclusion and accessibility have received attention in recent literature, equity is currently unexplored. We propose to address this gap using the Gini coefficient, a well-established metric used for estimating societal wealth inequality. Using our paradigm, we highlight the distressed state of diversity of current technologies for Indian (IN) languages, motivated by their linguistic diversity and large, varied speaker population. To improve upon these metrics, we demonstrate the importance of region-specific choices in model building and dataset creation and also propose a novel approach to optimal resource allocation during fine-tuning. Finally, we discuss steps that must be taken to mitigate these biases and call upon the community to incorporate our evaluation paradigm when building linguistically diverse technologies.

READ FULL TEXT
research
03/16/2022

Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Go

Aligning with ACL 2022 special Theme on "Language Diversity: from Low Re...
research
09/20/2022

NLP for Language Varieties of Italy: Challenges and the Path Forward

Italy is characterized by a one-of-a-kind linguistic diversity landscape...
research
04/20/2020

The State and Fate of Linguistic Diversity and Inclusion in the NLP World

Language technologies contribute to promoting multilingualism and lingui...
research
06/01/2022

What a Creole Wants, What a Creole Needs

In recent years, the natural language processing (NLP) community has giv...
research
06/15/2022

Location-based Twitter Filtering for the Creation of Low-Resource Language Datasets in Indonesian Local Languages

Twitter contains an abundance of linguistic data from the real world. We...
research
10/16/2022

Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World

Linguistic disparity in the NLP world is a problem that has been widely ...
research
03/26/2020

FFR V1.0: Fon-French Neural Machine Translation

Africa has the highest linguistic diversity in the world. On account of ...

Please sign up or login with your details

Forgot password? Click here to reset