Fairness research in Machine Learning (ML) has seen rapid growth in recent years; however, it remains largely rooted in Western concerns and histories: the injustices they focus on (e.g., along race and gender), the datasets they study (e.g., ImageNet), the measurement scales they use (e.g., Fitzpatrick), and the legal tenets they draw from (e.g., equal opportunity). This implicitly Western take is troublingly becoming a universal ethical framework for ML, e.g., AI strategies from India NITI, Tunisia TunisiaAI, and Mexico MexicoAI all derive from this work, but fails to account for the several assumptions conventional algorithmic fairness makes about the availability and efficacy of the surrounding institutions and infrastructures. However, these infrastructures, values, and legal systems cannot be naively generalised to various non-Western countries.
Let us consider the example of facial recognition technology, where demonstration of fairness failures resulted in bans and moratoria in the US. Several factors led to this outcome: decades of empiricism on proxies and metrics that correspond to subgroups in the Westfitzpatrick1988validity; public datasets, APIs, and laws enabling analysis of model outcomes MachineB33:online; HowFacia39:online; an ML research/industry responsive to bias reports from users and civil society Weareimp72:online; buolamwini2018gender; the existence of government representatives glued into technology policy JayapalJ44:online; and an active media that scrutinizes downstream impacts of AI HowFacia39:online. However, due to various cultural, ethnic, and infrastructural differences, these factors are often absent or irrelevant in much else of the world. Could fairness have structurally different meanings or mechanisms in non-Western contexts? How do social, economic, and infrastructural factors influence implementation of meaningful fairness?
In this position paper, we present insights from qualitative interviews with 36 Indian scholars and activists, and a discourse analysis of emerging algorithmic deployments in India. India is home to 1.38 billion people and their multiple languages, religions, cultural systems, and ethnicities. AI deployments are prolific in the public sector, e.g., in predictive policing PredictiveIndia, facial recognition FacialRecIndia, and agriculture AgricIndia. Despite this forward momentum, there is a dire lack of conversations on advancing algorithmic fairness for such a large population. Our paper presents a first step towards filling this important gap. We contend that conventional Fair-ML approach may be inappropriate and inimical in India, if it does not engage with local structures. For a country as deeply plural, complex, and contradictory as India—where the distance between models and oppressed communities is large—optimising model fairness alone can be mere tokenism. We call for action along three critical pathways towards algorithmic fairness in India: Recontextualising, Empowering, and Enabling (see Figure 2). The considerations we present involve collective responsibility of inter-disciplinary Fair-ML researchers, and push the bounds of what is considered to be fairness research.
We perform a synthesis of (i) 36 interviews with researchers and activists across various disciplines working with marginalized Indian communities, and (ii) observations of current algorithmic deployments and policies in India. We chose our respondents from diverse disciplines: Computer Science (11), Activism (9), Law and Public Policy (6), Science and Technology Studies (5), Development Economics (2), Sociology (2), and Journalism (1). All respondents had 10-30 years of experience working with on social justice in India, on areas including caste, gender, labour, disability, surveillance, health, and constitutional rights. In conjunction with qualitative interviews, we analyzed various algorithmic deployments and emerging policies in India, starting from 2009. We identified and analysed various Indian news publications, policy documents, grassroots fora, and prior research on automation in India. The systematic process followed for the collection, categorization, and synthesis of data to develop the key insights below are described in detail in Appendix A.
3 Summary of Findings
Our study finds that the conventional fairness approach may be inappropriate, insufficient, or even inimical in India, if it does not engage with local structures. In India where the distance between ML models and dis-empowered communities whom they aim to serve is large—via technical distance, social distance, ethical distance, temporal distance, and physical distance—a myopic take on localising ‘fair’ model outputs alone can backfire. We identify three high-level themes/clusters of challenges (Figure 1) that engulf the large distance between models deployed in India and the communities they impact. We summarize them below.
Flawed data and model assumptions: We found numerous ways data and model assumptions fail in India, primarily: data distortions owing to infrastructural challenges and technology usage patterns (e.g., SIM card sharing among women sambasivan2018privacy) that makes datasets not faithful representations of people and phenomena; data incompleteness due to the models gearing for the ‘good’ data profiles that largely favor middle-class men—in a society where 50% do not have Internet, especially women and lower caste people; axes of discrimination and their proxies and indicators that differ from how they manifest in the West (for a detailed account, see Appendix 1); and lack of Indic justice approaches like reservations/quotas that present new challenges to fairness evaluations and interventions.
ML makers’ distance: Several respondents described a transactional mindset towards Indians, seeing them as agency-less data subjects that generated large-scale behavioural traces to improve ML models (also see Digitalc91:online), resulting in poor recourse and redress for Indian users, especially for those dis-empowered by caste, religion, or gender. The human infrastructures who played a crucial role in providing recourse to marginalised Indian communities (e.g., street-side bureaucrats, and front line workers) were removed in ML systems. Many respondents also pointed out the lack of diverse representation in Indian tech workforce that the distant ML makers often fail to recognize.
AI aspiration: AI is aspirationally viewed in India, with high reliance for high-stakes domains, accompanied with high trust in automation, limited transparency and the lack of an empowered ecosystem to question algorithmic interventions. For instance, respondents pointed to the lack of a buffer zone of journalists, activists, and researchers to keep ML system builders accountable. Lack of transparency and stakeholder participation, along with the its ‘neutral’ and ‘human-free’ associations lent misplaced credence to its algorithmic authority, making it further inscrutable.
4 Towards Algorithmic Fairness in India
To account for the challenges outlined above, we need to understand and design for end-to-end chains of algorithmic power, including how AI systems are conceived, produced, utilised, appropriated, assessed, and contested in India. To this end, we propose a research agenda where we call for action along three critical and contingent pathways towards successful AI fairness in India: Recontextualising, Empowering, and Enabling (see Figure 2). These pathways present new sociotechnical research challenges and require cross-disciplinary and cross-institutional collaborations.
4.1 Recontextualising data and models
Data plays a critical role in measurements and mitigation of algorithmic bias. We must be (even more than usual) sceptical of Indian datasets (due to challenges outlined above), and study how fairness research could handle the known distortions in the data as well as how we must account for data voids golebiewski2019data for which statistical extrapolations might be invalid. The vibrant role played by human infrastructures in providing, negotiating, collecting, and stewarding data points should encourage us to re-imagine data as a dialogue rather than operations, i.e, products of both the beholder and the beheld. How might data consent work in this context? One approach could be to create transitive informed consent, built upon personal relationships and trust in data workers, with transparency on potential downstream applications. Ideas like collective consent MozillaF26:online, data trusts Nesta, and data co-ops may enhance community agency in datasets, while simultaneously increasing data reliability. Finally, we must question the categories and constructs we model in datasets, and how we measure them. When categories are appropriate for endemic goals (e.g., caste membership for quotas), what form should their distributions and annotations take? Linguistically and culturally pluralistic communities should be given voices in these negotiations in ways that respect Indian norms of representation.
Model (un)fairness detection and mitigation should incorporate the prominent axes of historical injustices in India (see Appendix 1) and tackle the challenges in operationalising them for testing AI; e.g.,
representational biases of caste and other sub-groups in NLP models, biases in Indic language NLP including challenges from code-mixing, Indian subgroup biases in computer vision, tackling online misinformation, benchmarking using Indic datasets, and fair allocation models in public welfare. It is important to note that operationalising fairness approaches from the West to these axes is often nontrivial. For instance, personal names act as a signifier for various socio-demographic attributes in India, however there are no large datasets of Indian names (like the US Census data, or the SSA data) that are readily available for fairness evaluations. Another important consideration is how the algorithmic fairness interventions work with the existing infrastructures in India that surrounds decision making processes;e.g., in the context of restorative justice initiatives such as reservations/quotas? The quota system gives rise to the problem of matching under distributional constraints kamada2015efficient; goto2017designing; ashlagi2020assignment that has not received much attention within the ML research.
4.2 Empowering communities
Participatory Fair-ML knowledge systems
are critical to effectively fill the distance between ML makers and communities. Context-free assumptions in fairness research, whether in homegrown or international ML systems, can not just fail, but produce harm inadvertently when applied to different infrastructural and cultural contexts. Systems should be created that enable marginalised communities to participate in the production of knowledge systems about themselves in ML frameworks. Grassroots efforts like Deep Learning Indabadlindaba and Khipu khipu are helpful examples in bootstrapping AI research in communities. Initiatives like Design Beku designbeku and SEWA sewa are good decolonial examples of participatorily co-designing with under-served communities. Diverse values in algorithmic fairness can come from justice systems of indigenous Adivasis xaxa2011tribes, dharma (social ethic) and karma (action) from Hinduism (note that Hindu concepts are sometimes rejected by Dalits for reinforcing hierarchy narayanan2001water) or (sanghas) (communism) espoused by Dr. B.R. Ambedkar ambedkar2011buddha, for example.
Low-resource considerations pertaining to India should guide ML researchers to think beyond making model outputs fair, and about India’s heterogeneous literacies, economics, and infrastructures that interfere with access to those fair outputs. Half the population of India is not online. Layers of the stack like interfaces, devices, connectivity, languages, and costs are all important in ensuring sustainable access and usage. Tackling resource constraints in computing disciplines like ICTD toyama2015geek and HCI4D dray2012human can help—where constraints have been embraced as design material, e.g., delay-tolerant connectivity, low cost devices, text-free interfaces, partnership with civil society, and capacity building brewer2005case; heimerl2013local; kumar2015mobile; medhi2006text; sambasivan2019they. Data infrastructures to build, share and sustain localised datasets would also enhance access equity to holding AI accountable (e.g., lacuna).
Care in deployments should be a primary concern. Critiques were raised in our study on how neo-liberal ML followed a broader pattern of extraction from the ‘bottom billion’ data subjects and AI labourers. Low costs, large and diverse populations, and policy infirmities have been cited as reasons for following double standards in India, e.g., in non-consensual human trials and waste dumping macklin2004double (also see mohamed2020decolonial). Past disasters in India, like the fatal Union Carbide gas leak in 1984—one of the world’s worst industrial accidents—point to faulty design and low quality standards for the ‘third world’ Courttol47:online. Similarly, overgrowth of ML capital, with unequal standards, inadequate safeguards, and dubious applications can lead to catastrophic effects (similar analogies have been made for content moderation roberts2016digital; sambasivan2019they). ML researchers should study how to ensure meaningful recourse within the ecosystems they are embedding, and respond and engage with Indian realities and user feedback.
4.3 Enabling fair ML ecosystems
Ecosystems for accountability requires enabling the civil society, media, industry, judiciary, and the state to meaningfully engage in holding AI interventions responsible. Moving from ivory tower research approaches to solidarity with various stakeholders through project partnerships, involvement in evidence-based policy, and policy maker education can help create a sustainable Fair-ML ecosystem based on sound empirical and ethical basis. Investigative journalism on algorithms is lacking in India, but sustainable partnerships may be created with rigorous media sources. Algorithmic fairness and ethics are not mainstream research topics in Indian academia today, but the academy has a pivotal role in advancing fairness in India. A concerted effort is required to create APIs, documentation, and socio-economic datasets to enable meaningful and equitable accountability in the ecosystem.
Radical transparency is required to counteract the inscrutability and authority created by the enthusiastic AI aspiration. Besides the role played by ecosystem-wide regulations and standards, radical transparency as a design principle should be espoused and enacted by Fair-ML researchers committed to India. Transparency on datasets, processes, and models (e.g., mitchell2019model; gebru2018datasheets; bender2018data), openly discussing limitations, failure cases, and intended use-cases can help bolster the practical limits to applying computing to human problems. It is also incumbent upon those in power to discuss current Fair-ML interventions as a flawed and evolving approach. Such approaches can help move from the ‘magic pill’ role of fairness as a checklist for ethical issues in India, to a more pragmatic function.
We are currently facing a watershed moment for AI, with extravagant hype and pervasive applications, with nations being turned into AI superpowers and rulersWhoeverl94:online. We bring to light how efforts in the West to make AI fairer may not transfer readily elsewhere, specifically in the case of India, through an empirical bottom-up analysis. We present a roadmap to go beyond localising data and model fairness alone, and present a holistic research agenda to operationalize Fair-ML in India. While it is important to incorporate Indian concepts of fairness into ML that impacts Indian communities, the broader call is to develop general approaches to fairness that reflect local considerations.
Appendix A Methodology details
We conducted qualitative interviews with 36 expert researchers, activists, and lawyers working closely with marginalised Indian communities at the grassroots. Expert interviews are a qualitative research technique used in exploratory phases, providing practical insider knowledge, surrogacy for a broader community, and importantly bogner2009interviewing. Our respondents were chosen from a wide range of areas to create a holistic analysis of algorithmic power in India. Respondents came from Computer Science (11), Activism (9), Law and Public Policy (6), Science and Technology Studies (5), Development Economics (2), Sociology (2), and Journalism (1). All respondents had 10-30 years of experience working with marginalised communities or on social justice. Specific expertise areas included caste, gender, labour, disability, surveillance, privacy, health, constitutional rights, and financial inclusion. 24 respondents were based in India, 2 in Europe, 1 in Southeast Asia, the rest in the USA; 25 of them self-identified as male, 10 as female, and 1 as non-binary. In conjunction with qualitative interviews, we conducted an analysis of various algorithmic deployments and emerging policies in India, starting from 2009. We identified and analysed various Indian news publications (e.g., TheWire.in, Times of India), policy documents (e.g., NITI Aayog, Srikrishna Bill), grassroots fora (e.g., Dalit Camera, Dalitality), and prior research on automation in India.
We recruited respondents via a combination of reaching out directly and personal contacts, using purposeful sampling palinkas2015purposeful—i.e., identifying and selecting experts with relevant experience—iterative until saturation. We conducted all interviews in English (preferred language of participants). Respondents were compensated for the study (giftcards of 100 USD, 85 EUR, and 2000 INR). Interviews lasted an hour each, and were conducted using video conferencing. Transcripts were coded and analyzed for patterns using an inductive approach Thomas2006general. From a careful reading of the transcripts, we developed categories and clustered excerpts, conveying key themes from the data. Two team members created a code book based on the themes. The three power structures that we describe below were then developed and applied iteratively to the codes.
We took great care to create a research ethics protocol to protect respondent privacy and safety, especially due to the sensitive nature of our inquiry. During recruitment, participants were informed of the purpose of the study, the question categories, and researcher affiliations. Participants signed informed consent acknowledging their awareness of the study purpose and researcher affiliation prior to the interview. At the beginning of each interview, the moderator additionally obtained verbal consent. We stored all data in a private Google Drive folder, with access limited to our team. To protect participant identity, we deleted all personally identifiable information in research files. We redact identifiable details when quoting participants. Every respondent was given the choice of default anonymity or being included in acknowledgements. All co-authors of this paper work at the intersection of under-served communities and technology, with backgrounds in HCI, critical data studies, and algorithmic fairness; some of us have had grassroots commitments with marginalised Indian communities for over a decade. The first author constructed the research approach, and both the first and second author moderated interviews. All researchers were involved in the research framing, data analysis, and synthesis. Three of us are Indian and two of us are Caucasian. All of us come from privileged positions of class and/or caste.
Appendix B Axes of Discrimination in India
In Table 1, we describe some of the prominent axes of injustice in India that were brought up in our interviews, enriched through secondary research and statistics from authoritative sources, to substantiate attributes and proxies to substantiate attributes and proxies.
|Sub-groups, Proxies and Harms|
(17% Dalits; 8% Adivasi; 40% Other Backward Class (OBC))CensusIndia2011
(80% Hindu, 14% Muslim, 6% Christians, Sikhs, Buddhists, Jains and indigeneous) CensusIndia2011
(5%–8%+ persons with disabilities) WorldBank2009Disabilities
(30% live below poverty line; 48% on $2–$10/day)Rangarajan2014Poverty CensusIndia2011
Gender Identity & Sexual Orientation
(No official LGBTQ+ data)
(4% NorthEast) CensusIndia2011