[Paper Review] Fairness in Language Models Beyond English: Gaps and Challenges

Krithika Ramesh, Sunayana Sitaram+1 more|arXiv (Cornell University)|Feb 24, 2023

Social Policy and Reform Studies4 citations

TL;DR

This paper surveys fairness challenges in multilingual and non-English language models, highlighting the lack of inclusive bias evaluation and mitigation methods beyond English. It argues that current dataset-driven approaches fail to scale across diverse languages and cultures, calling for intersectional, interdisciplinary, and culturally grounded fairness frameworks in multilingual NLP.

ABSTRACT

With language models becoming increasingly ubiquitous, it has become essential to address their inequitable treatment of diverse demographic groups and factors. Most research on evaluating and mitigating fairness harms has been concentrated on English, while multilingual models and non-English languages have received comparatively little attention. This paper presents a survey of fairness in multilingual and non-English contexts, highlighting the shortcomings of current research and the difficulties faced by methods designed for English. We contend that the multitude of diverse cultures and languages across the world makes it infeasible to achieve comprehensive coverage in terms of constructing fairness datasets. Thus, the measurement and mitigation of biases must evolve beyond the current dataset-driven practices that are narrowly focused on specific dimensions and types of biases and, therefore, impossible to scale across languages and cultures.

Motivation & Objective

• To identify and analyze the lack of fairness research in non-English and multilingual NLP contexts compared to the dominant Anglo-centric focus.
• To examine how linguistic and cultural diversity across languages undermines the scalability and validity of existing bias measurement and mitigation techniques.
• To highlight the limitations of current fairness metrics and datasets in capturing intersectional identities and sociocultural nuances in non-English languages.
• To advocate for interdisciplinary, multicultural teams and culturally aware evaluation benchmarks to improve fairness in multilingual language models.
• To call for a shift from dataset-driven, narrow bias evaluation toward holistic, context-sensitive fairness assessment that accounts for linguistic variation and power structures.

Proposed method

• Conducting a comprehensive survey of fairness literature in multilingual and non-English NLP, focusing on bias measurement and mitigation strategies.
• Classifying bias types into representational and allocational harms, with emphasis on how they manifest differently across languages and cultures.
• Analyzing existing fairness metrics—such as WEAT and SEAT—while critiquing their limitations in capturing intersectional and cultural dimensions of bias.
• Evaluating the risks of zero-shot translation and direct dataset translation in fairness-critical applications due to cultural and linguistic mismatches.
• Proposing the need for shared value systems and culturally aware model design that reflect diverse sociocultural contexts, especially in low-resource and grammatically gendered languages.
• Recommending the use of intersectional fairness metrics and human-in-the-loop evaluation to ensure validity and reliability of bias assessments.

Experimental results

Research questions

• RQ1 How do representational and allocational harms in language models differ across non-English and multilingual contexts compared to English-centric systems?
• RQ2 Why are current fairness metrics like WEAT and SEAT insufficient for capturing bias in multilingual and culturally diverse settings?
• RQ3 What are the key challenges in creating scalable, culturally aware fairness evaluation benchmarks for low-resource and grammatically gendered languages?
• RQ4 How do linguistic variations such as dialects, phonology, and syntactic structures contribute to bias in multilingual language models?
• RQ5 What role can interdisciplinary and multicultural teams play in identifying and mitigating bias in multilingual NLP systems?

Key findings

• Most fairness research in NLP remains Anglo-centric, with minimal attention to non-English and multilingual contexts, despite the global diversity of language users.
• Existing fairness metrics such as WEAT and SEAT are vulnerable to manipulation and fail to capture intersectional identities or cultural nuances in bias.
• Zero-shot translation and direct dataset translation for fairness evaluation often fail to preserve cultural context, leading to misleading or ineffective bias mitigation.
• Bias in training data is deeply entrenched, with models reflecting demographic imbalances and underrepresentation of marginalized communities in non-English corpora.
• The lack of diverse, high-quality fairness benchmarks for non-English languages severely limits the scalability and generalization of fairness interventions.
• Current approaches to fairness are insufficient for addressing allocational harms in multilingual systems, especially in low-resource and socioculturally complex settings.