[Paper Review] Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages

Simran Khanuja, Sebastian Ruder+1 more|arXiv (Cornell University)|May 25, 2022

Speech and dialogue systems4 citations

TL;DR

This paper proposes a holistic evaluation framework for NLP technology based on diversity, equity, and inclusion, using the Gini coefficient to quantify linguistic equity. It applies the framework to Indian languages, revealing significant inequities in model performance, and introduces a novel computational method for optimal annotation budget allocation during fine-tuning that outperforms uniform or single-source strategies.

ABSTRACT

In order for NLP technology to be widely applicable, fair, and useful, it needs to serve a diverse set of speakers across the world's languages, be equitable, i.e., not unduly biased towards any particular language, and be inclusive of all users, particularly in low-resource settings where compute constraints are common. In this paper, we propose an evaluation paradigm that assesses NLP technologies across all three dimensions. While diversity and inclusion have received attention in recent literature, equity is currently unexplored. We propose to address this gap using the Gini coefficient, a well-established metric used for estimating societal wealth inequality. Using our paradigm, we highlight the distressed state of current technologies for Indian (IN) languages (a linguistically large and diverse set, with a varied speaker population), across all three dimensions. To improve upon these metrics, we demonstrate the importance of region-specific choices in model building and dataset creation, and more importantly, propose a novel, generalisable approach to optimal resource allocation during fine-tuning. Finally, we discuss steps to mitigate these biases and encourage the community to employ multi-faceted evaluation when building linguistically diverse and equitable technologies.

Motivation & Objective

• To address the lack of multi-dimensional evaluation in NLP by introducing a framework that assesses diversity, equity, and inclusion at the language level.
• To quantify linguistic equity in NLP models using the Gini coefficient, a metric traditionally used for income inequality.
• To identify and mitigate biases in NLP technology for low-resource Indian languages, which are linguistically diverse and underrepresented.
• To propose a data-efficient, algorithmic approach for optimal annotation budget allocation during fine-tuning in multilingual settings.
• To advocate for community-driven, grass-roots data collection and representative evaluation sets to improve model inclusivity.

Proposed method

• The authors define a language-agnostic evaluation paradigm that measures diversity via speaker population-weighted task performance, equity via the Gini coefficient of model performance across languages, and inclusion via model efficiency and deployability.
• They apply the framework to evaluate state-of-the-art multilingual models on four NLP tasks—NER, POS, NLI, and QA—across 23 Indian languages.
• Region-specific pre-training and Hindi as a transfer language during fine-tuning are evaluated as key modeling choices to improve performance on under-resourced languages.
• A novel computational method is proposed to model the source-target language space and derive optimal annotation budget allocation across multiple source languages to maximize performance on the proposed metrics.
• The method uses a greedy algorithm to allocate labeling resources based on predicted performance gains, outperforming uniform or single-source selection strategies.
• The study emphasizes the importance of low-resource, community-involving data collection and the need for representative evaluation sets to track progress.

Experimental results

Research questions

• RQ1 To what extent are current NLP models equitable across Indian languages, and how can linguistic inequity be quantified?
• RQ2 How do region-specific pre-training and transfer language choices affect model performance on low-resource Indian languages?
• RQ3 What is the optimal allocation of a fixed annotation budget across multiple source languages to maximize downstream performance and inclusivity?
• RQ4 How does model efficiency and runtime impact the practical inclusion of NLP technology in low-compute environments?
• RQ5 What role can community-driven data collection play in improving linguistic diversity and representativeness in evaluation sets?

Key findings

• The Gini coefficient reveals significant performance inequity across Indian languages, with high-performing languages receiving disproportionate model capacity.
• Region-specific pre-trained models and Hindi as the transfer language during fine-tuning yield the best performance across most Indian languages.
• Smaller models are more efficient for syntactic tasks like POS tagging, while larger models perform better on semantic tasks like NLI and QA.
• The proposed computational annotation allocation method outperforms both uniform and single-source data selection strategies in improving model performance across diverse languages.
• Many officially recognized Indian languages lack evaluation data, highlighting a critical gap in benchmarking and progress tracking.
• Even with optimal modeling choices, current NLP systems remain far from achieving equitable and inclusive deployment for all Indian language speakers.