[Paper Review] Building Socio-culturally Inclusive Stereotype Resources with Community Engagement

Sunipa Dev, Jaya Goyal+3 more|arXiv (Cornell University)|Jul 20, 2023

Hate Speech and Cyberbullying DetectionComputer Science3 citations

TL;DR

This paper proposes SPICE, a socio-culturally inclusive stereotype resource for India built through community engagement, using open-ended surveys with diverse participants to collect over 2,000 context-specific stereotypes across identity axes like caste, region, and religion. The method significantly expands representation beyond Western-centric datasets, enabling more accurate detection of stereotypical biases in English language models.

ABSTRACT

With rapid development and deployment of generative language models in global settings, there is an urgent need to also scale our measurements of harm, not just in the number and types of harms covered, but also how well they account for local cultural contexts, including marginalized identities and the social biases experienced by them. Current evaluation paradigms are limited in their abilities to address this, as they are not representative of diverse, locally situated but global, socio-cultural perspectives. It is imperative that our evaluation resources are enhanced and calibrated by including people and experiences from different cultures and societies worldwide, in order to prevent gross underestimations or skews in measurements of harm. In this work, we demonstrate a socio-culturally aware expansion of evaluation resources in the Indian societal context, specifically for the harm of stereotyping. We devise a community engaged effort to build a resource which contains stereotypes for axes of disparity that are uniquely present in India. The resultant resource increases the number of stereotypes known for and in the Indian context by over 1000 stereotypes across many unique identities. We also demonstrate the utility and effectiveness of such expanded resources for evaluations of language models. CONTENT WARNING: This paper contains examples of stereotypes that may be offensive.

Motivation & Objective

• To address the lack of socio-culturally representative stereotype evaluation resources in global AI safety research.
• To overcome the Western bias and researcher-centric worldview in existing stereotype datasets.
• To include marginalized identities and locally situated perspectives from Indian society in stereotype resource creation.
• To demonstrate the utility of community-engaged data collection for improving harm evaluation in generative language models.
• To create a scalable, inclusive methodology applicable to other cultural contexts beyond India.

Proposed method

• Conducting open-ended, community-driven surveys with a diverse cross-section of Indian participants to collect free-form stereotype descriptions.
• Focusing data collection on identity axes unique to India, including caste, regional origin, religion, and gender.
• Pooling and curating over 2,000 stereotypes from participant responses, with attention to offensive and stigmatizing attributes.
• Using the resulting SPICE dataset to evaluate stereotypical associations in English language models.
• Applying a mixed-methods approach combining community insights with model evaluation to validate resource utility.
• Limiting data collection to English-language expressions due to linguistic and logistical constraints, acknowledging the need for multilingual expansion.

Experimental results

Research questions

• RQ1 How can stereotype evaluation resources be expanded to include underrepresented socio-cultural perspectives from the Global South?
• RQ2 To what extent do existing stereotype datasets fail to represent locally salient biases in non-Western contexts like India?
• RQ3 Can community-engaged, open-ended data collection yield more comprehensive and representative stereotype resources than researcher-driven or LLM-generated methods?
• RQ4 How effective is the SPICE dataset in detecting stereotypical inferences made by large language models in English?
• RQ5 What are the limitations of community-based data collection in capturing the full spectrum of societal stereotypes?

Key findings

• The SPICE dataset contains over 2,000 stereotypes specific to Indian socio-cultural contexts, significantly expanding existing resources.
• The dataset captures previously underrepresented or missing stereotypes related to caste (e.g., 'Bhangi', 'untouchable'), regional identity (e.g., 'Haryanvi', 'Bihari'), and religion (e.g., 'Muslim', 'Jat').
• Many offensive attributes such as 'gangster', 'criminal', and 'poor' were linked to specific regional or caste identities, reflecting systemic marginalization.
• The dataset revealed that stereotypes involving dark skin color (e.g., 'kaalu') and racial slurs (e.g., 'chink') are prevalent and contextually charged in Indian discourse.
• Language models were found to reproduce or reflect many of these stereotypes, confirming the dataset’s utility in evaluating model behavior.
• Despite its breadth, the dataset remains incomplete due to sampling limitations and the subjective nature of stereotype perception.