[Paper Review] Do Generative AI Models Output Harm while Representing Non-Western Cultures: Evidence from A Community-Centered Approach

Sourojit Ghosh, Pranav Narayanan Venkit+3 more|arXiv (Cornell University)|Jul 20, 2024

Psychology of Moral and Emotional Judgment4 citations

TL;DR

This study investigates how text-to-image generative AI models misrepresent non-Western cultures, focusing on Indian subcultures through a community-centered approach. Using grounded theory analysis of 5 focus groups, it identifies novel representational harms—exoticism and cultural misappropriation—highlighting systemic biases in AI outputs and proposing sociotechnical design guidelines to promote cultural accuracy and inclusivity in global AI development.

ABSTRACT

Our research investigates the impact of Generative Artificial Intelligence (GAI) models, specifically text-to-image generators (T2Is), on the representation of non-Western cultures, with a focus on Indian contexts. Despite the transformative potential of T2Is in content creation, concerns have arisen regarding biases that may lead to misrepresentations and marginalizations. Through a community-centered approach and grounded theory analysis of 5 focus groups from diverse Indian subcultures, we explore how T2I outputs to English prompts depict Indian culture and its subcultures, uncovering novel representational harms such as exoticism and cultural misappropriation. These findings highlight the urgent need for inclusive and culturally sensitive T2I systems. We propose design guidelines informed by a sociotechnical perspective, aiming to address these issues and contribute to the development of more equitable and representative GAI technologies globally. Our work also underscores the necessity of adopting a community-centered approach to comprehend the sociotechnical dynamics of these models, complementing existing work in this space while identifying and addressing the potential negative repercussions and harms that may arise when these models are deployed on a global scale.

Motivation & Objective

• To investigate how text-to-image (T2I) models represent Indian cultures and their subcultures, especially in non-Western contexts.
• To uncover novel forms of representational harm in AI-generated content that marginalize or misrepresent diverse Indian communities.
• To develop culturally grounded, community-informed design principles for more equitable and accurate T2I systems.
• To challenge the dominance of Western-centric frameworks in AI ethics by centering marginalized voices from the Global South.
• To advocate for a sociotechnical approach to AI design that integrates cultural knowledge and community agency in model development.

Proposed method

• Conducted 5 community-based focus groups with 25 participants from diverse Indian subcultures and regions to examine T2I outputs in response to culturally specific prompts.
• Applied grounded theory analysis (Charmaz, 2017) to systematically identify and categorize patterns of representational harm in AI-generated images.
• Used live-generated outputs from the Stable Diffusion model in response to participant-chosen English prompts to analyze visual representations.
• Defined and operationalized two novel forms of representational harm: exoticism and cultural misappropriation, based on thematic coding of visual and narrative content.
• Developed design guidelines through a sociotechnical lens, integrating insights from community participants and cultural experts.
• Ensured transparency and reproducibility by publicly sharing research materials via a GitHub repository.

Experimental results

Research questions

• RQ1 How do T2I outputs represent Indian culture and its subcultures, and what are the implications of these representations for diverse cultural groups?
• RQ2 What novel forms of representational harms emerge from the interactions between T2I systems and Indian cultural contexts?
• RQ3 How can community-centered design principles mitigate these harms and improve cultural accuracy in generative AI systems?
• RQ4 Why do attempts to prompt for non-exotic, accurate representations fail to produce culturally appropriate outputs?
• RQ5 What role do surface-level cultural cues play in perpetuating misrepresentation despite user intent?

Key findings

• T2I models consistently produce exoticized portrayals of Indian culture, characterized by overrepresentation of rural settings, traditional attire like sarees on all female-presenting figures, and overly colorful depictions of markets or daily life.
• Exoticism persists even when users attempt to refine prompts to avoid stereotypical imagery, indicating systemic bias in model training data and generation logic.
• Cultural misappropriation is prevalent, including incorrect fusion of regional breakfast items, misrepresentation of regional clothing draping styles, and inaccurate use of traditional dance ornaments.
• Participants reported that such misrepresentations lead to cultural erasure and marginalization, especially for less visible subcultures and regional identities.
• The study identifies a lack of nuanced cultural understanding in T2I models, which treat cultural features as isolated, stereotypical signifiers rather than contextually embedded practices.
• Despite user efforts to generate accurate representations, the models fail to deliver culturally appropriate outputs, underscoring the need for community-informed data and design.