[Paper Review] Attesting Biases and Discrimination using Language Semantics

Xavier Ferrer Aran, José M. Such+1 more|arXiv (Cornell University)|Sep 10, 2019

Ethics and Social Impacts of AI4 citations

TL;DR

This paper proposes a framework to detect and attest digital discrimination in AI agents by analyzing language semantics, focusing on how human unconscious biases embedded in language are inherited by NLP systems. It advocates for using linguistic corpora—both human- and AI-generated—to identify discriminatory patterns, with applications in auditing black-box AI systems and enabling self-awareness in agents to mitigate bias.

ABSTRACT

AI agents are increasingly deployed and used to make automated decisions that affect our lives on a daily basis. It is imperative to ensure that these systems embed ethical principles and respect human values. We focus on how we can attest to whether AI agents treat users fairly without discriminating against particular individuals or groups through biases in language. In particular, we discuss human unconscious biases, how they are embedded in language, and how AI systems inherit those biases by learning from and processing human language. Then, we outline a roadmap for future research to better understand and attest problematic AI biases derived from language.

Motivation & Objective

• To address the growing problem of digital discrimination in AI systems that make automated decisions affecting individuals and groups.
• To investigate how unconscious human biases are encoded in language and inherited by NLP systems through training data.
• To develop methods for detecting problematic biases in language corpora used by AI agents, especially in black-box systems.
• To explore whether AI-generated text can reveal traces of training data biases, enabling indirect auditing of opaque models.
• To investigate the potential for AI agents to recognize and address their own learned biases through semantic analysis.

Proposed method

• Utilize linguistic corpora—both human-authored and AI-generated—to analyze semantic associations indicative of bias.
• Apply techniques such as word embedding analysis to detect implicit associations (e.g., gender or racial stereotypes) in language models.
• Leverage the Implicit Association Test (IAT) as a conceptual model to detect strength of associations between social categories and attributes in text.
• Analyze AI-generated text (e.g., chat logs from personal assistants) to infer biases present in training data, even without direct access to the data.
• Study the evolution of bias in reinforcement learning systems through interaction with human users, using real-world examples like Microsoft’s Tay chatbot.
• Integrate data-driven NLP methods with knowledge-based normative systems to enable AI agents to recognize and respond to their own biases.

Experimental results

Research questions

• RQ1 How are unconscious human biases encoded in language and transmitted to NLP systems?
• RQ2 To what extent can AI-generated text corpora reflect the biases present in the training data of black-box AI systems?
• RQ3 Can semantic analysis of language used by AI agents reveal hidden discriminatory patterns or social norms?
• RQ4 How do reinforcement learning agents internalize and propagate societal biases through human interactions?
• RQ5 What mechanisms can enable AI agents to self-attest and correct for their own learned biases?

Key findings

• Word embedding models exhibit strong gender stereotypes, such as associating 'leader' with male names and 'helper' with female names, confirming that NLP systems inherit societal biases.
• Search engines have been observed to suggest more arrest records for black-sounding names than for white-sounding names, indicating that language models can encode and amplify racial biases.
• AI-generated text, such as chat logs from personal assistants, can reflect underlying biases in training data, even when the data source is unknown.
• Reinforcement learning systems like Microsoft’s Tay can rapidly learn and propagate offensive or discriminatory language from user interactions, demonstrating the risk of bias amplification.
• There is strong potential for using semantic analysis of AI-generated language to indirectly audit black-box systems and infer training data characteristics.
• Combining data-driven NLP with normative knowledge systems may enable AI agents to recognize and mitigate their own biases, paving the way for self-aware, ethical AI.