[Paper Review] "Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Models

Abel Salinas, Louis Penafiel+2 more|arXiv (Cornell University)|Oct 13, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes a prompt-based, knowledge graph–driven method to automatically discover and visualize hidden stereotypes in the internal knowledge of large language models (LLMs), such as GPT-3. By dynamically generating structured representations of bias-related associations, the approach identifies systematic disparities in how LLMs associate nationalities and social groups with warmth and competence, revealing entrenched societal biases in model knowledge without relying on static templates.

ABSTRACT

Large language models (LLMs) have garnered significant attention for their remarkable performance in a continuously expanding set of natural language processing tasks. However, these models have been shown to harbor inherent societal biases, or stereotypes, which can adversely affect their performance in their many downstream applications. In this paper, we introduce a novel, purely prompt-based approach to uncover hidden stereotypes within any arbitrary LLM. Our approach dynamically generates a knowledge representation of internal stereotypes, enabling the identification of biases encoded within the LLM's internal knowledge. By illuminating the biases present in LLMs and offering a systematic methodology for their analysis, our work contributes to advancing transparency and promoting fairness in natural language processing systems.

Motivation & Objective

• To address the lack of flexible, scalable methods for detecting biases in LLMs that are not dependent on hand-crafted templates.
• To map the stereotypical knowledge embedded in LLMs by extracting and structuring associations between social groups and evaluative traits like warmth and competence.
• To provide a systematic, automated framework for identifying and visualizing biases in the internal knowledge base of any LLM, regardless of group type (e.g., race, gender, nationality).
• To enable practitioners to assess potential biases before deployment, enhancing transparency and fairness in NLP systems.
• To lay the groundwork for future analysis of how internal stereotypes may propagate into downstream model behavior.

Proposed method

• Employing a prompt-based approach inspired by Cohen et al. (2023), the method dynamically queries the LLM to extract associations between social groups and evaluative traits.
• Using a seed set of demographic groups (e.g., nationalities, ethnicities), the method generates prompts that elicit stereotypical associations, such as 'People from X are often seen as...' or 'X is commonly associated with...'
• The model’s responses are parsed and structured into a knowledge graph, where nodes represent groups and traits, and edges represent the strength or likelihood of association.
• The knowledge graph is then analyzed to identify patterns in how different groups are stereotypically represented across dimensions like warmth and competence.
• The method is applied to GPT-3, enabling visualization and quantitative comparison of bias distributions across nationalities and social categories.
• External models are used to assess the semantic plausibility of extracted associations, though this introduces potential bias from the auxiliary models.

(a) Ethnicity

Experimental results

Research questions

• RQ1 What types of stereotypes and preconceived notions are encoded in the internal knowledge of large language models like GPT-3?
• RQ2 How do associations between social groups and traits like warmth and competence vary across different nationalities and demographic categories?
• RQ3 To what extent can a prompt-based, knowledge graph–generation approach uncover biases without relying on static, hand-crafted templates?
• RQ4 How do the distributions of stereotype-related knowledge differ between competence-related and warmth-related traits across groups?
• RQ5 Can this method detect and visualize systematic disparities in how marginalized or historically stigmatized groups are represented in LLM knowledge?

Key findings

• The method successfully generated knowledge graphs revealing that nationalities such as 'Russian people' are disproportionately associated with competence but least associated with warmth, indicating a potential competence-over-competence bias.
• A clear pattern emerged where nationalities were more frequently linked to competence-related traits than warmth-related ones, suggesting a systematic imbalance in stereotypical representation.
• The model exhibited strong associations between certain nationalities and negative traits—e.g., 'Chinese people' were frequently linked to 'stereotypical' or 'negative' descriptors in the knowledge graph, reflecting historical biases in training data.
• The knowledge graph revealed that while most countries were associated with both warmth and competence, the strength of these associations varied significantly, with some groups receiving notably higher or lower scores.
• The approach identified that the LLM’s internal knowledge reflects societal biases present in training data, such as the overrepresentation of certain nationalities in high-competence roles and underrepresentation in warmth-related roles.
• The study confirmed that awareness of stereotypes in the model’s knowledge base does not necessarily imply their use in generation, but the presence of such knowledge poses a risk for downstream bias propagation.

(b) Gender Identity