[Paper Review] Generative Language Models Exhibit Social Identity Biases

Tiancheng Hu, Yara Kyrychenko+4 more|arXiv (Cornell University)|Oct 24, 2023

Topic Modeling4 citations

TL;DR

This study demonstrates that 51 large language models (LLMs) exhibit social identity biases, including ingroup solidarity and outgroup hostility, when prompted to complete sentences like "We are...". The models show bias levels comparable to or exceeding human-written internet text, and fine-tuning on partisan Twitter data amplifies these biases—while removing biased training data significantly reduces them, indicating that curation of training data can mitigate such biases.

ABSTRACT

The surge in popularity of large language models has given rise to concerns about biases that these models could learn from humans. We investigate whether ingroup solidarity and outgroup hostility, fundamental social identity biases known from social psychology, are present in 56 large language models. We find that almost all foundational language models and some instruction fine-tuned models exhibit clear ingroup-positive and outgroup-negative associations when prompted to complete sentences (e.g., "We are..."). Our findings suggest that modern language models exhibit fundamental social identity biases to a similar degree as humans, both in the lab and in real-world conversations with LLMs, and that curating training data and instruction fine-tuning can mitigate such biases. Our results have practical implications for creating less biased large-language models and further underscore the need for more research into user interactions with LLMs to prevent potential bias reinforcement in humans.

Motivation & Objective

• To investigate whether foundational and instruction-tuned LLMs exhibit social identity biases such as ingroup solidarity and outgroup hostility.
• To compare the levels of these biases in LLM-generated text against human-written text from the internet.
• To examine how fine-tuning on partisan training data influences the emergence of social identity biases in LLMs.
• To evaluate whether removing biased sentences from training data reduces social identity biases in LLMs.
• To provide evidence that LLMs can learn and amplify social biases present in their training data, with implications for AI fairness and human-AI interaction.

Proposed method

• The study uses a prompt-based approach, asking 51 LLMs to complete sentences starting with "We are..." and "They are..." to elicit ingroup and outgroup expressions.
• Sentiment analysis is applied using VADER and RoBERTa to classify generated sentences as positive, negative, or neutral, with sentiment polarity used as a proxy for bias.
• Logistic regression models estimate the odds of positive or negative sentiment for ingroup vs. outgroup sentences, controlling for sentence length and type-to-token ratio.
• Mixed-effects logistic regression models are used to estimate overall ingroup solidarity and outgroup hostility across models, with model name as a random intercept.
• For fine-tuning, models are trained on a corpus of US partisan Twitter data, with varying proportions of ingroup-positive and outgroup-negative sentences removed to assess impact on bias.
• Data curation involves filtering sentences containing "We" or "They" using LIWC 2022, classifying sentiment with VADER, and removing sentences above/below sentiment thresholds of ±0.05.

Experimental results

Research questions

• RQ1 Do large language models exhibit social identity biases such as ingroup solidarity and outgroup hostility?
• RQ2 How do the levels of ingroup solidarity and outgroup hostility in LLM-generated text compare to those in human-written internet text?
• RQ3 Does fine-tuning LLMs on partisan Twitter data amplify social identity biases?
• RQ4 Can reducing or removing biased sentences from the training data significantly decrease ingroup solidarity and outgroup hostility in LLMs?
• RQ5 To what extent is the social identity bias in LLMs shaped by the composition of their training data?

Key findings

• All but three of the 51 tested LLMs exhibit measurable levels of ingroup solidarity and outgroup hostility when prompted with "We are..." and "They are...".
• The level of ingroup solidarity and outgroup hostility in LLM-generated text is comparable to, and in some cases exceeds, that found in human-written internet text from sources like Wikipedia, Reddit, and news sites.
• Fine-tuning LLMs on a corpus of US partisan Twitter data results in a marked increase in both ingroup solidarity and outgroup hostility, demonstrating that models can learn and internalize biases from their training data.
• Removing either ingroup-positive or outgroup-negative sentences (or both) from the fine-tuning data leads to a significant reduction in both ingroup solidarity and outgroup hostility, indicating that bias mitigation is achievable through data curation.
• The odds ratio for ingroup solidarity increases significantly when models are fine-tuned on data rich in ingroup-positive content, and the odds ratio for outgroup hostility increases even more strongly when outgroup-negative content is present.
• The study confirms that social identity biases in LLMs are not inherent but are learned from training data, and thus can be reduced by selective data filtering.