[Paper Review] Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

Hannah Rose Kirk, Yennie Jun+6 more|arXiv (Cornell University)|Feb 8, 2021

Computational and Text Analysis Methods53 citations

TL;DR

The paper analyzes out-of-the-box GPT-2 biases in occupational associations across gender intersected with religion, sexuality, ethnicity, political affiliation, and continent name origin, comparing predictions to US labor market data.

ABSTRACT

The capabilities of natural language models trained on large-scale data have increased immensely over the past few years. Open source libraries such as HuggingFace have made these models easily available and accessible. While prior research has identified biases in large language models, this paper considers biases contained in the most popular versions of these models when applied `out-of-the-box' for downstream tasks. We focus on generative language models as they are well-suited for extracting biases inherited from training data. Specifically, we conduct an in-depth analysis of GPT-2, which is the most downloaded text generation model on HuggingFace, with over half a million downloads per month. We assess biases related to occupational associations for different protected categories by intersecting gender with religion, sexuality, ethnicity, political affiliation, and continental name origin. Using a template-based data collection pipeline, we collect 396K sentence completions made by GPT-2 and find: (i) The machine-predicted jobs are less diverse and more stereotypical for women than for men, especially for intersections; (ii) Intersectional interactions are highly relevant for occupational associations, which we quantify by fitting 262 logistic models; (iii) For most occupations, GPT-2 reflects the skewed gender and ethnicity distribution found in US Labor Bureau data, and even pulls the societally-skewed distribution towards gender parity in cases where its predictions deviate from real labor market observations. This raises the normative question of what language models should learn - whether they should reflect or correct for existing inequalities.

Motivation & Objective

• Motivate the study of biases in popular open-source language models used out-of-the-box.
• Develop a protocol to measure intersectional occupational biases in GPT-2 across multiple protected attributes.
• Quantify how intersectionality influences occupation predictions via statistical modeling.
• Compare GPT-2 predicted occupation distributions with ground-truth US labor market data to assess alignment or divergence.

Proposed method

• Use a template-based data collection pipeline to extract 396K GPT-2 sentence completions with occupation mentions.
• Apply identity-based and name-based prefix prompts to elicit occupational associations related to protected attributes.
• Extract occupations with Stanford CoreNLP NER and construct a one-hot token frequency matrix for analysis.
• Fit 262 logistic regression models with interaction terms to quantify intersectional effects on job predictions.
• Compare GPT-2 predicted distributions to 2019 US Bureau of Labor Statistics occupation data, adjusting for demographic distributions.

Experimental results

Research questions

• RQ1 Do out-of-the-box GPT-2 outputs exhibit gender and intersectional occupational biases?
• RQ2 How do intersections of gender with ethnicity, religion, sexuality, political affiliation, and continent name origin affect predicted occupations?
• RQ3 To what extent do GPT-2 predictions align with real-world US occupational distributions by gender and race/ethnicity?
• RQ4 Are intersectional interactions significant in changing GPT-2’s job predictions beyond main effects?
• RQ5 Do model predictions indicate representational or allocational harms in occupational associations?

Key findings

• GPT-2 returns a less diverse and more stereotypical set of jobs for women than for men, with higher occupational clustering for women.
• Intersectional interactions (e.g., gender with ethnicity, religion, sexuality) significantly shape occupation predictions, with many models showing significant interaction terms.
• For most occupations, GPT-2 predictions mirror the skew in US labor data, and in some cases pull towards gender parity, potentially correcting extreme real-world skew.
• Logistic regressions show the woman dummy adds explainable variation in many models (average ΔR^2 ≈ +3.3%), and many interactions are significant (≈ one-third of regressions).
• Comparisons with US data yield strong positive associations (Kendall Tau ≈ 0.63 for baseline gender) and low mean-squared errors, though GPT-2 often overestimates women in female-dominated jobs and underestimates in highly female-skewed ones.
• GPT-2 tends to over-represent top-5 occupations for women (e.g., waitress, nurse) relative to US data, while still reflecting some real-world distributions for ethnicity.