[Paper Review] AI-generated faces influence gender stereotypes and racial homogenization

Nouar AlDahoul, Talal Rahwan+1 more|arXiv (Cornell University)|Feb 1, 2024

Ethics and Social Impacts of AI4 citations

TL;DR

This paper proposes SDXL-Inc and SDXL-Div, two novel debiasing methods for Stable Diffusion XL that reduce racial and gender stereotypes in generated faces. Using a state-of-the-art face classifier, the authors quantify biases across six races, two genders, five age groups, 32 professions, and eight attributes, demonstrating that their methods significantly improve representation fairness and facial diversity compared to prior approaches, with results open-sourced for public use.

ABSTRACT

Text-to-image generative AI models such as Stable Diffusion are used daily by millions worldwide. However, the extent to which these models exhibit racial and gender stereotypes is not yet fully understood. Here, we document significant biases in Stable Diffusion across six races, two genders, 32 professions, and eight attributes. Additionally, we examine the degree to which Stable Diffusion depicts individuals of the same race as being similar to one another. This analysis reveals significant racial homogenization, e.g., depicting nearly all Middle Eastern men as bearded, brown-skinned, and wearing traditional attire. We then propose debiasing solutions that allow users to specify the desired distributions of race and gender when generating images while minimizing racial homogenization. Finally, using a preregistered survey experiment, we find evidence that being presented with inclusive AI-generated faces reduces people's racial and gender biases, while being presented with non-inclusive ones increases such biases, regardless of whether the images are labeled as AI-generated. Taken together, our findings emphasize the need to address biases and stereotypes in text-to-image models.

Motivation & Objective

• To quantify racial and gender stereotypes in Stable Diffusion XL-generated faces across diverse demographic categories.
• To develop and evaluate novel debiasing solutions that outperform state-of-the-art methods in promoting equitable representation.
• To analyze the degree of facial homogenization across racial and gender groups in AI-generated images.
• To propose open-source, deployable solutions that enhance diversity in text-to-image generation.
• To address limitations of prompt-based debiasing, including linguistic ambiguity and opacity, through automated, GPT-4-augmented prompt regulation.

Proposed method

• A three-stage pipeline was developed: face detection using MTCNN, face embedding extraction via VGGFace ResNet-50 CNN, and classification using SVM with RBF kernel for race, gender, and age prediction.
• The trained classifier achieved state-of-the-art performance and was used to quantify biases in Stable Diffusion XL outputs across six races, two genders, five age groups, 32 professions, and eight attributes.
• SDXL-Inc was developed by fine-tuning Stable Diffusion XL using learnable prompts and CLIP embeddings to align image and text features, enabling balanced generation across demographic groups.
• SDXL-Div was introduced to increase facial diversity by injecting random race and gender attributes into prompts when unspecified, using GPT-4 to detect and regulate prompt content.
• A GPT-4-in-the-loop system was implemented to analyze user prompts and inject missing demographic attributes (race, gender) to ensure balanced representation.
• The ITI-GEN model was retrained on curated data from the Flickr-Faces-HQ dataset using the authors’ classifier to ensure balanced gender and race representation in training.

Experimental results

Research questions

• RQ1 To what extent do Stable Diffusion XL-generated faces exhibit racial and gender stereotypes across diverse demographic categories?
• RQ2 How effective are classifier-guided prompting and GPT-4-in-the-loop prompt regulation in reducing representation bias in text-to-image generation?
• RQ3 Can debiasing methods like SDXL-Inc and SDXL-Div outperform existing state-of-the-art approaches in promoting equitable facial representation?
• RQ4 To what degree do AI-generated faces homogenize facial features within racial and gender groups, and how can this be mitigated?
• RQ5 How does the integration of automated prompt injection via LLMs affect the fairness and diversity of generated images?

Key findings

• The proposed face classifier achieved state-of-the-art performance in race, gender, and age prediction, enabling accurate quantification of biases in Stable Diffusion XL outputs.
• Stable Diffusion XL exhibited strong stereotyping, such as depicting most Middle Eastern males as dark-skinned, bearded, and wearing traditional headwear.
• SDXL-Inc and SDXL-Div significantly outperformed existing debiasing methods in promoting balanced representation across all six races and two genders.
• The GPT-4-in-the-loop system successfully injected missing demographic attributes into prompts, improving fairness without requiring manual prompt engineering.
• Re-training ITI-GEN with the authors’ classifier produced a model with nearly equal racial representation, except for Latinx due to facial similarity to White and Middle Eastern groups.
• The study’s debiasing solutions and all training data are publicly available via a GitHub repository, enabling reproducibility and community adoption.