[Paper Review] Uncovering Bias in Face Generation Models

Cristian Muñoz, Sara Zannone+2 more|arXiv (Cornell University)|Feb 22, 2023

Face recognition and analysis4 citations

TL;DR

This paper proposes a comprehensive analysis of bias in face generation models, evaluating generators, attribute modifiers (InterfaceGAN), and post-processing mitigators using metrics like FID, Face Matching Error, and self-similarity. It finds that generators exhibit strong biases—75–85% preference for whiteness and 60–80% for female gender—while post-processing mitigators offer the most efficient fairness improvement with minimal quality loss, though they alter feature concentration in embedding spaces.

ABSTRACT

Recent advancements in GANs and diffusion models have enabled the creation of high-resolution, hyper-realistic images. However, these models may misrepresent certain social groups and present bias. Understanding bias in these models remains an important research question, especially for tasks that support critical decision-making and could affect minorities. The contribution of this work is a novel analysis covering architectures and embedding spaces for fine-grained understanding of bias over three approaches: generators, attribute modifier, and post-processing bias mitigators. This work shows that generators suffer from bias across all social groups with attribute preferences such as between 75%-85% for whiteness and 60%-80% for the female gender (for all trained CelebA models) and low probabilities of generating children and older men. Modifier and mitigators work as post-processor and change the generator performance. For instance, attribute channel perturbation strategies modify the embedding spaces. We quantify the influence of this change on group fairness by measuring the impact on image quality and group features. Specifically, we use the Fréchet Inception Distance (FID), the Face Matching Error and the Self-Similarity score. For Interfacegan, we analyze one and two attribute channel perturbations and examine the effect on the fairness distribution and the quality of the image. Finally, we analyzed the post-processing bias mitigators, which are the fastest and most computationally efficient way to mitigate bias. We find that these mitigation techniques show similar results on KL divergence and FID score, however, self-similarity scores show a different feature concentration on the new groups of the data distribution. The weaknesses and ongoing challenges described in this work must be considered in the pursuit of creating fair and unbiased face generation models.

Motivation & Objective

• To identify and quantify bias in state-of-the-art face generation models, including GANs and diffusion models.
• To investigate how architectural choices and embedding space manipulations influence fairness and image quality.
• To evaluate the effectiveness of attribute modifiers (e.g., InterfaceGAN) and post-processing bias mitigators in reducing group-level bias.
• To assess the trade-offs between fairness, image fidelity, and feature distribution when applying bias mitigation techniques.
• To provide a systematic benchmark for measuring bias and fairness in generated face datasets using multiple complementary metrics.

Proposed method

• Trained and evaluated multiple face generation models: DDPM, LDM, StyleGAN2, and CIPS on CelebA and FFHQ datasets.
• Applied attribute control via latent space manipulation in Z-space, W-space, PCA space (GANSpace), and style space (StyleSpace) for fine-grained attribute editing.
• Used InterfaceGAN to perform one- and two-channel attribute perturbations in the style space to analyze fairness and quality trade-offs.
• Measured image quality using Fréchet Inception Distance (FID), face matching error, and self-similarity score to detect distributional shifts and artifacts.
• Evaluated post-processing bias mitigators as computationally efficient alternatives to retraining, comparing their impact on KL divergence and FID.
• Analyzed data distributions across protected attributes (gender, race, age, emotion) to quantify underrepresentation and bias in generated samples.

((a))

Experimental results

Research questions

• RQ1 To what extent do pre-trained face generation models exhibit bias toward specific social groups, such as race, gender, and age?
• RQ2 How do attribute manipulation techniques like InterfaceGAN affect fairness and image quality across different embedding spaces?
• RQ3 What is the impact of post-processing bias mitigators on image fidelity and distributional fairness compared to direct model fine-tuning?
• RQ4 How do standard metrics like FID fail to capture bias-related artifacts, and what additional metrics (e.g., self-similarity) are needed for robust evaluation?
• RQ5 What trade-offs exist between bias mitigation, image quality, and feature concentration in the latent space?

Key findings

• Generators trained on CelebA exhibit strong bias, with 75–85% preference for whiteness and 60–80% for female gender across all models.
• Children and older men are significantly underrepresented, with only 10–20% of generated images falling into the 10–19 and 50+ age groups.
• Post-processing bias mitigators are the most computationally efficient method, showing similar FID and KL divergence scores to other methods but altering feature concentration in the data distribution.
• Self-similarity scores reveal that mitigators and modifiers shift feature concentration, indicating potential distributional distortions not captured by FID alone.
• Large perturbations in the style space (e.g., via InterfaceGAN) introduce unwanted artifacts that degrade image quality, even when FID remains stable.
• The combination of FID and self-similarity provides a more robust evaluation than FID alone, as self-similarity detects anomalies in attribute-specific feature distributions.

((b))