[Paper Review] Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models

Eddie L. Ungless, Björn Roß+1 more|arXiv (Cornell University)|May 26, 2023

Cinema and Media StudiesEconomics, Econometrics and Finance3 citations

TL;DR

This paper investigates how text-to-image models misrepresent non-cisgender identities through stereotyping, sexualization, and dehumanization, using manual analysis of model outputs, surveys, and interviews with non-cisgender individuals. It reveals that models consistently generate less realistic, more sexualized, and more stereotyped images for non-cisgender identities, and calls for community-led design, curated training data, and customizable representations to ensure ethical and inclusive AI development.

ABSTRACT

Cutting-edge image generation has been praised for producing high-quality images, suggesting a ubiquitous future in a variety of applications. However, initial studies have pointed to the potential for harm due to predictive bias, reflecting and potentially reinforcing cultural stereotypes. In this work, we are the first to investigate how multimodal models handle diverse gender identities. Concretely, we conduct a thorough analysis in which we compare the output of three image generation models for prompts containing cisgender vs. non-cisgender identity terms. Our findings demonstrate that certain non-cisgender identities are consistently (mis)represented as less human, more stereotyped and more sexualised. We complement our experimental analysis with (a)~a survey among non-cisgender individuals and (b) a series of interviews, to establish which harms affected individuals anticipate, and how they would like to be represented. We find respondents are particularly concerned about misrepresentation, and the potential to drive harmful behaviours and beliefs. Simple heuristics to limit offensive content are widely rejected, and instead respondents call for community involvement, curated training data and the ability to customise. These improvements could pave the way for a future where change is led by the affected community, and technology is used to positively ``[portray] queerness in ways that we haven't even thought of'' rather than reproducing stale, offensive stereotypes.

Motivation & Objective

• To investigate how text-to-image models represent non-cisgender identities compared to cisgender identities, focusing on harmful biases.
• To identify specific forms of misrepresentation—such as dehumanization, sexualization, and stereotyping—through systematic manual analysis of model outputs.
• To understand the anticipated harms and desired representation from non-cisgender communities via surveys and interviews.
• To provide actionable recommendations for model developers based on community input, emphasizing community involvement and curated training data.
• To challenge technocratic solutions by centering the voices of marginalized communities in shaping ethical AI representation.

Proposed method

• Conducted a manual annotation study comparing image outputs from three text-to-image models (e.g., Stable Diffusion) for prompts containing cisgender vs. non-cisgender identity terms.
• Analyzed generated images for photorealism, presence of facial features, sexualization, and stereotypical attributes (e.g., pride flag colors, exaggerated gendered traits).
• Administered a survey to non-cisgender individuals to identify anticipated harms and preferred representation in AI-generated images.
• Conducted semi-structured interviews with non-cisgender participants to explore lived experiences of misrepresentation and desired model behavior.
• Used qualitative thematic analysis to interpret survey and interview data, focusing on concerns around misrepresentation, objectification, and harm.
• Collected and curated a dataset of model-generated images for analysis, made available only upon request to prevent misuse.

Figure 1: Four images generated by Stable Diffusion model in response to “Transgender women” . The black square indicates the model did not produce an output due to risk of NSFW content.

Experimental results

Research questions

• RQ1 How do text-to-image models differ in their representation of non-cisgender identities compared to cisgender identities in terms of realism, sexualization, and stereotyping?
• RQ2 What forms of representational harm do non-cisgender individuals anticipate from AI-generated images of their identities?
• RQ3 How do non-cisgender communities envision being represented in future text-to-image models?
• RQ4 What types of technical and design interventions are preferred by affected communities to reduce harmful representation?
• RQ5 To what extent do current technical solutions (e.g., content filters) align with the needs and values of non-cisgender users?

Key findings

• Text-to-image models consistently generate less realistic, more sexualized, and more stereotyped images for non-cisgender identities, particularly for transgender women and non-binary individuals.
• A significant proportion of generated images for non-cisgender identities lack clear facial features and are rendered in stereotypical colors (e.g., pride flag hues), contributing to dehumanization.
• Non-cisgender survey respondents expressed strong concerns about misrepresentation, especially the risk of reinforcing harmful societal beliefs and enabling discriminatory behaviors.
• Simple technical fixes like content filters were widely rejected by participants, who instead prioritized community involvement and curated training data.
• Interviewees emphasized the need for customizable representation and the importance of involving marginalized communities in model development to avoid tokenism and bias.
• The study highlights a gap between technocratic solutions and community-driven design, advocating for models that reflect queer identities in innovative, non-stereotypical ways.

(a) Photorealism