[Paper Review] Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman, Angelina Wang+5 more|arXiv (Cornell University)|May 2, 2023

Psychology of Moral and Emotional JudgmentNeuroscience3 citations

TL;DR

This paper introduces a taxonomy of four distinct types of representational harms in image tagging systems—reifying, erasing, demeaning, and stereotyping social groups—and analyzes how five computational fairness measurement approaches map unevenly across these harms. It demonstrates that no single measurement method can definitively identify a specific harm type, and that mitigating one type of harm may conflict with mitigating another, highlighting the need for context-aware, value-sensitive design in fairness evaluation.

ABSTRACT

In this paper, we examine computational approaches for measuring the "fairness" of image tagging systems, finding that they cluster into five distinct categories, each with its own analytic foundation. We also identify a range of normative concerns that are often collapsed under the terms "unfairness," "bias," or even "discrimination" when discussing problematic cases of image tagging. Specifically, we identify four types of representational harms that can be caused by image tagging systems, providing concrete examples of each. We then consider how different computational measurement approaches map to each of these types, demonstrating that there is not a one-to-one mapping. Our findings emphasize that no single measurement approach will be definitive and that it is not possible to infer from the use of a particular measurement approach which type of harm was intended to be measured. Lastly, equipped with this more granular understanding of the types of representational harms that can be caused by image tagging systems, we show that attempts to mitigate some of these types of harms may be in tension with one another.

Motivation & Objective

• To move beyond vague terms like 'bias' or 'unfairness' in image tagging fairness discussions by identifying specific, normatively grounded types of representational harms.
• To categorize existing computational fairness measurement approaches for image tagging into five distinct analytic foundations.
• To examine the mapping between these measurement approaches and the four identified types of representational harms.
• To reveal that no single measurement approach can definitively identify which type of harm is being assessed, challenging assumptions about interpretability of fairness metrics.
• To explore tensions between mitigating different types of representational harms, emphasizing that trade-offs are inherent in system design.

Proposed method

• The authors analyze 2305.01776, a peer-reviewed paper on image tagging fairness, to identify and define four distinct types of representational harms: reifying, erasing, demeaning, and stereotyping social groups.
• They cluster previously proposed computational fairness measurement methods into five categories based on their underlying analytic foundations, such as statistical parity, equal opportunity, and distributional similarity.
• The authors map each measurement approach to each of the four harm types, demonstrating that a single approach can be used to assess multiple types of harms.
• They use concrete examples from prior work—such as gender classification errors, underrepresentation of marginalized groups, and stereotypical tagging in pornographic content—to illustrate each harm type.
• The paper evaluates mitigation strategies for each harm type, showing that actions like removing tags to prevent stereotyping may inadvertently erase social groups, and vice versa.
• The analysis is grounded in real-world cases, including Tumblr’s adult content ban and alt-text generation for visually impaired users, to show context-dependent trade-offs.

Figure 1: Examples of how computational measurement approaches belonging to each of the five categories described earlier might be used to measure each of the four different types of representational harms that can be caused by image tagging systems. This mapping is by no means exhaustive.

Experimental results

Research questions

• RQ1 How do different computational fairness measurement approaches for image tagging systems relate to distinct types of representational harms?
• RQ2 What are the four distinct types of representational harms that image tagging systems can cause, and how do they reproduce harmful social hierarchies?
• RQ3 To what extent can a single fairness measurement approach be used to detect multiple types of representational harms?
• RQ4 Why is it not possible to infer the intended harm type from the use of a particular measurement approach alone?
• RQ5 In what ways do mitigation strategies for different types of representational harms conflict with one another?

Key findings

• The five computational fairness measurement approaches identified—such as statistical parity, equal opportunity, and distributional similarity—each have distinct analytic foundations and are not interchangeable in their interpretation.
• Four distinct types of representational harms were identified: reifying social groups (evidencing their existence), erasing them (rendering them invisible), demeaning them (applying offensive or stereotypical tags), and stereotyping (applying biased attribute distributions).
• Each of the five measurement approaches can be used to assess all four types of representational harms, meaning that the same metric cannot definitively signal which harm is being targeted.
• The study demonstrates that removing tags to prevent stereotyping or demeaning may lead to the erasure of marginalized groups, especially when applied to sensitive content like queer or sex-positive media.
• Mitigation strategies for one type of harm—such as removing tags to avoid reifying social groups—can exacerbate another, such as erasing them, revealing inherent tensions in fairness engineering.
• The context of use (e.g., alt-text for the blind vs. social media moderation) critically determines which harms are prioritized and which mitigations are appropriate.