[Paper Review] "What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairness

McKane Andrus, Elena Spitzer+2 more|arXiv (Cornell University)|Oct 30, 2020

Ethics and Social Impacts of AI56 references20 citations

TL;DR

This paper investigates the practical challenges practitioners face in obtaining demographic data for algorithmic fairness, revealing that legal, ethical, and technical barriers often prevent access despite its necessity. It argues against simply lowering data collection barriers, instead advocating for normative frameworks and inclusive practices to ethically assess and mitigate bias without relying on direct demographic collection.

ABSTRACT

As calls for fair and unbiased algorithmic systems increase, so too does the number of individuals working on algorithmic fairness in industry. However, these practitioners often do not have access to the demographic data they feel they need to detect bias in practice. Even with the growing variety of toolkits and strategies for working towards algorithmic fairness, they almost invariably require access to demographic attributes or proxies. We investigated this dilemma through semi-structured interviews with 38 practitioners and professionals either working in or adjacent to algorithmic fairness. Participants painted a complex picture of what demographic data availability and use look like on the ground, ranging from not having access to personal data of any kind to being legally required to collect and use demographic data for discrimination assessments. In many domains, demographic data collection raises a host of difficult questions, including how to balance privacy and fairness, how to define relevant social categories, how to ensure meaningful consent, and whether it is appropriate for private companies to infer someone's demographics. Our research suggests challenges that must be considered by businesses, regulators, researchers, and community groups in order to enable practitioners to address algorithmic bias in practice. Critically, we do not propose that the overall goal of future work should be to simply lower the barriers to collecting demographic data. Rather, our study surfaces a swath of normative questions about how, when, and whether this data should be procured, and, in cases where it is not, what should still be done to mitigate bias.

Motivation & Objective

• To understand the real-world challenges practitioners encounter in procuring demographic data for algorithmic fairness.
• To examine the ethical, legal, and technical constraints affecting demographic data use in industry and research.
• To assess whether alternative fairness methods—such as proxies, inference, or third-party audits—can effectively substitute for direct demographic data collection.
• To explore how data privacy regulations and anti-discrimination laws interact with fairness measurement in practice.
• To evaluate the role of data subjects in fairness processes and the feasibility of inclusive, consent-based data collection models.

Proposed method

• Conducted semi-structured interviews with 38 practitioners and professionals working in or adjacent to algorithmic fairness.
• Analyzed practitioner experiences across diverse domains to map the spectrum of demographic data availability and use.
• Explored legal and regulatory frameworks, including GDPR and U.S. anti-discrimination laws, in relation to data procurement.
• Evaluated privacy-preserving alternatives such as differential privacy, cryptographic methods, and third-party data handling.
• Assessed decentralized approaches like federated learning for enabling fairness analysis without centralized demographic data.
• Examined the role of data subject consent and community engagement in shaping ethical data use for fairness evaluation.

Experimental results

Research questions

• RQ1 What are the primary barriers practitioners face in accessing demographic data for algorithmic fairness assessments?
• RQ2 How do legal and regulatory frameworks such as GDPR influence the collection and use of demographic data in fairness work?
• RQ3 To what extent can privacy-preserving techniques like differential privacy or third-party data handling substitute for direct demographic data collection?
• RQ4 How do practitioners navigate ethical concerns around consent, representation, and group saliency when collecting demographic data?
• RQ5 What role can data subjects and community stakeholders play in shaping fairness evaluation processes without relying on traditional demographic data collection?

Key findings

• Many practitioners lack access to demographic data due to legal restrictions, privacy concerns, or organizational policies, even when such data is essential for bias detection.
• Legal frameworks like GDPR often restrict demographic data use, yet some jurisdictions (e.g., UK) have issued guidance permitting its use under specific fairness audit conditions.
• Alternative methods such as inference, proxies, or post-hoc group identification can reduce direct data collection but introduce risks of misrepresentation and reduced fairness accuracy.
• Third-party data collection and privacy-preserving techniques (e.g., differential privacy) shift responsibility and may not resolve core ethical questions about data saliency and group representation.
• Practitioners express strong concerns about data reliability, consent, and the potential for reinforcing systemic biases through flawed demographic categorizations.
• Inclusion of data subjects and communities in fairness processes can improve data relevance and ethical accountability, though such models are complex and costly to implement.