[Paper Review] Leveraging Administrative Data for Bias Audits: Assessing Disparate Coverage with Mobility Data for COVID-19 Policy

Amanda Coston, Neel Guha+4 more|arXiv (Cornell University)|Nov 14, 2020

Imbalanced Data Classification Techniques50 references64 citations

TL;DR

This paper proposes using voter roll administrative data to audit demographic bias in smartphone-based mobility data, such as SafeGraph, for COVID-19 policy. By linking individual-level voter turnout, race, and age data from North Carolina’s 2018 election to POI-level mobility data, the authors identify significant underrepresentation of older and non-white populations in mobility datasets, which risks distorting public health resource allocation and exacerbating health inequities.

ABSTRACT

Anonymized smartphone-based mobility data has been widely adopted in devising and evaluating COVID-19 response strategies such as the targeting of public health resources. Yet little attention has been paid to measurement validity and demographic bias, due in part to the lack of documentation about which users are represented as well as the challenge of obtaining ground truth data on unique visits and demographics. We illustrate how linking large-scale administrative data can enable auditing mobility data for bias in the absence of demographic information and ground truth labels. More precisely, we show that linking voter roll data -- containing individual-level voter turnout for specific voting locations along with race and age -- can facilitate the construction of rigorous bias and reliability tests. These tests illuminate a sampling bias that is particularly noteworthy in the pandemic context: older and non-white voters are less likely to be captured by mobility data. We show that allocating public health resources based on such mobility data could disproportionately harm high-risk elderly and minority groups.

Motivation & Objective

• To address the lack of independent validation for smartphone-based mobility data used in high-stakes public health policy during the COVID-19 pandemic.
• To investigate whether mobility data from platforms like SafeGraph systematically underrepresents older and non-white populations—key risk groups for severe COVID-19 outcomes.
• To develop a method for auditing demographic bias in mobility data without requiring ground truth labels or demographic information from the mobility dataset itself.
• To demonstrate how administrative data can serve as a high-fidelity proxy for ground truth in bias audits when direct validation is unavailable.
• To illustrate the policy consequences of relying on biased mobility data, particularly in resource allocation decisions for vulnerable populations.

Proposed method

• Leverages North Carolina’s 2018 general election voter turnout records as a ground truth proxy, linking individual voter data (including race, age, and polling location) to POI-level mobility data.
• Uses voter turnout at specific polling locations as a proxy for actual visitation, assuming that registered voters who voted are accurately recorded in the mobility data if they visited the polling place.
• Constructs a bias audit framework that compares observed mobility data coverage (fraction of visits recorded) to expected turnout (from voter rolls) at each POI, stratified by age and race.
• Applies statistical tests (e.g., t-tests) to detect significant differences in coverage across demographic subgroups, formalizing assumptions about data reliability and representativeness.
• Estimates demographic disparities in coverage by binning polling locations into four age-race groups based on median proportions of over-65 and non-white voters.
• Evaluates policy implications by comparing resource allocation based on SafeGraph traffic versus actual voter turnout, identifying misallocation risks.

Experimental results

Research questions

• RQ1 To what extent is SafeGraph mobility data biased in its coverage of older and non-white populations?
• RQ2 Can administrative data such as voter turnout records serve as a valid proxy for ground truth in auditing mobility data for demographic bias?
• RQ3 How do demographic disparities in mobility data coverage affect policy decisions such as public health resource allocation?
• RQ4 What are the quantitative impacts of relying on biased mobility data for proportional allocation of resources like testing sites or masks?
• RQ5 How can bias in mobility data be corrected or mitigated in policy-relevant applications?

Key findings

• SafeGraph mobility data systematically underrepresents older and non-white populations, with coverage for the oldest and most non-white polling locations being significantly lower than actual voter turnout.
• The analysis reveals a 37% under-allocation of resources to the oldest and most non-white demographic group when relying solely on SafeGraph data, compared to optimal allocation based on voter turnout.
• Conversely, the youngest and whitest group receives a 33% over-allocation of resources under SafeGraph-based decisions, indicating a substantial policy distortion.
• The disparity in coverage is statistically significant (p < 0.05) across demographic subgroups, confirming systematic bias in the data.
• Even when controlling for geographic and demographic variation, the bias persists, suggesting that the underlying sampling mechanism favors younger and white users.
• The study demonstrates that without adjustment, mobility data can exacerbate existing health inequities by failing to target high-risk populations effectively.