[Paper Review] Weights and Methodology Brief for the COVID-19 Symptom Survey by University of Maryland and Carnegie Mellon University, in Partnership with Facebook

Neta Barkay, Curtiss Cobb+6 more|arXiv (Cornell University)|Sep 25, 2020

Data-Driven Disease SurveillanceMedicine3 references62 citations

TL;DR

This paper outlines the sampling design and two-stage weighting approach (IPSW and post-stratification) used to make Facebook’s US CMU and global UMD COVID-19 symptom surveys representative, with privacy protections.

ABSTRACT

Facebook is partnering with academic institutions to support COVID-19 research. Currently, we are inviting Facebook app users in the United States to take a survey collected by faculty at Carnegie Mellon University (CMU) Delphi Research Center, and we are inviting Facebook app users in more than 200 countries or territories globally to take a survey collected by faculty at the University of Maryland (UMD) Joint Program in Survey Methodology (JPSM). As part of this initiative, we are applying best practices from survey statistics to design and execute two components: (1) sampling design and (2) survey weights, which make the sample more representative of the general population. This paper describes the methods we used in these efforts in order to allow data users to execute their analyses using the weights.

Motivation & Objective

• Explain the sampling design and target population for the Facebook COVID-19 symptom surveys.
• Describe the two-stage weighting methodology to reduce non-response and coverage error.
• Clarify privacy-preserving practices and data access for researchers.
• Provide guidelines for using the survey weights in analysis and variance estimation.

Proposed method

• Define the sampling frame as the Facebook Active User Base (FAUB) aged 18+ across 200+ countries/territories.
• Use daily repeated cross-sections with stratified random sampling and differential sampling across administrative boundaries.
• Apply Inverse Propensity Score Weighting (IPSW) to adjust for non-response using Facebook-derived covariates.
• Transform continuous covariates into buckets to match distributions and apply regularization and weight trimming.
• Apply Post-Stratification (PS) using benchmarks (US CPS 2018; UN 2019 projections) and IPSW inputs to represent the general adult population.
• Provide two sets of weights (for CLI estimates and for a larger set answering at least two questions) and guidance for variance estimation.

Experimental results

Research questions

• RQ1 How well do IPSW-based non-response adjustments represent the Facebook Active User Base?
• RQ2 How effective is post-stratification in aligning survey weights with country- or region-level population benchmarks?
• RQ3 What guidance is provided for using the weights in population and subpopulation estimates, including variance estimation?
• RQ4 What privacy-preserving practices accompany the weighting process and data access for researchers?

Key findings

• Weights are generated in two stages: IPSW for non-response adjustment and post-stratification for coverage adjustment.
• Covariates for non-response come from internal Facebook data and include age, gender, and geography; weights reflect how many adults in the population are represented by a respondent.
• Final weights enable region- or country-level statistics where administrative regions are included in post-stratification; otherwise, country-level statistics apply.
• Weights were designed to be simple, robust, and easily usable by researchers, with an option for further bias-correction by users.
• Aggregated weighted estimates are publicly available through UMD and CMU; non-aggregated data access requires Data Use Agreement.
• Early US weights were adjusted post hoc due to prior scaling decisions; users with older weights are advised to adopt new weights.