[Paper Review] Quasi-experimental Designs for Assessing Response on Social Media to Policy Changes

Yijun Tian, Rumi Chunara|arXiv (Cornell University)|Mar 30, 2020

Smoking Behavior and Cessation25 references4 citations

TL;DR

This paper proposes using quasi-experimental designs (QEDs) to analyze social media reactions to rapid tobacco policy changes, such as e-cigarette flavor bans and taxation, controlling for time and location confounders. By applying QEDs to Twitter and Reddit data, it reveals strong negative sentiment toward policies and government actions, offering real-time insights complementary to traditional surveys.

ABSTRACT

Regulation of tobacco products is rapidly evolving. Understanding public sentiment in response to changes is very important as authorities assess how to effectively protect population health. Social media systems are widely recognized to be useful for collecting data about human preferences and perceptions. However, how social media data may be used, in rapid policy change settings, given challenges of narrow time periods and specific locations and non-representative the population using social media is an open question. In this paper we apply quasi-experimental designs, which have been used previously in observational data such as social media, to control for time and location confounders on social media, and then use content analysis of Twitter and Reddit posts to illustrate the content of reactions to tobacco flavor bans and the effect of taxation on e-cigarettes. Conclusions distill the potential role of social media in settings of rapidly changing regulation, in complement to what is learned by traditional denominator-based representative surveys.

Motivation & Objective

• To address the limitations of traditional surveys in capturing real-time public sentiment on rapidly changing tobacco policies.
• To evaluate the feasibility of using social media data for assessing policy response in time-sensitive regulatory contexts.
• To apply quasi-experimental designs to control for time and location confounders in observational social media data.
• To examine public sentiment on e-cigarette flavor bans and taxation using content analysis of Twitter and Reddit.
• To demonstrate that social media can serve as a timely, complementary signal to traditional surveys in public health policy evaluation.

Proposed method

• Applying quasi-experimental designs (QEDs) to observational social media data to reduce confounding from time and location.
• Using content analysis to categorize and quantify sentiment in Twitter and Reddit posts related to tobacco policy changes.
• Focusing on specific policy events: e-cigarette flavor bans and e-cigarette taxation in the U.S.
• Employing keyword-based data collection for e-cigarette products, policies, and related terms across platforms.
• Comparing sentiment trends before and after policy implementation using synthetic control methods.
• Validating findings against existing survey data to contextualize social media sentiment.

Experimental results

Research questions

• RQ1 How do public sentiments on social media respond to the implementation of e-cigarette flavor bans?
• RQ2 What is the nature and direction of sentiment toward e-cigarette taxation on platforms like Twitter and Reddit?
• RQ3 To what extent can quasi-experimental designs reduce confounding in social media data for policy evaluation?
• RQ4 How does social media sentiment compare to findings from traditional representative surveys on tobacco policy?
• RQ5 What role can real-time social media data play in informing public health policy decisions?

Key findings

• Social media reactions to e-cigarette flavor bans and taxation showed strong negative sentiment toward both policies and government actions.
• The sentiment was particularly pronounced in response to regulatory interventions, suggesting public resistance or skepticism toward top-down policy changes.
• Despite limitations in representativeness, social media provided real-time data on public response, capturing immediate reactions that surveys often miss.
• The study found no evidence of widespread public support for flavor bans in social media discourse, contrasting with some survey findings that report higher support.
• Quasi-experimental designs effectively controlled for time and location confounders, enabling more reliable inference from observational social media data.
• The results suggest that social media can serve as a timely, early-warning signal for public sentiment, especially for policies with rapid implementation and high public visibility.