[Paper Review] OMG U got flu? Analysis of shared health messages for bio-surveillance

Nigel Collier, Nguyễn Trường Sơn+1 more|Oct 13, 2011

Misinformation and Its Impacts13 citations

TL;DR

This paper proposes using self-protective health behaviors reported in Twitter messages—such as avoiding crowds or increasing hygiene—as real-time indicators for bio-surveillance. Using supervised learning with unigrams, bigrams, and regular expressions, the authors classify tweets into four protective behavior categories and a diagnosis category, achieving high inter-annotator agreement (kappa = 0.86) and showing a moderate correlation (Spearman’s rho) with official WHO/NREVSS influenza data, supporting social media as a low-cost early warning system for disease outbreaks.

ABSTRACT

Background: Micro-blogging services such as Twitter offer the potential to crowdsource epidemics in real-time. However, Twitter posts ('tweets') are often ambiguous and reactive to media trends. In order to ground user messages in epidemic response we focused on tracking reports of self-protective behaviour such as avoiding public gatherings or increased sanitation as the basis for further risk analysis. Results: We created guidelines for tagging self protective behaviour based on Jones and Salathé (2009)'s behaviour response survey. Applying the guidelines to a corpus of 5283 Twitter messages related to influenza like illness showed a high level of inter-annotator agreement (kappa 0.86). We employed supervised learning using unigrams, bigrams and regular expressions as features with two supervised classifiers (SVM and Naive Bayes) to classify tweets into 4 self-reported protective behaviour categories plus a self-reported diagnosis. In addition to classification performance we report moderately strong Spearman's Rho correlation by comparing classifier output against WHO/NREVSS laboratory data for A(H1N1) in the USA during the 2009-2010 influenza season. Conclusions: The study adds to evidence supporting a high degree of correlation between pre-diagnostic social media signals and diagnostic influenza case data, pointing the way towards low cost sensor networks. We believe that the signals we have modelled may be applicable to a wide range of diseases.

Motivation & Objective

• To develop a method for identifying self-protective health behaviors in social media as early indicators of disease outbreaks.
• To address the ambiguity and media-reactive nature of flu-related tweets by focusing on behavioral responses rather than self-diagnoses.
• To create a reliable annotation framework for classifying protective behaviors in Twitter data.
• To evaluate the correlation between social media signals and official influenza surveillance data.

Proposed method

• Developed annotation guidelines based on Jones and Salathé (2009) for labeling self-protective behaviors in 5,283 flu-related tweets.
• Applied supervised learning using unigrams, bigrams, and regular expressions as features with SVM and Naive Bayes classifiers.
• Classified tweets into four protective behavior categories and one self-reported diagnosis category.
• Measured inter-annotator agreement using Cohen’s Kappa, achieving a high value of 0.86.
• Compared classifier outputs with WHO/NREVSS laboratory-confirmed A(H1N1) data for the 2009–2010 season using Spearman’s Rho correlation.
• Used a curated corpus of real-time Twitter messages to model behavioral signals as proxies for epidemic trends.

Experimental results

Research questions

• RQ1 Can self-protective behaviors expressed in social media be reliably detected and classified using NLP techniques?
• RQ2 How well do these detected behavioral signals correlate with official influenza surveillance data?
• RQ3 What level of agreement can be achieved among human annotators when labeling self-protective health behaviors in social media?
• RQ4 Can social media signals based on behavioral responses serve as a low-cost alternative to traditional disease surveillance systems?
• RQ5 To what extent do social media trends in protective behaviors anticipate official flu case reports?

Key findings

• The annotation scheme achieved high inter-annotator agreement with a Cohen’s Kappa value of 0.86, indicating strong reliability in labeling protective behaviors.
• The supervised classifiers (SVM and Naive Bayes) effectively distinguished between four protective behavior categories and self-reported diagnosis.
• A moderate but statistically significant Spearman’s Rho correlation was observed between the classifier output and official WHO/NREVSS A(H1N1) laboratory data.
• The study demonstrates that pre-diagnostic social media signals based on behavioral responses correlate with real-world disease trends.
• The results support the feasibility of using social media as a low-cost, real-time sensor network for epidemic detection.