[Paper Review] Crowdbreaks: Tracking Health Trends using Public Social Media Data and Crowdsourcing

Martin Mueller, Marcel Salathé|arXiv (Cornell University)|May 14, 2018

Data-Driven Disease Surveillance21 references3 citations

TL;DR

Crowdbreaks is an open, real-time platform that leverages crowdsourced labeling of public social media data—primarily Twitter—to track health trends using active learning and continuous model retraining. By automating data collection, filtering, labeling, and classifier training, it addresses concept drift and high annotation costs, enabling dynamic, transparent, and reusable public health surveillance tools with demonstrated utility in monitoring vaccine sentiment and disease-related trends.

ABSTRACT

In the past decade, tracking health trends using social media data has shown great promise, due to a powerful combination of massive adoption of social media around the world, and increasingly potent hardware and software that enables us to work with these new big data streams. At the same time, many challenging problems have been identified. First, there is often a mismatch between how rapidly online data can change, and how rapidly algorithms are updated, which means that there is limited reusability for algorithms trained on past data as their performance decreases over time. Second, much of the work is focusing on specific issues during a specific past period in time, even though public health institutions would need flexible tools to assess multiple evolving situations in real time. Third, most tools providing such capabilities are proprietary systems with little algorithmic or data transparency, and thus little buy-in from the global public health and research community. Here, we introduce Crowdbreaks, an open platform which allows tracking of health trends by making use of continuous crowdsourced labelling of public social media content. The system is built in a way which automatizes the typical workflow from data collection, filtering, labelling and training of machine learning classifiers and therefore can greatly accelerate the research process in the public health domain. This work introduces the technical aspects of the platform and explores its future use cases.

Motivation & Objective

• To address the limitations of static, non-reusable NLP models in public health surveillance that suffer from concept drift and outdated training data.
• To reduce the high cost and inefficiency of manual data labeling by implementing a continuous, crowdsourced labeling pipeline.
• To create a transparent, open platform that supports real-time, multi-class health trend monitoring for public health institutions and researchers.
• To enable flexible, reusable, and up-to-date machine learning models for tracking evolving health behaviors and sentiments on social media.
• To support fine-grained categorization of health-related content, such as vaccine hesitancy or disease symptoms, through scalable human-in-the-loop labeling.

Proposed method

• The platform uses a streaming pipeline to collect real-time Twitter data using keyword-based filtering for health-related topics.
• It implements an active learning framework that prioritizes tweets with high prediction uncertainty and recent temporal relevance for labeling.
• Crowdsourced annotations are collected through a web-based user interface built with Ruby on Rails and PostgreSQL, enabling scalable data labeling.
• Labelled data is used to retrain supervised machine learning classifiers, such as fastText for sentiment analysis, in a continuous feedback loop.
• The system integrates with Elasticsearch for data storage and Kibana for visualization, and uses Redis for message queuing and priority management.
• A Flask API connects the streaming pipeline with the user interface, enabling real-time data flow and model updates.

Experimental results

Research questions

• RQ1 How can continuous, real-time labeling of public social media data improve the robustness and reusability of health trend detection models?
• RQ2 To what extent can active learning with uncertainty sampling reduce labeling costs while maintaining or improving model performance in health surveillance?
• RQ3 Can an open, transparent, and community-driven platform like Crowdbreaks overcome the limitations of proprietary systems such as Google Flu Trends?
• RQ4 How effective is the platform in capturing nuanced health-related sentiments, such as vaccine hesitancy, in real time?
• RQ5 What is the impact of temporal relevance and label uncertainty on the efficiency of data labeling and model training in dynamic social media environments?

Key findings

• Crowdbreaks enables real-time tracking of health trends, such as vaccine sentiment, through continuous data collection and active learning-based labeling.
• The platform successfully mitigates concept drift by retraining models with newly labeled data, improving long-term model performance and reusability.
• By prioritizing uncertain and recent data, the system reduces labeling costs and increases labeling efficiency compared to random sampling.
• The use of fastText classifiers trained on crowdsourced labels achieved stable performance in real-time sentiment classification of vaccine-related tweets.
• The platform's architecture supports scalable, transparent, and extensible health surveillance, with potential for integration with other data sources like Wikipedia click rates for hybrid disease forecasting.
• The system demonstrates feasibility for large-scale, multi-class annotation of health-related content, enabling fine-grained public health monitoring.