[Paper Review] Survey of Text-based Epidemic Intelligence: A Computational Linguistic Perspective

Aditya Joshi, Sarvnaz Karimi+3 more|arXiv (Cornell University)|Mar 14, 2019

Data-Driven Disease SurveillanceMedicine61 references3 citations

TL;DR

This survey presents a computational linguistic framework for text-based epidemic intelligence, categorizing approaches into health mention classification (identifying disease-relevant text) and health event detection (tracking outbreaks over time or space). It reviews NLP techniques, annotation resources, and evaluation strategies, identifying key research gaps in reducing false alerts, enabling near-real-time detection, and integrating overlapping syndromes across physical, mental, and zoonotic health domains.

ABSTRACT

Epidemic intelligence deals with the detection of disease outbreaks using formal (such as hospital records) and informal sources (such as user-generated text on the web) of information. In this survey, we discuss approaches for epidemic intelligence that use textual datasets, referring to it as `text-based epidemic intelligence'. We view past work in terms of two broad categories: health mention classification (selecting relevant text from a large volume) and health event detection (predicting epidemic events from a collection of relevant text). The focus of our discussion is the underlying computational linguistic techniques in the two categories. The survey also provides details of the state-of-the-art in annotation techniques, resources and evaluation strategies for epidemic intelligence.

Motivation & Objective

• To provide a comprehensive computational linguistic perspective on text-based epidemic intelligence using informal textual data from the web.
• To address the challenge of detecting disease outbreaks early using unstructured text from sources like social media and news.
• To identify research gaps in reducing false alerts, improving timeliness, and expanding surveillance coverage to include overlapping syndromes.
• To evaluate existing annotation resources, evaluation strategies, and state-of-the-art NLP techniques for health mention and event detection.

Proposed method

• Classifies text-based epidemic intelligence into two stages: health mention classification (detecting disease-relevant content in individual texts) and health event detection (aggregating and analyzing relevant texts over time or space).
• Reviews computational linguistic techniques such as medical ontologies, statistical classifiers, topic models, and neural networks for health mention classification.
• Applies time-series and spatial analysis methods, including exponentially weighted moving averages, to detect temporal and geographical disease outbreaks.
• Evaluates performance using standard metrics and highlights the importance of task-specific features and pipeline integration in NLP components.
• Proposes integration of figurative language detection and spam filtering to improve precision and reduce false alerts.
• Explores the potential of combining surveillance across physical illnesses, mental health, and zoonotic diseases through shared symptom modeling.

Experimental results

Research questions

• RQ1 How can computational linguistic techniques improve the accuracy and efficiency of detecting disease outbreaks from unstructured text?
• RQ2 What are the key challenges in distinguishing literal health mentions from figurative or non-clinical language in social media?
• RQ3 How can health event detection systems achieve near-real-time performance for timely public health responses?
• RQ4 In what ways can overlapping syndromes and symptoms across physical, mental, and animal health be modeled to enhance surveillance coverage?
• RQ5 What role do medical ontologies and structured knowledge bases play in improving the performance of text-based epidemic intelligence systems?

Key findings

• More than 60% of initial epidemic reports originate from informal, text-based sources such as social media and online forums.
• Health mention classification benefits significantly from the use of medical ontologies and task-specific features, improving detection precision.
• Integrating target identification before personal health mention detection enhances the performance of downstream health event detection.
• False alerts remain a major challenge, particularly due to figurative language and spam in social media, necessitating improved filtering techniques.
• Near-real-time outbreak detection is feasible using social media data, though challenges in data integration and event modeling persist.
• Future epidemic intelligence systems should integrate related syndromes—such as mental health and zoonotic diseases—by modeling shared symptom patterns.