[Paper Review] Clustering of Social Media Messages for Humanitarian Aid Response during Crisis

Swati Padhee, Tanay Kumar Saha+2 more|arXiv (Cornell University)|Jul 23, 2020

Public Relations and Crisis Communication30 references4 citations

TL;DR

This paper proposes using state-of-the-art deep learning models, particularly RoBERTa, to automatically classify and cluster social media messages during crises into three humanitarian aid classification tasks: informativeness, need/supply intent, and UN cluster categories (Food, Shelter, Health, WASH). The method achieves high performance—RoBERTa reaches 89.20% F1 on informativeness and over 75% F1 on intent and cluster classification—demonstrating that contextual embeddings significantly outperform traditional methods and can support scalable, automated crisis response systems.

ABSTRACT

Social media has quickly grown into an essential tool for people to communicate and express their needs during crisis events. Prior work in analyzing social media data for crisis management has focused primarily on automatically identifying actionable (or, informative) crisis-related messages. In this work, we show that recent advances in Deep Learning and Natural Language Processing outperform prior approaches for the task of classifying informativeness and encourage the field to adopt them for their research or even deployment. We also extend these methods to two sub-tasks of informativeness and find that the Deep Learning methods are effective here as well.

Motivation & Objective

• To improve the automatic identification of informative crisis-related social media messages using modern deep learning models.
• To extend classification beyond binary informativeness to include intent (Need/Supply) and humanitarian aid type (UN clusters: Food, Shelter, Health, WASH).
• To evaluate the generalization capability of models trained on one crisis (Hurricane Dorian) to unseen crisis events.
• To provide reproducible datasets and annotation guidelines to support future research in crisis NLP.

Proposed method

• The authors use a pre-trained RoBERTa model fine-tuned on crisis-related Twitter data for three classification tasks: informativeness, need/supply intent, and UN cluster categories.
• A multi-label classification framework is applied to allow a single tweet to be labeled with multiple categories (e.g., both 'Food' and 'WASH').
• The model is fine-tuned using a dataset of 14,073 informative tweets collected during Hurricane Dorian using lexicon and location-based filtering.
• The authors compare RoBERTa with BERT, TF-IDF, and word embeddings (GloVe, Word2Vec) to evaluate performance gains from contextual representations.
• Generalization is tested by applying the Dorian-trained model to public datasets from Hurricane Harvey, Hurricane Maria, and Sri Lanka floods.
• The model is evaluated using standard NLP metrics: accuracy, F1-score (micro and macro), and precision/recall.

Experimental results

Research questions

• RQ1 Can pre-trained language models like RoBERTa significantly improve the classification of informative crisis-related social media messages compared to classical machine learning methods?
• RQ2 Can deep learning models effectively classify tweets into finer-grained humanitarian intent categories (Need vs. Supply) and UN cluster categories (Food, Shelter, Health, WASH)?
• RQ3 How well does a model trained on one crisis event (Hurricane Dorian) generalize to other, unseen crisis events?
• RQ4 To what extent do contextual embeddings (e.g., RoBERTa) outperform non-contextual embeddings (e.g., GloVe, Word2Vec) in crisis NLP tasks?

Key findings

• RoBERTa achieved the highest performance across all three tasks, with an F1 score of 89.20% on the informativeness classification task.
• For the Need vs. Supply intent classification, RoBERTa achieved a Micro-F1 of 71.98% for 'Need' and 75.98% for 'Supply'.
• On the UN cluster classification task, RoBERTa achieved the highest accuracy and Micro-F1 across all four categories (Food, Shelter, Health, WASH), with 56.42% of tweets classified as 'Food' and 30.1% as 'WASH'.
• The model predicted 8,370 (59.48%) tweets as seeking help and 9,296 (66.1%) as offering supply, with 3,593 tweets labeled for both.
• When tested on unseen crises, RoBERTa achieved 82.12% accuracy on the Sri Lanka floods dataset but only 54.30% and 59.85% on Hurricane Harvey and Maria, respectively, indicating limited zero-shot generalization.
• The model correctly identified a tweet as 'Non-Informative' despite a human label of 'Informative' when the tweet did not discuss humanitarian aid, validating the task's definition.