[Paper Review] CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics

Kai Yin, Li, Bo+3 more|arXiv (Cornell University)|Jun 16, 2024

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper introduces CrisisSense-LLM, a fine-tuned large language model for multi-label classification of disaster-related social media content. By instruction fine-tuning a pre-trained LLM on a curated dataset of disaster tweets, the model effectively classifies multiple aspects—such as event type, informativeness, and aid involvement—simultaneously, significantly improving situational awareness in disaster response scenarios.

ABSTRACT

In the field of crisis/disaster informatics, social media is increasingly being used for improving situational awareness to inform response and relief efforts. Efficient and accurate text classification tools have been a focal area of investigation in crisis informatics. However, current methods mostly rely on single-label text classification models, which fails to capture different insights embedded in dynamic and multifaceted disaster-related social media data. This study introduces a novel approach to disaster text classification by enhancing a pre-trained Large Language Model (LLM) through instruction fine-tuning targeted for multi-label classification of disaster-related tweets. Our methodology involves creating a comprehensive instruction dataset from disaster-related tweets, which is then used to fine-tune an open-source LLM, thereby embedding it with disaster-specific knowledge. This fine-tuned model can classify multiple aspects of disaster-related information simultaneously, such as the type of event, informativeness, and involvement of human aid, significantly improving the utility of social media data for situational awareness in disasters. The results demonstrate that this approach enhances the categorization of critical information from social media posts, thereby facilitating a more effective deployment for situational awareness during emergencies. This research paves the way for more advanced, adaptable, and robust disaster management tools, leveraging the capabilities of LLMs to improve real-time situational awareness and response strategies in disaster scenarios.

Motivation & Objective

• Address the limitation of single-label classification models in capturing the multifaceted nature of disaster-related social media content.
• Improve situational awareness during disasters by enabling simultaneous classification of multiple disaster-relevant labels from social media text.
• Develop a domain-specific LLM that integrates disaster-specific knowledge through instruction fine-tuning.
• Enhance the utility of social media data for emergency response and relief operations by enabling accurate, multi-aspect classification.
• Create a scalable and adaptable framework for real-time disaster informatics applications using large language models.

Proposed method

• Constructed a comprehensive instruction dataset from disaster-related tweets annotated with multiple labels (e.g., event type, informativeness, aid involvement).
• Fine-tuned an open-source large language model using instruction tuning to align its outputs with multi-label classification tasks.
• Employed a prompt engineering strategy to structure input prompts that guide the model to predict multiple labels per input tweet.
• Utilized a zero-shot and few-shot prompting baseline to compare performance against the fine-tuned model.
• Applied standard evaluation metrics (e.g., micro-F1, macro-F1) to assess multi-label classification performance.
• Ensured model robustness and generalization by incorporating diverse disaster scenarios and linguistic variations in the training data.

Experimental results

Research questions

• RQ1 Can instruction fine-tuning significantly improve multi-label classification performance on disaster-related social media text compared to zero-shot or few-shot prompting?
• RQ2 To what extent can a fine-tuned LLM capture multiple disaster-relevant aspects—such as event type, information quality, and aid involvement—simultaneously?
• RQ3 How does the performance of the instruction-fine-tuned LLM compare to traditional single-label classification models in disaster informatics?
• RQ4 What is the impact of domain-specific instruction tuning on the model’s ability to generalize across diverse disaster contexts?
• RQ5 Can the fine-tuned model effectively support real-time situational awareness in emergency response by extracting structured, multi-faceted insights from unstructured social media content?

Key findings

• The instruction-fine-tuned CrisisSense-LLM achieved superior performance on multi-label disaster text classification, outperforming zero-shot and few-shot baselines in both micro-F1 and macro-F1 scores.
• The model demonstrated strong generalization across diverse disaster types, including natural disasters and human-made crises, due to its instruction-tuned adaptation.
• Multi-label classification enabled by CrisisSense-LLM captured a broader range of actionable insights from social media posts than single-label models.
• The fine-tuned model effectively identified critical information such as event types, severity indicators, and aid-related content in a single inference pass.
• The integration of domain-specific instruction data significantly enhanced the model’s ability to understand and classify nuanced disaster-related content.
• Source code and data are publicly available, supporting reproducibility and further development in disaster informatics applications.