[Paper Review] Landslide Detection in Real-Time Social Media Image Streams

Ferda Ofli, Muhammad Ali Imran+5 more|arXiv (Cornell University)|Oct 3, 2021

Landslides and related hazards4 citations

TL;DR

This paper proposes a real-time computer vision system that detects landslides in social media images using a large, expert-annotated dataset of over 11,000 images. The best-performing model achieves an F1 score of 0.701 on the test set, enabling scalable, automated landslide detection to support global hazard mapping and emergency response.

ABSTRACT

Lack of global data inventories obstructs scientific modeling of and response to landslide hazards which are oftentimes deadly and costly. To remedy this limitation, new approaches suggest solutions based on citizen science that requires active participation. However, as a non-traditional data source, social media has been increasingly used in many disaster response and management studies in recent years. Inspired by this trend, we propose to capitalize on social media data to mine landslide-related information automatically with the help of artificial intelligence (AI) techniques. Specifically, we develop a state-of-the-art computer vision model to detect landslides in social media image streams in real time. To that end, we create a large landslide image dataset labeled by experts and conduct extensive model training experiments. The experimental results indicate that the proposed model can be deployed in an online fashion to support global landslide susceptibility maps and emergency response.

Motivation & Objective

• Address the global lack of comprehensive landslide inventories that hinder scientific modeling and emergency response.
• Overcome limitations of traditional data collection methods that rely on manual curation and active citizen participation.
• Leverage passive, real-time social media image streams as a scalable, low-cost data source for landslide detection.
• Develop a high-accuracy, deployable AI model for automatic landslide detection in diverse visual conditions.
• Enable real-time situational awareness and support for global landslide susceptibility mapping and humanitarian response.

Proposed method

• Constructed a large, diverse dataset of 11,000+ social media images from Twitter, Google, and BGS, annotated by three domain experts with high inter-annotator agreement.
• Trained and evaluated multiple deep learning architectures (e.g., ResNet, DenseNet) using transfer learning with data augmentation and class balancing strategies.
• Optimized model hyperparameters including learning rate, weight decay, and optimizer (Adam, SGD) through extensive experimentation.
• Employed class activation maps (CAMs) to interpret model attention and diagnose failure modes in false positives and negatives.
• Designed a real-time pipeline using Twitter Streaming API to collect tweets with landslide-related keywords, extract image URLs, and classify images via the trained model.
• Integrated geolocation inference and data persister modules to enable real-time dashboard visualization and database storage of detections.

Experimental results

Research questions

• RQ1 Can a deep learning model trained on diverse, expert-annotated social media images achieve high performance in real-time landslide detection?
• RQ2 How do different CNN architectures, optimizers, and hyperparameters affect model performance on landslide image classification?
• RQ3 What are the primary failure modes of the model, and how can attention visualization help diagnose misclassifications?
• RQ4 To what extent can passive social media image streams serve as a scalable, low-cost alternative to active citizen reporting for landslide monitoring?
• RQ5 What system architecture enables reliable, real-time deployment of a landslide detection model for global emergency response?

Key findings

• The best-performing model achieved an F1 score of 0.701 on the held-out test set, indicating strong generalization despite a significant drop in F1 compared to the validation set.
• The model exhibited a notable performance gap between validation (F1: 0.805) and test (F1: 0.701), primarily due to increased false positives (128 vs. 42) and false negatives (178 vs. 60) on the test set.
• Class activation maps revealed that the model successfully localizes landslide features in true positives and avoids misleading regions in true negatives, such as muddy roads or rocky beaches.
• False positives and negatives often occurred when the model failed to focus on relevant image regions or was misled by visually similar but non-landslide scenes.
• The study demonstrated that social media images can be a viable data source for landslide detection, with the model showing sufficient performance for real-world deployment.
• The proposed pipeline enables continuous, automated monitoring of social media for landslide-related images, supporting real-time situational awareness and emergency decision-making.