[Paper Review] Multi-Modal Machine Learning for Flood Detection in News, Social Media and Satellite Sequences

Kashif Ahmad, Konstantin Pogorelov+5 more|arXiv (Cornell University)|Oct 7, 2019

Anomaly Detection Techniques and Applications11 references4 citations

TL;DR

This paper presents a multi-modal machine learning framework for flood detection across news, social media, and satellite imagery in the MediaEval 2019 challenge. It combines deep features from pre-trained CNNs (AlexNet, VGG, ResNet) with color and scene-level features using early and late fusion, achieving an F1-score of 82.63 on news image topic disambiguation and 58.82 on satellite-based flood detection.

ABSTRACT

In this paper we present our methods for the MediaEval 2019 Mul-timedia Satellite Task, which is aiming to extract complementaryinformation associated with adverse events from Social Media andsatellites. For the first challenge, we propose a framework jointly uti-lizing colour, object and scene-level information to predict whetherthe topic of an article containing an image is a flood event or not.Visual features are combined using early and late fusion techniquesachieving an average F1-score of82.63,82.40,81.40and76.77. Forthe multi-modal flood level estimation, we rely on both visualand textual information achieving an average F1-score of58.48and46.03, respectively. Finally, for the flooding detection in time-based satellite image sequences we used a combination of classicalcomputer-vision and machine learning approaches achieving anaverage F1-score of58.82%

Motivation & Objective

• To develop a multi-modal system for detecting flood events using diverse data sources including news articles, social media posts, and satellite image sequences.
• To address the challenge of classifying flood-related content in news images with high accuracy using visual and textual features.
• To estimate flood levels in social media images by detecting whether people are standing in water above the knee.
• To detect flooding events in time-series satellite images using a hybrid computer vision and machine learning approach.
• To overcome data limitations such as cloud cover, seasonal changes, and image misalignment in satellite sequences.

Proposed method

• Extracted deep visual features using pre-trained CNNs (AlexNet, VGG, ResNet) on ImageNet and Places datasets for object and scene-level understanding.
• Combined features using early fusion (concatenation of feature vectors) and late fusion (simple averaging and PSO-based weighted averaging) for improved classification.
• Integrated color and texture features via the LIRE library using Joint Composite Descriptor (JCD) for global image representation.
• For flood level estimation, applied a two-stage visual method: first classifying flood vs. non-flood images using deep features, then detecting human body keypoints with OpenPose to assess water level relative to knee height.
• For textual analysis, applied both Bag-of-Words (BoW) and LSTM models to classify articles based on flood-related content.
• For satellite sequences, applied a multi-stage image processing pipeline: cloud masking via thresholding, image resizing to 128×128, HSV color space conversion, S/V channel masking, morphological filtering, and texture analysis using grey-level co-occurrence matrices (GLCM) followed by random forest classification.

Experimental results

Research questions

• RQ1 Can early and late fusion of multi-scale visual features (color, object, scene) improve flood detection accuracy in news images?
• RQ2 To what extent can visual and textual features complement each other in estimating flood levels in social media content?
• RQ3 How effective is a classical image processing pipeline with morphological filtering and texture analysis for detecting flooding in satellite image sequences with cloud cover and misalignment?
• RQ4 Does PSO-based late fusion outperform equal-weighted or early fusion strategies in multi-modal flood detection?
• RQ5 Can human body keypoint detection via OpenPose reliably indicate whether a person is standing in water above the knee in flood-related images?

Key findings

• The PSO-based late fusion method achieved the highest F1-score of 82.63 on the NITD task, outperforming equal-weighted fusion (82.40) and early fusion (76.77).
• Incorporating JCD-based color features did not improve performance and slightly degraded results, with F1-score dropping to 81.40 in the run including color features.
• Visual-based flood level estimation achieved an F1-score of 58.48, significantly outperforming the textual approach (46.03) using BoW and LSTM models.
• The satellite-based flood detection system achieved an F1-score of 58.82 on the CSS task, demonstrating effectiveness despite challenges like cloud cover and image misalignment.
• The system struggled to distinguish flooding-induced changes from seasonal vegetation growth, indicating a limitation in generalization under similar visual changes.
• Ensemble learning with re-sampled datasets improved performance on imbalanced data, particularly for the rare class (flood-related images).