[Paper Review] Floods Detection in Twitter Text and Images

Naina Said, Kashif Ahmad+3 more|arXiv (Cornell University)|Nov 30, 2020

Flood Risk Assessment and Management8 references4 citations

TL;DR

This paper presents a multimodal approach for detecting flood-related events in Italian-language Twitter data by combining textual and visual features. It employs BERT, Bag-of-Words, and pre-trained CNNs (DenseNet, VGG, ResNet) for feature extraction, fuses scores via late fusion, and achieves an F1-score of 0.80 on the development set, demonstrating the superiority of multimodal fusion over unimodal methods.

ABSTRACT

In this paper, we present our methods for the MediaEval 2020 Flood Related Multimedia task, which aims to analyze and combine textual and visual content from social media for the detection of real-world flooding events. The task mainly focuses on identifying floods related tweets relevant to a specific area. We propose several schemes to address the challenge. For text-based flood events detection, we use three different methods, relying on Bog of Words (BOW) and an Italian Version of Bert individually and in combination, achieving an F1-score of 0.77%, 0.68%, and 0.70% on the development set, respectively. For the visual analysis, we rely on features extracted via multiple state-of-the-art deep models pre-trained on ImageNet. The extracted features are then used to train multiple individual classifiers whose scores are then combined in a late fusion manner achieving an F1-score of 0.75%. For our mandatory multi-modal run, we combine the classification scores obtained with the best textual and visual schemes in a late fusion manner. Overall, better results are obtained with the multimodal scheme achieving an F1-score of 0.80% on the development set.

Motivation & Objective

• To develop a multimodal system for detecting real-world flood events in Italian Twitter data using both textual and visual content.
• To evaluate the effectiveness of unimodal approaches (text-only and image-only) versus multimodal fusion for flood detection.
• To address class imbalance in both textual and visual data using SMOTE for synthetic data augmentation.
• To compare traditional BoW with deep learning-based BERT for Italian text representation in flood detection.
• To explore late fusion strategies that combine scores from multiple models to improve classification performance.

Proposed method

• Textual features are extracted using Bag-of-Words (BoW) and fine-tuned Italian BERT, followed by training a Naive Bayes and logistic regression classifier, respectively.
• Visual features are extracted using three pre-trained CNNs (DenseNet, VGG-19, ResNet) on ImageNet, with SVM classifiers trained on these features.
• Class imbalance is mitigated using SMOTE, increasing the minority class (flooded) by a factor of 3 to balance the dataset.
• Late fusion combines posterior probabilities from individual models by averaging scores, with equal weights assigned to all models in the multimodal run.
• The final multimodal system fuses the best-performing text and image models using late fusion to improve detection accuracy.
• All models are trained and evaluated on a dataset of Italian tweets with associated images, focusing on flood relevance in a specific Italian region.

Experimental results

Research questions

• RQ1 Can a combination of textual and visual features from Italian Twitter improve flood detection performance compared to unimodal approaches?
• RQ2 How effective is the Italian BERT model compared to traditional BoW for classifying flood-related tweets in Italian?
• RQ3 Does late fusion of multiple deep learning models trained on visual features outperform individual models in flood detection?
• RQ4 To what extent does data imbalance affect model performance, and can SMOTE effectively mitigate this issue in both text and image classification?
• RQ5 Does multimodal late fusion with equal weighting consistently improve F1-score over the best unimodal baseline?

Key findings

• The multimodal late fusion approach achieved the highest F1-score of 0.80 on the development set, outperforming all unimodal methods.
• The BoW-based text model achieved the best individual performance among text methods with an F1-score of 0.77 on the development set.
• The visual-only model achieved an F1-score of 0.75 on the development set, indicating strong performance from deep CNN features.
• The joint use of BoW and BERT in late fusion resulted in a slightly lower F1-score (0.70) than BoW alone, suggesting BERT did not improve performance in this context.
• The test set performance was significantly lower across all runs, with the multimodal run achieving only 0.093 F1-score, indicating potential distribution shift or test set difficulty.
• The average F1-score across all teams was below 20%, highlighting the high complexity of the flood detection task on the test set.