[Paper Review] Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response

Ferda Ofli, Firoj Alam+1 more|arXiv (Cornell University)|Apr 14, 2020

Public Relations and Crisis Communication31 references69 citations

TL;DR

The paper proposes a multimodal deep learning architecture that fuses Twitter text and attached images to improve disaster-response classification tasks on CrisisMMD, outperforming unimodal baselines.

ABSTRACT

Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning techniques. Specifically, we utilize convolutional neural networks to define a multimodal deep learning architecture with a modality-agnostic shared representation. Extensive experiments on real-world disaster datasets show that the proposed multimodal architecture yields better performance than models trained using a single modality (e.g., either text or image).

Motivation & Objective

• Motivate robust disaster-response analysis by leveraging both textual and visual cues from social media.
• Investigate whether multimodal fusion yields better predictions than unimodal text or image models.
• Provide baseline multimodal and unimodal results on CrisisMMD for two tasks: informativeness and humanitarian categorization.
• Explore challenges and future directions for multimodal crisis data analysis.

Proposed method

• Construct two parallel networks for text and image: a CNN-based text model using pre-trained word2vec embeddings and a VGG16-based image model pre-trained on ImageNet.
• Extract high-level text features from a 5-layer CNN with multiple filter sizes and max-pooling, followed by dense layers.
• Extract image features from VGG16 penultimate fc2 features, with a modified final layer for task-specific softmax.
• Create a shared representation by concatenating 1,000-dimensional hidden representations from both modalities, followed by a dense layer and softmax for prediction.
• Train three settings: (i) text-only, (ii) image-only, and (iii) multimodal (text+image); evaluate on fixed test set.
• Use Adam optimizer, early stopping, and standard preprocessing for text and image data.

Experimental results

Research questions

• RQ1 Can a joint multimodal representation improve informativeness classification for crisis-related social media over unimodal approaches?
• RQ2 Can multimodal models improve humanitarian-category classification compared to text-only or image-only models?
• RQ3 What are the comparative gains of multimodal fusion across the CrisisMMD dataset for the two tasks?
• RQ4 What challenges arise when aligning text and image data with potentially conflicting signals in social media posts?

Key findings

• Multimodal model achieves F1=84.2 on informativeness, outperforming both text-only (80.9) and image-only (83.2).
• Multimodal model achieves F1=78.3 on humanitarian classification, outperforming text-only (67.7) and image-only (76.3).
• Image-only models generally outperform text-only models in both tasks, but multimodal fusion yields additional gains (≈1% in informativeness and ≈2% in humanitarian) over image-only baselines.
• Training on CrisisMMD provides baseline unimodal and multimodal results for both tasks within a single study.
• The authors note substantial room for improvement with larger, more optimized architectures and richer fusion strategies.