[论文解读] Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises

Dat Tien Nguyen, Firoj Alam+2 more|arXiv (Cornell University)|Apr 9, 2017

Advanced Image and Video Retrieval Techniques参考文献 1被引用 59

一句话总结

本文提出一种实时图像过滤流水线，使用迁移学习的 CNN 进行相关性过滤，并用感知哈希进行去重，以提高危机相关图像数据质量和标注效率。

ABSTRACT

The extensive use of social media platforms, especially during disasters, creates unique opportunities for humanitarian organizations to gain situational awareness and launch relief operations accordingly. In addition to the textual content, people post overwhelming amounts of imagery data on social networks within minutes of a disaster hit. Studies point to the importance of this online imagery content for emergency response. Despite recent advances in the computer vision field, automatic processing of the crisis-related social media imagery data remains a challenging task. It is because a majority of which consists of redundant and irrelevant content. In this paper, we present an image processing pipeline that comprises de-duplication and relevancy filtering mechanisms to collect and filter social media image content in real-time during a crisis event. Results obtained from extensive experiments on real-world crisis datasets demonstrate the significance of the proposed pipeline for optimal utilization of both human and machine computing resources.

研究动机与目标

• Purify noisy social media image data by removing irrelevant content.
• Eliminate duplicate and near-duplicate images to reduce data redundancy.
• Demonstrate that filtering improves annotation budgets and machine learning robustness.
• Adapt state-of-the-art deep learning models to relevancy and damage classification tasks in crisis data.
• Develop a real-time pipeline for analyzing social media images at crisis onset.

提出的方法

• Build an automatic image filtering pipeline with a Tweet Collector and Image Collector.
• Use relevancy filtering by fine-tuning a pre-trained VGG-16 CNN on a binary relevant/irrelevant task for damage assessment relevance.
• Apply perceptual hashing (pHash) to detect exact and near-duplicate images and keep a rolling 100k hash window.
• Tune a Hamming-distance threshold (d) for de-duplication by manual inspection on 1,100 image pairs, selecting d = 10.
• Assess impact on four real-world disaster datasets (Nepal Earthquake, Ecuador Earthquake, Typhoon Ruby, Hurricane Matthew) using 60/20/20 train/validation/test splits for relevancy and 5-fold cross-validation for damage classification.
• Evaluate using accuracy, precision, recall, F1, and AUC.

实验结果

研究问题

• RQ1 How effectively can relevancy filtering distinguish images that are informative for damage assessment from irrelevant images?
• RQ2 How does de-duplication using perceptual hashing impact data volume and the quality of downstream damage classification models?
• RQ3 What is the effect of image filtering on the accuracy and robustness of a damage assessment classifier trained on crisis-related images?
• RQ4 What data reduction is achieved by applying relevancy filtering and de-duplication in real-time crisis scenarios?

主要发现

• Relevancy filter achieves high discriminative performance with AUC 0.98, precision 0.99, recall 0.97, and F1 0.98 on the test set.
• De-duplication removes 58% of severe, 50% of mild, and 30% of none images, yielding an overall reduction of 62% in the raw collection.
• Using duplicates and irrelevant images inflates budget waste; removing duplicates saved about 1,178 labeled images worth roughly 20% of the budget in the experiment.
• In the damage classification task, removing duplicates (S2) improves reliability and avoids artificial boosts from duplicate leakage; removing both duplicates and irrelevant images (S4) yields a ~2% macro F1 improvement over S2.
• The three-class damage classifier (severe, mild, none) remains hardest for the mild category due to class imbalance and lower prevalence.
• A real-time pipeline and web-accessible system for crisis image analysis is demonstrated.