[Paper Review] DeepDamageNet: A two-step deep-learning model for multi-disaster building damage segmentation and classification using satellite imagery

Irene Alisjahbana, Jiawei Li+3 more|arXiv (Cornell University)|May 8, 2024

Remote Sensing and Land Use9 citations

TL;DR

The paper proposes end-to-end and two-step deep-learning models for simultaneous building segmentation and damage classification from pre- and post-disaster satellite imagery, with the two-step model (semantic segmentation + damage classifier) achieving the best results and surpassing the xView2 baseline.

ABSTRACT

Satellite imagery has played an increasingly important role in post-disaster building damage assessment. Unfortunately, current methods still rely on manual visual interpretation, which is often time-consuming and can cause very low accuracy. To address the limitations of manual interpretation, there has been a significant increase in efforts to automate the process. We present a solution that performs the two most important tasks in building damage assessment, segmentation and classification, through deep-learning models. We show our results submitted as part of the xView2 Challenge, a competition to design better models for identifying buildings and their damage level after exposure to multiple kinds of natural disasters. Our best model couples a building identification semantic segmentation convolutional neural network (CNN) to a building damage classification CNN, with a combined F1 score of 0.66, surpassing the xView2 challenge baseline F1 score of 0.28. We find that though our model was able to identify buildings with relatively high accuracy, building damage classification across various disaster types is a difficult task due to the visual similarity between different damage levels and different damage distribution between disaster types, highlighting the fact that it may be important to have a probabilistic prior estimate regarding disaster damage in order to obtain accurate predictions.

Motivation & Objective

• Address rapid post-disaster building damage assessment via automated methods instead of manual interpretation.
• Develop models capable of handling multiple disaster types (earthquakes, floods, hurricanes, etc.).
• Compare end-to-end versus two-step architectures for segmentation and damage classification.
• Evaluate preprocessing, segmentation, and classification strategies on the xBD/xView2 dataset.
• Investigate how disaster-type information and additional features influence classification performance.

Proposed method

• Evaluate end-to-end and two-step architectures for joint segmentation and damage classification.
• Use U-Net with ResNet-34 encoder for end-to-end model with pre/post-disaster image difference as input.
• Develop two-step models: (a) instance/semantic segmentation to extract building footprints, (b) per-building damage classification using pre- and post-disaster crops and various feature fusion schemes.
• Instance segmentation with Mask R-CNN and semantic segmentation with ResNet-50 + FPN for building footprints.
• Damage classification using a twin-tower CNN that fuses pre- and post-disaster features; compare simple pre/post concatenation, disaster-type conditioning, and SSIM-based features.
• Train and evaluate on the xBD/xView2 dataset with metrics including mIoU and weighted F1, and report a final combined F1 of the challenge.

Figure 1: Pre-disaster (left) and its corresponding post-disaster satellite image

Experimental results

Research questions

• RQ1 Can a two-step segmentation + classification pipeline outperform an end-to-end model for multi-disaster damage assessment?
• RQ2 Does incorporating disaster-type information as an auxiliary feature improve damage classification accuracy across disaster types?
• RQ3 Do hand-engineered features like SSIM provide measurable gains in damage classification?
• RQ4 How does semantic segmentation-based footprint extraction compare to instance segmentation in this context?
• RQ5 To what extent does cross-disaster generalization affect damage level classification?

Key findings

• The end-to-end model achieved 0.79584 segmentation F1 on validation but 0.23875 overall F1; classification results were not available for the end-to-end setup on the test set.
• Semantic segmentation outperformed instance segmentation for building footprint extraction (mIoU: 0.85 vs 0.70 on validation).
• Two-step models with semantic segmentation plus damage classification significantly outperformed the xView2 baseline; the best was Semantic Segmentation + Pre & Post Concatenation + Disaster with an overall F1 of 0.6641.
• Baseline xView2 results (for reference) showed Segmentation F1 0.80482, Classification F1 0.06091, Overall F1 0.28408.
• Disaster-type conditioning improved classification performance (Pre & Post Concatenation + Disaster achieved higher validation accuracy than the basic concatenation).
• Adding SSIM as an additional feature improved performance over the basic pre/post concatenation but was still below disaster-conditioned models.

Figure 2: Distribution of building damage level in training set