[Paper Review] xFBD: Focused Building Damage Dataset and Analysis

Dennis Melamed, Cameron Johnson+5 more|arXiv (Cornell University)|Dec 23, 2022

Infrastructure Maintenance and Monitoring4 citations

TL;DR

This paper introduces xFBD, a new augmented dataset and object-level evaluation metrics to assess building damage detection models in isolation, addressing limitations in the xView2 competition where models rely heavily on contextual cues. The study reveals that top xView2 models fail on isolated damage cases, and proposes new loss functions (4PS-Focal, 4PS-Contour) and baselines that improve object-level separability and localization, demonstrating that building damage detection remains an open challenge for fine-grained, context-independent analysis.

ABSTRACT

The xView2 competition and xBD dataset spurred significant advancements in overhead building damage detection, but the competition's pixel level scoring can lead to reduced solution performance in areas with tight clusters of buildings or uninformative context. We seek to advance automatic building damage assessment for disaster relief by proposing an auxiliary challenge to the original xView2 competition. This new challenge involves a new dataset and metrics indicating solution performance when damage is more local and limited than in xBD. Our challenge measures a network's ability to identify individual buildings and their damage level without excessive reliance on the buildings' surroundings. Methods that succeed on this challenge will provide more fine-grained, precise damage information than original xView2 solutions. The best-performing xView2 networks' performances dropped noticeably in our new limited/local damage detection task. The common causes of failure observed are that (1) building objects and their classifications are not separated well, and (2) when they are, the classification is strongly biased by surrounding buildings and other damage context. Thus, we release our augmented version of the dataset with additional object-level scoring metrics (https://drive.google.com/drive/folders/1VuQZuAg6-Yo8r5J4OCx3ZRpa_fv9aaDX?usp=sharing) to test independence and separability of building objects, alongside the pixel-level performance metrics of the original competition. We also experiment with new baseline models which improve independence and separability of building damage predictions. Our results indicate that building damage detection is not a fully-solved problem, and we invite others to use and build on our dataset augmentations and metrics.

Motivation & Objective

• To address the over-reliance of xView2 models on contextual information for damage classification, which undermines performance on isolated damaged buildings.
• To develop a new dataset augmentation method that isolates single damaged buildings in otherwise undamaged scenes using Poisson blending.
• To introduce object-level evaluation metrics that measure model performance on individual buildings rather than pixel-level averages.
• To evaluate whether existing top-performing xView2 models generalize poorly to isolated damage scenarios.
• To propose new loss functions (4PS-Focal, 4PS-Contour) that enhance object separability and boundary accuracy in building damage detection.

Proposed method

• Augment the xBD dataset by synthetically inserting a single damaged building into undamaged pre-disaster scenes using Poisson blending to create the xFBD dataset.
• Introduce object-level metrics to evaluate model performance on individual buildings, including localization F1 and damage classification F1 at the object level.
• Design a modified 4PS loss function with focal loss (L_R) and soft dice (L_SD) components to improve classification of isolated buildings.
• Implement a contour loss (L_Contour) that combines classification and dice loss to enhance boundary accuracy and object separability.
• Train and evaluate new baselines using Mask-RCNN and modified 4PS models on both xBD and xFBD to compare performance across metrics.
• Apply a filtering mechanism to exclude buildings whose damage is not visible from roof-level (e.g., flood-affected buildings) to ensure meaningful evaluation.

Experimental results

Research questions

• RQ1 How do top-performing xView2 models perform on isolated damaged buildings when context is removed?
• RQ2 To what extent do current models rely on surrounding context for damage classification rather than on the building's own features?
• RQ3 Can new loss functions (4PS-Focal, 4PS-Contour) improve the separability and accuracy of individual building damage predictions?
• RQ4 How do object-level metrics differ from pixel-level metrics in revealing model failures on isolated damage cases?
• RQ5 Can Mask-RCNN, as an object detection model, outperform instance segmentation-based models in isolated building damage detection?

Key findings

• The top five xView2 models performed poorly on the xFBD challenge, with object-level F1 scores below 0.05, indicating strong failure in detecting isolated damaged buildings.
• The primary failure modes were poor separation of building objects and strong bias in damage classification due to surrounding context, even when the building itself was clearly damaged.
• The 4PS-Focal and 4PS-Contour loss variants improved object-level F1 scores over the original 4PS baseline, with 4PS-Focal achieving 0.040 and 4PS-Contour 0.050 on the xFBD dataset.
• Mask-RCNN outperformed 4PS-based models on object-level localization, achieving 0.210 F1, suggesting inherent advantages in object-aware architectures.
• The object-level metrics revealed significant performance gaps not visible in pixel-level metrics, such as a high xView2 score of 0.887 despite missing the damaged building entirely.
• The study confirms that building damage detection is not fully solved and that context-independent, fine-grained analysis remains a critical open challenge.