[Paper Review] Semantic Change Detection with Hypermaps

Suzuki, Teppei, Soma Shirakabe+4 more|arXiv (Cornell University)|Apr 26, 2016

Remote-Sensing Image Classification16 references17 citations

TL;DR

This paper introduces semantic change detection (SCD), a novel approach that enhances traditional change detection by integrating semantic meaning into detected change regions. By proposing multi-scale weighted hypermaps—improved hypercolumns using region-based, multi-scale CNN features—it achieves state-of-the-art performance of 71.18% and 66.44% on the re-annotated TSUNAMI dataset for t₀ and t₁, respectively, significantly outperforming baseline methods including standard hypercolumns.

ABSTRACT

Change detection is the study of detecting changes between two different images of a scene taken at different times. By the detected change areas, however, a human cannot understand how different the two images. Therefore, a semantic understanding is required in the change detection research such as disaster investigation. The paper proposes the concept of semantic change detection, which involves intuitively inserting semantic meaning into detected change areas. We mainly focus on the novel semantic segmentation in addition to a conventional change detection approach. In order to solve this problem and obtain a high-level of performance, we propose an improvement to the hypercolumns representation, hereafter known as hypermaps, which effectively uses convolutional maps obtained from convolutional neural networks (CNNs). We also employ multi-scale feature representation captured by different image patches. We applied our method to the TSUNAMI Panoramic Change Detection dataset, and re-annotated the changed areas of the dataset via semantic classes. The results show that our multi-scale hypermaps provided outstanding performance on the re-annotated TSUNAMI dataset.

Motivation & Objective

• To address the limitation of conventional change detection, which identifies 'where' changes occur but not 'how' they differ semantically, especially in disaster scenarios.
• To improve semantic understanding of changed areas by integrating semantic segmentation with change detection, enabling intuitive interpretation of changes.
• To enhance feature representation for change detection by extending hypercolumns into region-based, multi-scale, and weighted hypermaps using pre-trained CNNs.
• To evaluate the proposed method on a re-annotated version of the TSUNAMI panoramic change detection dataset, where changed regions are labeled with semantic classes.
• To demonstrate that multi-scale and weighted feature aggregation significantly improves performance in semantic change detection tasks.

Proposed method

• The method extends hypercolumns by aggregating region-based feature maps instead of pixel-wise features, creating a more robust representation for semantic change detection.
• It employs multi-scale image patches (e.g., 30×30) to capture contextual and hierarchical features, improving generalization across varying spatial scales.
• A weighted fusion strategy using Gaussian kernels is applied to combine features from different scales, enhancing the representation of salient change regions.
• The approach uses pre-trained DeCAF features from ImageNet to enable transfer learning, leveraging rich, data-driven features over hand-crafted ones like SIFT or HOG.
• The final representation, termed 'multi-scale weighted hypermaps,' is used as input to a classifier for semantic change detection, replacing conventional pixel-wise hypercolumns.
• The method is evaluated on a re-annotated version of the TSUNAMI dataset, where changed areas are labeled with semantic classes such as 'building,' 'rubble,' and 'other'.

Experimental results

Research questions

• RQ1 Can multi-scale feature representation improve the performance of semantic change detection in disaster imagery?
• RQ2 Does region-based aggregation of CNN features (via hypermaps) outperform pixel-wise hypercolumns in capturing semantic changes?
• RQ3 How effective is the weighted fusion of multi-scale features in enhancing the detection of complex changes like rubble or structural damage?
• RQ4 To what extent does transfer learning from pre-trained CNNs (e.g., DeCAF) improve semantic understanding of changed regions compared to hand-crafted features?
• RQ5 Can the proposed method achieve superior performance on a re-annotated dataset with semantic labels for changed areas?

Key findings

• The proposed multi-scale weighted hypermaps achieved 71.18% and 66.44% F1-scores on the t₀ and t₁ splits of the re-annotated TSUNAMI dataset, respectively, outperforming standard hypercolumns.
• The method improved performance by +4.64% on t₀ and +3.54% on t₁ compared to the baseline hypercolumn representation.
• Multi-scale representation alone improved performance by +2.38% on t₀ and +13.83% on t₁, indicating strong benefits for complex, rubble-laden scenes.
• The DeCAF feature representation significantly outperformed hand-crafted features like SIFT+BoW and HOG, demonstrating the advantage of deep learning-based features.
• The weighted fusion of multi-scale features with Gaussian kernels enhanced feature representation, contributing to +4.65% and +3.03% gains on t₀ and t₁ over standard hypercolumns.
• The method demonstrated robustness in capturing semantic meaning across diverse change types, including structural damage and debris, as shown in qualitative comparisons with ground truth.