[Paper Review] Time Travelling Pixels: Bitemporal Features Integration with Foundation Model for Remote Sensing Image Change Detection

Keyan Chen, Chengyang Liu+6 more|arXiv (Cornell University)|Dec 23, 2023

Remote-Sensing Image Classification4 citations

TL;DR

This paper proposes Time Travelling Pixels (TTP), a novel method that integrates the Segment Anything Model (SAM) foundation model into remote sensing change detection by leveraging low-rank fine-tuning and a time-traveling activation gate to model bitemporal image differences. TTP achieves state-of-the-art performance on LEVIR-CD, attaining 92.1% F1 and 85.6% IoU, demonstrating superior generalization and robustness in complex spatiotemporal scenarios.

ABSTRACT

Change detection, a prominent research area in remote sensing, is pivotal in observing and analyzing surface transformations. Despite significant advancements achieved through deep learning-based methods, executing high-precision change detection in spatio-temporally complex remote sensing scenarios still presents a substantial challenge. The recent emergence of foundation models, with their powerful universality and generalization capabilities, offers potential solutions. However, bridging the gap of data and tasks remains a significant obstacle. In this paper, we introduce Time Travelling Pixels (TTP), a novel approach that integrates the latent knowledge of the SAM foundation model into change detection. This method effectively addresses the domain shift in general knowledge transfer and the challenge of expressing homogeneous and heterogeneous characteristics of multi-temporal images. The state-of-the-art results obtained on the LEVIR-CD underscore the efficacy of the TTP. The Code is available at \url{https://kychen.me/TTP}.

Motivation & Objective

• Address the challenge of limited annotated data and domain shift in remote sensing change detection by transferring general knowledge from foundation models.
• Overcome the limitation of existing foundation models in capturing both spatial homogeneity and temporal heterogeneity across bitemporal remote sensing images.
• Enhance model generalization in complex spatiotemporal environments where traditional deep learning models struggle due to data scarcity and domain shift.
• Develop an efficient, parameter-efficient method that preserves the semantic understanding of foundation models while adapting them to change detection tasks.
• Enable high-precision, dense pixel-level change detection by integrating temporal modeling into the semantic feature space of a vision foundation model.

Proposed method

• Utilizes the SAM (Segment Anything Model) as a visual backbone to transfer general-purpose image understanding capabilities to remote sensing change detection.
• Applies low-rank fine-tuning to the SAM encoder to adapt its spatial semantics to remote sensing imagery, mitigating domain shift without fine-tuning the full backbone.
• Introduces a time-traveling activation gate that enables temporal feature interaction by allowing features from one time point to influence the semantic representation of the other, enhancing bitemporal modeling.
• Employs a multi-level, lightweight decoding head to predict dense change maps from high-level features, preserving spatial resolution and detail.
• Freezes the SAM backbone during training and only trains the low-rank adapter and the time-traveling gate, ensuring parameter efficiency and fast convergence.
• Uses data augmentation (rotation, flipping, cropping, photometric distortion) and a cosine annealing scheduler with linear warmup to improve training stability and generalization.

Experimental results

Research questions

• RQ1 Can the general knowledge of a vision foundation model like SAM be effectively transferred to remote sensing change detection despite domain shift?
• RQ2 How can a foundation model be adapted to capture both homogeneous (unchanged) and heterogeneous (changed) regions across bitemporal remote sensing images?
• RQ3 What architectural components are necessary to enable temporal modeling within a foundation model’s semantic feature space?
• RQ4 Does low-rank fine-tuning effectively reduce domain shift while preserving the model’s generalization capacity in remote sensing applications?
• RQ5 Can a lightweight, efficient decoding head maintain high performance without requiring full fine-tuning of the foundation model?

Key findings

• TTP achieves state-of-the-art performance on the LEVIR-CD dataset with a 92.1% F1 score and 85.6% IoU, outperforming previous SOTA methods like WNet (90.7% F1, 82.9% IoU) and CSTSUNet (90.7% F1, 83.0% IoU).
• Ablation studies confirm that removing the time-traveling gate (ttg) reduces F1 to 91.1% and IoU to 84.2%, demonstrating its critical role in modeling temporal differences.
• Removing the multi-level decoding head (ml) further degrades performance to 90.6% F1 and 82.8% IoU, indicating its importance in preserving spatial detail.
• Eliminating low-rank fine-tuning leads to a drastic performance drop to 74.6% F1 and 59.5% IoU, proving its necessity for bridging the domain gap between natural and remote sensing images.
• The full TTP model achieves 99.2% Overall Accuracy, indicating high consistency in both changed and unchanged region classification.
• The model maintains strong performance with only fine-tuning a small number of parameters, confirming its efficiency and scalability in data-scarce remote sensing settings.