[Paper Review] Change detection needs change information: improving deep 3D point cloud change detection

Iris de Gélis, Thomas Corpetti+1 more|arXiv (Cornell University)|Apr 25, 2023

Remote Sensing and LiDAR Applications41 references4 citations

TL;DR

This paper proposes enhancing deep 3D point cloud change detection by integrating change information early in the network architecture, using hand-crafted features and novel Siamese-based models. The best architecture, Encoder Fusion SiamKPConv, improves mean Intersection over Union (mIoU) over change classes by over 5% compared to state-of-the-art methods, demonstrating that early fusion of change-related features significantly boosts segmentation accuracy.

ABSTRACT

Change detection is an important task that rapidly identifies modified areas, particularly when multi-temporal data are concerned. In landscapes with a complex geometry (e.g., urban environment), vertical information is a very useful source of knowledge that highlights changes and classifies them into different categories. In this study, we focus on change segmentation using raw three-dimensional (3D) point clouds (PCs) directly to avoid any information loss due to the rasterization processes. While deep learning has recently proven its effectiveness for this particular task by encoding the information through Siamese networks, we investigate herein the idea of also using change information in the early steps of deep networks. To do this, we first propose to provide a Siamese KPConv state-of-the-art (SoTA) network with hand-crafted features, especially a change-related one, which improves the mean of the Intersection over Union (IoU) over the classes of change by 4.70%. Considering that a major improvement is obtained due to the change-related feature, we then propose three new architectures to address 3D PC change segmentation: OneConvFusion, Triplet KPConv, and Encoder Fusion SiamKPConv. All these networks consider the change information in the early steps and outperform the SoTA methods. In particular, Encoder Fusion SiamKPConv overtakes the SoTA approaches by more than 5% of the mean of the IoU over the classes of change, emphasizing the value of having the network focus on change information for the change detection task. The code is available at https://github.com/IdeGelis/torch-points3d-SiamKPConvVariants.

Motivation & Objective

• To improve 3D point cloud change detection by incorporating change-related information earlier in the network, rather than only in the decoder.
• To address the limitation of existing methods that rely on rasterized data or late fusion of change features, which can lead to information loss.
• To design end-to-end deep learning architectures that directly process raw 3D point clouds and encode change features at multiple scales.
• To evaluate whether explicitly modeling change information through hand-crafted features or learned architectures enhances segmentation performance.
• To establish a new state-of-the-art for 3D point cloud change segmentation using raw point clouds without rasterization.

Proposed method

• Proposes adding a hand-crafted change-related feature (e.g., stability metric) as input to the Siamese KPConv network to guide early feature learning.
• Introduces three new architectures: OneConvFusion, Triplet KPConv, and Encoder Fusion SiamKPConv, all designed to fuse mono-date and change-related features early in the network.
• Employs multi-scale convolutional operations on the difference of nearest-neighbor features to generate change-related representations directly in the encoder.
• Uses Siamese KPConv as a backbone, with modified fusion mechanisms that combine features from both temporal inputs at multiple levels.
• Applies skip connections and multi-scale fusion to preserve spatial details and improve feature discrimination.
• Trains models end-to-end on raw 3D point clouds using the Urb3DCD and AHN-CD datasets, optimizing for mean IoU over change classes.

Experimental results

Research questions

• RQ1 Can early integration of change-related features improve 3D point cloud change detection performance?
• RQ2 How does the architecture design influence the network's ability to detect and segment changes in raw 3D point clouds?
• RQ3 Does learning change features through convolution on feature differences outperform hand-crafted features?
• RQ4 To what extent does early fusion of change and mono-date features improve segmentation accuracy compared to late fusion?
• RQ5 Can a custom architecture that inherently encodes change information surpass the performance of a standard Siamese network with added hand-crafted features?

Key findings

• Adding a single hand-crafted change-related feature (e.g., stability) to the Siamese KPConv network increases the mean IoU over change classes by 3.70%.
• The Encoder Fusion SiamKPConv architecture achieves a mIoU improvement of over 5% compared to the state-of-the-art, outperforming existing methods on both Urb3DCD and AHN-CD datasets.
• The OneConvFusion network underperforms due to the lack of multi-scale mono-date features, confirming the importance of preserving original data representations.
• The Triplet KPConv and Encoder Fusion SiamKPConv architectures achieve performance comparable to or better than the Siamese KPConv with 10 hand-crafted features, indicating that architectural design can replace manual feature engineering.
• The Encoder Fusion SiamKPConv model renders the hand-crafted stability feature redundant, as it learns effective change representations internally.
• The results confirm that early encoding of change information significantly enhances feature learning and segmentation accuracy in 3D point cloud change detection.