[Paper Review] Deep Learning Guided Building Reconstruction from Satellite Imagery-derived Point Clouds

Bo Xu, Xu Zhang+4 more|arXiv (Cornell University)|May 19, 2020

Remote Sensing and LiDAR Applications32 references4 citations

TL;DR

This paper presents a deep learning-guided, end-to-end framework for robust 3D building reconstruction from noisy, satellite-derived point clouds. It uses PointNet for roof shape segmentation and a multi-cue hierarchical RANSAC method to extract planar, cylindrical, and spherical primitives, achieving 83.0% roof shape recognition accuracy and over 70% completeness and correctness compared to airborne LiDAR ground truth across four urban areas.

ABSTRACT

3D urban reconstruction of buildings from remotely sensed imagery has drawn significant attention during the past two decades. While aerial imagery and LiDAR provide higher resolution, satellite imagery is cheaper and more efficient to acquire for large scale need. However, the high, orbital altitude of satellite observation brings intrinsic challenges, like unpredictable atmospheric effect, multi view angles, significant radiometric differences due to the necessary multiple views, diverse land covers and urban structures in a scene, small base-height ratio or narrow field of view, all of which may degrade 3D reconstruction quality. To address these major challenges, we present a reliable and effective approach for building model reconstruction from the point clouds generated from multi-view satellite images. We utilize multiple types of primitive shapes to fit the input point cloud. Specifically, a deep-learning approach is adopted to distinguish the shape of building roofs in complex and yet noisy scenes. For points that belong to the same roof shape, a multi-cue, hierarchical RANSAC approach is proposed for efficient and reliable segmenting and reconstructing the building point cloud. Experimental results over four selected urban areas (0.34 to 2.04 sq km in size) demonstrate the proposed method can generate detailed roof structures under noisy data environments. The average successful rate for building shape recognition is 83.0%, while the overall completeness and correctness are over 70% with reference to ground truth created from airborne lidar. As the first effort to address the public need of large scale city model generation, the development is deployed as open source software.

Motivation & Objective

• Address the challenge of reconstructing detailed 3D building models from low-quality, noisy point clouds generated by multi-view satellite imagery.
• Overcome limitations of existing methods designed for high-resolution aerial or LiDAR data when applied to satellite-derived point clouds.
• Enable large-scale, automated 3D urban modeling by integrating deep learning with traditional geometric segmentation techniques.
• Improve reconstruction robustness in complex urban scenes with diverse roof types, noise, and data voids.
• Develop a publicly available, open-source software pipeline for scalable city modeling from satellite data.

Proposed method

• Employ a PointNet-based deep learning model to classify roof shapes directly from raw point clouds, using a novel data synthesis method to reduce reliance on manual labeling.
• Apply moving least squares and median filtering to suppress structured noise and fill voids in satellite-derived point clouds before segmentation.
• Implement a multi-cue hierarchical RANSAC algorithm that leverages both surface normals and image colors to robustly detect planar, cylindrical, and spherical roof primitives.
• Use a hierarchical RANSAC strategy to accelerate computation and improve segmentation reliability by prioritizing simpler shapes first.
• Perform boundary regularization and roof topology reconstruction to generate coherent, closed 3D building models from segmented primitives.
• Integrate the entire pipeline into an end-to-end, automated system deployable on cloud platforms like Amazon Web Services.

Experimental results

Research questions

• RQ1 Can deep learning effectively classify complex roof shapes in noisy, satellite-derived point clouds where traditional methods fail?
• RQ2 How can geometric segmentation be made robust to the high noise and data sparsity inherent in satellite point clouds?
• RQ3 To what extent can a multi-cue hierarchical RANSAC approach improve the accuracy and efficiency of primitive detection compared to standard RANSAC?
• RQ4 Can the proposed method achieve high-quality 3D building reconstruction at scale without relying on high-resolution aerial or LiDAR data?
• RQ5 How does the performance of the end-to-end pipeline compare to airborne LiDAR ground truth in terms of completeness, correctness, and IoU?

Key findings

• The proposed method achieved an average roof shape recognition rate of 83.0% across four urban test areas, demonstrating strong generalization on complex scenes.
• For two larger test areas (0.96 and 2.04 km²), the overall completeness and correctness of the reconstructed models exceeded 70%, with 3D IoU scores between 70% and 75% relative to airborne LiDAR ground truth.
• The method successfully reconstructed complex roof types, including spherical and curved shapes, as validated in AOI 4, where 4 out of 6 spherical roofs were accurately captured.
• The absence of cylindrical and spherical models in the segmentation pipeline led to cracked and incomplete reconstructions, highlighting the importance of shape diversity in the primitive library.
• The multi-cue hierarchical RANSAC significantly improved segmentation reliability and reduced computation time compared to standard RANSAC, especially in noisy environments.
• The open-source implementation of the pipeline enables large-scale deployment and integration into cloud-based urban modeling services, fulfilling the goal of scalable, automated 3D city modeling.