[Paper Review] Primitive-based 3D Building Modeling, Sensor Simulation, and Estimation

Xia Li, Yen-Liang Lin+3 more|arXiv (Cornell University)|Jan 16, 2019

Remote Sensing and LiDAR Applications7 references4 citations

TL;DR

This paper proposes a synthetic data-driven approach for primitive-based 3D building modeling that formulates building decomposition as instance segmentation followed by primitive fitting. By generating diverse synthetic buildings with varied shapes, heights, and distortions, the method achieves high accuracy in reconstructing compact, interpretable 3D models from satellite imagery, reducing polygon counts by up to 99.9% while maintaining 90%+ F1-score on real-world WorldView-3 data.

ABSTRACT

As we begin to consider modeling large, realistic 3D building scenes, it becomes necessary to consider a more compact representation over the polygonal mesh model. Due to the large amounts of annotated training data, which is costly to obtain, we leverage synthetic data to train our system for the satellite image domain. By utilizing the synthetic data, we formulate the building decomposition as an application of instance segmentation and primitive fitting to decompose a building into a set of primitive shapes. Experimental results on WorldView-3 satellite image dataset demonstrate the effectiveness of our 3D building modeling approach.

Motivation & Objective

• To address the challenge of creating compact, interpretable 3D building models from remote sensing data.
• To overcome the limitations of polygonal mesh representations, which are verbose and lack semantic structure.
• To reduce reliance on costly real-world annotated training data for 3D building reconstruction.
• To develop a scalable, data-driven method for primitive-based 3D modeling using synthetic data generation and deep learning.
• To enable robust instance segmentation and primitive fitting for complex building shapes in satellite imagery.

Proposed method

• A synthetic building generation pipeline creates varied building footprints by recursively partitioning space into non-overlapping rectangles and assigning random heights and primitive types (e.g., rectangular prisms, triangular prisms, elliptical cylinders).
• Stereo simulation generates realistic height maps by adding Gaussian noise, boundary perturbations, and smoothing to mimic satellite stereo reconstruction artifacts (e.g., from s2p).
• A cascading instance segmentation network is trained on synthetic data to decompose buildings into sections, with data augmentation via random rotation (0° to 45°) to improve generalization.
• Each segmented section undergoes primitive classification and fitting using Coherent Point Drift (CPD) to align predicted primitives to 3D point clouds via rigid transformation (rotation, translation, scale).
• The best-fitting primitive is selected based on minimal fitting error, and the final 3D model is textured using orthographic color images.
• A correction step fills interior gaps between sections to improve decomposition continuity.

Experimental results

Research questions

• RQ1 Can synthetic data effectively train a deep learning model for 3D building decomposition without real-world annotations?
• RQ2 How well can instance segmentation and primitive fitting reconstruct complex building shapes from satellite-derived digital height models?
• RQ3 To what extent does primitive-based modeling reduce geometric complexity while preserving reconstruction accuracy?
• RQ4 How does the method perform across diverse urban environments with varying building densities and structures?
• RQ5 What are the key failure modes in decomposition and fitting, and how can they be mitigated?

Key findings

• The method achieved 93.9% 2D completeness, 97.8% 2D correctness, and 92.0% 2D Jaccard index on AOI-1, with performance above 90% on all four test areas.
• In 3D, the method achieved 93.7% completeness, 96.2% correctness, and 90.4% Jaccard index on AOI-1, demonstrating strong geometric fidelity.
• On average, the method reduced vertex count by 99.9% and face count by 99.9% compared to dense triangulated models, with only 994 vertices and 1,680 faces on average per building.
• The highest performance was observed in AOI-4 (Jacksonville, FL), with 96.0% 2D completeness and 96.0% 3D completeness, indicating robustness in low-density urban areas.
• The method outperformed baseline mesh-based models, which had over 7 million faces on average in AOI-4, while maintaining high reconstruction accuracy.
• Limitations were observed in stacked building structures and box decomposition errors, suggesting room for improvement in hierarchical and CSG-based modeling.