[Paper Review] Building3D: An Urban-Scale Dataset and Benchmarks for Learning Roof Structures from Point Clouds

Ruisheng Wang, Shangfeng Huang+1 more|arXiv (Cornell University)|Jul 21, 2023

Remote Sensing and LiDAR ApplicationsEnvironmental Science3 citations

TL;DR

This paper introduces Building3D, the first large-scale urban dataset with 161,910 buildings across 16 Estonian cities, including aerial LiDAR point clouds, 3D mesh models, and novel wireframe representations. It establishes supervised and self-supervised learning baselines for 3D roof reconstruction, demonstrating that self-supervised pre-training improves performance on limited labeled data, setting new benchmarks for urban-scale 3D modeling.

ABSTRACT

Urban modeling from LiDAR point clouds is an important topic in computer vision, computer graphics, photogrammetry and remote sensing. 3D city models have found a wide range of applications in smart cities, autonomous navigation, urban planning and mapping etc. However, existing datasets for 3D modeling mainly focus on common objects such as furniture or cars. Lack of building datasets has become a major obstacle for applying deep learning technology to specific domains such as urban modeling. In this paper, we present a urban-scale dataset consisting of more than 160 thousands buildings along with corresponding point clouds, mesh and wire-frame models, covering 16 cities in Estonia about 998 Km2. We extensively evaluate performance of state-of-the-art algorithms including handcrafted and deep feature based methods. Experimental results indicate that Building3D has challenges of high intra-class variance, data imbalance and large-scale noises. The Building3D is the first and largest urban-scale building modeling benchmark, allowing a comparison of supervised and self-supervised learning methods. We believe that our Building3D will facilitate future research on urban modeling, aerial path planning, mesh simplification, and semantic/part segmentation etc.

Motivation & Objective

• To address the lack of large-scale, real-world 3D building datasets for urban modeling, especially for roof structure reconstruction from LiDAR point clouds.
• To provide a comprehensive benchmark for evaluating both supervised and self-supervised deep learning methods on real urban scenes.
• To introduce wireframe models as a novel, compact representation for polyhedral buildings, enabling efficient modeling and downstream tasks.
• To evaluate the performance of state-of-the-art deep learning and handcrafted feature-based methods on a challenging, large-scale dataset with high noise, data imbalance, and intra-class variance.
• To enable future research in urban modeling, aerial path planning, mesh simplification, and semantic/part segmentation through a publicly available, high-quality dataset.

Proposed method

• The dataset comprises 875.39 million aerial LiDAR points and 161,910 3D building models across 16 cities in Estonia, covering ~998 km².
• Mesh models are generated from point clouds using standard 3D reconstruction pipelines, and then converted into wireframe models by extracting vertices and edges.
• A supervised baseline is established using a point cloud-based deep learning model for roof structure prediction, trained on full annotations.
• A self-supervised baseline is introduced by pre-training a model on masked autoencoding (e.g., Point-MAE) on unlabeled point clouds before fine-tuning on limited labeled data.
• The evaluation framework includes metrics such as RMSE, IoU, and Edge Recall (ER), with ablation studies on data ratios (1% to 80%) to assess generalization under data scarcity.
• Traditional methods like 2.5D Dual Contouring, PolyFit, and Topology Aware Modeling are evaluated for comparison, with parameter tuning by experts.

Experimental results

Research questions

• RQ1 How do self-supervised learning methods perform on 3D roof reconstruction from large-scale, noisy, and incomplete LiDAR point clouds?
• RQ2 Can wireframe models serve as an effective and efficient representation for 3D building reconstruction, especially in comparison to mesh models?
• RQ3 What is the performance gap between deep learning-based methods and traditional handcrafted feature-based methods on real-world urban point clouds?
• RQ4 How does model performance vary with limited labeled data, and can self-supervised pre-training mitigate data scarcity in urban modeling?
• RQ5 To what extent do challenges like high intra-class variance, data imbalance, and noise affect the generalization of 3D reconstruction models?

Key findings

• The Building3D dataset is the first and largest urban-scale benchmark for 3D roof modeling, covering 161,910 buildings and 875.39 million LiDAR points across 16 cities in Estonia.
• Self-supervised pre-training significantly improves performance on limited labeled data, outperforming Point-MAE at the same data ratios (e.g., 1%, 10%, 20%, 50%, 80%).
• Despite lower RMSE and IoU than traditional methods, the deep learning model produces smoother, lower-face-count meshes due to the wireframe ground truth guiding the optimization.
• Handcrafted methods like PolyFit and 2.5D Dual Contouring require dense, complete point clouds and expert tuning, while deep learning methods work on open, incomplete point clouds without parameter tuning.
• All methods struggle with Edge Recall (ER), indicating persistent challenges in detecting edges and corners due to data incompleteness and noise in real-world LiDAR scans.
• The dataset enables diverse downstream applications, including semantic segmentation, mesh simplification, footprint detection, and aerial path planning, due to its rich annotations and multi-level representations.