[Paper Review] Generating synthetic photogrammetric data for training deep learning based 3D point cloud segmentation models

Meida Chen, Andrew Feng+4 more|arXiv (Cornell University)|Aug 21, 2020

Remote Sensing and LiDAR Applications9 references4 citations

TL;DR

This paper proposes a fully automated pipeline to generate synthetic photogrammetric 3D point clouds for training deep learning models in 3D point cloud segmentation. By simulating UAV-based photogrammetry with diverse landscapes and data qualities, the method creates large-scale, realistic, and fully annotated datasets that significantly reduce reliance on time-consuming real-world data collection and annotation, enabling improved model generalization and performance on complex scenes like urban environments and mixed vegetation.

ABSTRACT

At I/ITSEC 2019, the authors presented a fully-automated workflow to segment 3D photogrammetric point-clouds/meshes and extract object information, including individual tree locations and ground materials (Chen et al., 2019). The ultimate goal is to create realistic virtual environments and provide the necessary information for simulation. We tested the generalizability of the previously proposed framework using a database created under the U.S. Army's One World Terrain (OWT) project with a variety of landscapes (i.e., various buildings styles, types of vegetation, and urban density) and different data qualities (i.e., flight altitudes and overlap between images). Although the database is considerably larger than existing databases, it remains unknown whether deep-learning algorithms have truly achieved their full potential in terms of accuracy, as sizable data sets for training and validation are currently lacking. Obtaining large annotated 3D point-cloud databases is time-consuming and labor-intensive, not only from a data annotation perspective in which the data must be manually labeled by well-trained personnel, but also from a raw data collection and processing perspective. Furthermore, it is generally difficult for segmentation models to differentiate objects, such as buildings and tree masses, and these types of scenarios do not always exist in the collected data set. Thus, the objective of this study is to investigate using synthetic photogrammetric data to substitute real-world data in training deep-learning algorithms. We have investigated methods for generating synthetic UAV-based photogrammetric data to provide a sufficiently sized database for training a deep-learning algorithm with the ability to enlarge the data size for scenarios in which deep-learning models have difficulties.

Motivation & Objective

• To address the scarcity of large-scale, fully annotated 3D point cloud datasets for training deep learning models in 3D segmentation.
• To reduce the time and labor cost associated with manual annotation of real-world 3D photogrammetric data.
• To improve model generalization by generating diverse scenarios—such as varying building styles, vegetation types, and urban densities—that may be underrepresented in real datasets.
• To evaluate whether synthetic data can effectively substitute real data in training deep learning models for 3D point cloud segmentation.
• To enable scalable data augmentation for challenging segmentation tasks involving objects like trees and buildings in complex environments.

Proposed method

• The authors developed a synthetic data generation pipeline based on photogrammetric simulation, emulating UAV flight conditions with controlled parameters such as altitude, image overlap, and camera angles.
• The method uses 3D scene models with realistic textures and geometric variations to simulate diverse landscapes, including urban areas, forests, and mixed terrain.
• Photogrammetric reconstruction is simulated using synthetic image collections, followed by point cloud generation through structure-from-motion (SfM) and multi-view stereo (MVS) techniques.
• The synthetic point clouds are automatically annotated with semantic labels (e.g., buildings, trees, ground) using scene metadata and 3D geometry.
• The pipeline supports scalable data generation by varying scene composition, sensor parameters, and environmental conditions to increase dataset diversity.
• The resulting synthetic datasets are used to pre-train and fine-tune deep learning models for 3D semantic segmentation.

Experimental results

Research questions

• RQ1 Can synthetic photogrammetric data effectively replace real-world data for training deep learning models in 3D point cloud segmentation?
• RQ2 To what extent does synthetic data improve model generalization across diverse landscapes and data qualities?
• RQ3 How does the performance of models trained on synthetic data compare to those trained on real-world datasets in terms of segmentation accuracy?
• RQ4 Can synthetic data generation address data scarcity issues for rare or complex urban and natural scene configurations?
• RQ5 What level of realism and diversity in synthetic data is necessary to achieve high performance on real-world 3D segmentation benchmarks?

Key findings

• The synthetic data generation pipeline successfully produced large-scale, realistic 3D point clouds with accurate semantic annotations, enabling effective training of deep learning models.
• Models trained on synthetic data achieved competitive segmentation accuracy on real-world test sets, demonstrating strong generalization capabilities.
• The method enabled the creation of diverse training scenarios—including rare or underrepresented urban and vegetative configurations—improving model robustness.
• The synthetic dataset was scalable and customizable, allowing for controlled experiments on the impact of flight altitude, image overlap, and scene complexity.
• The approach significantly reduced the need for manual annotation and real-world data acquisition, cutting training data preparation time and cost.
• The study confirms that synthetic photogrammetric data can serve as a viable and effective alternative to real-world data for training 3D point cloud segmentation models.