[Paper Review] Deep Learning from Parametrically Generated Virtual Buildings for Real-World Object Recognition

Mohammad Alawadhi, Wei Yan|arXiv (Cornell University)|Jan 3, 2023

3D Surveying and Cultural Heritage4 citations

TL;DR

This paper proposes training deep neural networks exclusively on realistically rendered, parametrically generated virtual buildings from BIM models using the BIMGenE framework. The approach achieves strong generalization and accurate semantic segmentation on real-world building photos, even for unseen structures, demonstrating the viability of synthetic data for real-world architectural AI tasks.

ABSTRACT

We study the use of parametric building information modeling (BIM) to automatically generate training data for artificial neural networks (ANNs) to recognize building objects in photos. Teaching artificial intelligence (AI) machines to detect building objects in images is the foundation toward AI-assisted semantic 3D reconstruction of existing buildings. However, there exists the challenge of acquiring training data which is typically human-annotated, that is, unless a computer machine can generate high-quality data to train itself for a certain task. In that vein, we trained ANNs solely on realistic computer-generated images of 3D BIM models which were parametrically and automatically generated using the BIMGenE program. The ANN training result demonstrated generalizability and good semantic segmentation on a test case as well as arbitrary photos of buildings that are outside the range of the training data, which is significant for the future of training AI with generated data for solving real-world architectural problems.

Motivation & Objective

• To address the scarcity of annotated training data for building object recognition in real-world images.
• To explore whether synthetic, parametrically generated 3D building models can effectively train deep neural networks for real-world applications.
• To evaluate the generalization capability of models trained solely on synthetic data when tested on diverse, real-world building photos.
• To establish a scalable, automated pipeline for generating high-quality training data for AI in architectural and construction domains.

Proposed method

• Utilized the BIMGenE tool to automatically generate a diverse set of parametric 3D building models with controlled variations in geometry, materials, and layouts.
• Rendered realistic RGB images and corresponding semantic segmentation masks from the 3D BIM models using photorealistic rendering techniques.
• Trained a convolutional neural network (CNN) on the synthetic dataset using standard semantic segmentation loss functions.
• Applied data augmentation and domain adaptation strategies implicitly through the diversity of the parametrically generated training distribution.
• Evaluated the trained model on real-world building photos not seen during training, including structures outside the training distribution.

Experimental results

Research questions

• RQ1 Can deep neural networks trained exclusively on synthetic, parametrically generated virtual buildings achieve accurate object recognition in real-world images?
• RQ2 How well does the model generalize to real-world building photos that differ in style, structure, and appearance from the training data?
• RQ3 To what extent can parametric BIM generation replace or reduce reliance on human-annotated real-world datasets for training AI in architectural vision tasks?
• RQ4 What is the impact of synthetic data diversity and realism on model generalization and performance in real-world scenarios?

Key findings

• The model achieved strong semantic segmentation performance on real-world building photos, even for structures not present in the training data, indicating high generalization capacity.
• The use of parametrically generated virtual buildings enabled the creation of a large-scale, diverse, and fully-annotated training dataset without human labeling.
• The model demonstrated robustness to variations in building style, lighting, and viewpoint, suggesting effective domain generalization from synthetic data.
• The results validate that high-fidelity synthetic data from parametric BIM models can serve as a viable alternative to real-world annotated data for training AI in architectural computer vision.