[Paper Review] Material Recognition for Automated Progress Monitoring using Deep Learning Methods

Hadi Mahami, Navid Ghassemi+8 more|arXiv (Cornell University)|Jun 29, 2020

Infrastructure Maintenance and Monitoring43 references26 citations

TL;DR

This paper proposes a deep learning-based material recognition system using pre-trained convolutional neural networks (VGG16, ResNet, DenseNet, NASNet-Mobile) to enable accurate automated progress monitoring in construction. Trained on a publicly available dataset of 1,231 high-quality images across 11 material classes, the method achieves up to 97.35% classification accuracy, demonstrating robustness to lighting variations and camera angles, significantly improving upon prior work in material recognition for construction monitoring.

ABSTRACT

Recent advancements in Artificial intelligence, especially deep learning, has changed many fields irreversibly by introducing state of the art methods for automation. Construction monitoring has not been an exception; as a part of construction monitoring systems, material classification and recognition have drawn the attention of deep learning and machine vision researchers. However, to create production-ready systems, there is still a long path to cover. Real-world problems such as varying illuminations and reaching acceptable accuracies need to be addressed in order to create robust systems. In this paper, we have addressed these issues and reached a state of the art performance, i.e., 97.35% accuracy rate for this task. Also, a new dataset containing 1231 images of 11 classes taken from several construction sites is gathered and publicly published to help other researchers in this field.

Motivation & Objective

• To develop a robust deep learning-based method for automated recognition of construction materials to improve progress monitoring accuracy.
• To address the challenge of low accuracy and error propagation in existing automated construction monitoring systems.
• To evaluate the performance of state-of-the-art CNN architectures under varying environmental conditions such as illumination and camera angles.
• To provide a publicly available, high-quality dataset of 1,231 construction material images to support future research in this domain.
• To investigate data augmentation techniques to mitigate overfitting in small-scale construction material classification tasks.

Proposed method

• Trained multiple pre-trained deep neural networks (VGG16, ResNet, DenseNet, NASNet-Mobile) on a custom dataset of 1,231 high-resolution images from real construction sites.
• Employed data augmentation techniques, including rotation, flipping, and brightness adjustment, to improve model generalization and reduce overfitting.
• Utilized transfer learning by fine-tuning pre-trained models on the 11-class construction material dataset to enhance performance with limited data.
• Evaluated model performance under varying illumination conditions using a subset of images with controlled lighting variations.
• Compared inference time and accuracy across different hardware platforms (Raspberry Pi 3, iPhone 11 Pro, Huawei P30 lite, Samsung Galaxy A50) to assess real-time deployment feasibility.
• Publicly released the dataset via GitHub to support reproducibility and further research in construction material recognition.

Experimental results

Research questions

• RQ1 How accurately can state-of-the-art deep learning models classify construction materials from real site images?
• RQ2 How do different CNN architectures (VGG16, ResNet, DenseNet, NASNet-Mobile) perform under varying illumination and camera conditions?
• RQ3 To what extent does data augmentation improve generalization and reduce overfitting in small construction material datasets?
• RQ4 How does the proposed method compare in accuracy and inference speed to previous approaches in automated construction progress monitoring?
• RQ5 Can the proposed system be effectively deployed on low-cost embedded devices like the Raspberry Pi 3 for real-time monitoring?

Key findings

• The VGG16 model achieved the highest classification accuracy of 97.35% on the full dataset, outperforming other models.
• The ResNet model achieved 90.48% accuracy under low-light conditions with only 5 images per class, demonstrating robustness.
• The NASNet-Mobile model achieved 89.95% accuracy on the 5-image subset, indicating strong performance on limited data.
• The DenseNet model achieved 96.30% accuracy on the full dataset, showing strong generalization with minimal misclassification.
• The confusion matrices revealed high inter-class separation, with most materials correctly classified (e.g., 27/27 for brick, 22/22 for sand).
• The system demonstrated robustness to varying lighting and camera angles, with consistent performance across different environmental conditions.