[Paper Review] SAM-based instance segmentation models for the automation of structural damage detection

Zehao Ye, Lucy Lovell+2 more|arXiv (Cornell University)|Jan 27, 2024

Infrastructure Maintenance and Monitoring4 citations

TL;DR

This paper proposes novel SAM-based instance segmentation methods for automated masonry crack detection, introducing a learnable self-generating prompter and a prompt-free decoder integration. Fine-tuned with LoRA, the models exceed state-of-the-art performance by ~3% overall and ~6% on cracks, while enabling accurate crack size estimation within 10% of laser scanning via monocular camera and Hough Line Transform-based orthographic projection.

ABSTRACT

Automating visual inspection for capturing defects based on civil structures appearance is crucial due to its currently labour-intensive and time-consuming nature. An important aspect of automated inspection is image acquisition, which is rapid and cost-effective considering the pervasive developments in both software and hardware computing in recent years. Previous studies largely focused on concrete and asphalt, with less attention to masonry cracks. The latter also lacks publicly available datasets. In this paper, we first present a corresponding data set for instance segmentation with 1,300 annotated images (640 pixels x 640 pixels), named as MCrack1300, covering bricks, broken bricks, and cracks. We then test several leading algorithms for benchmarking, including the latest large-scale model, the prompt-based Segment Anything Model (SAM). We fine-tune the encoder using Low-Rank Adaptation (LoRA) and proposed two novel methods for automation of SAM execution. The first method involves abandoning the prompt encoder and connecting the SAM encoder to other decoders, while the second method introduces a learnable self-generating prompter. In order to ensure the seamless integration of the two proposed methods with SAM encoder section, we redesign the feature extractor. Both proposed methods exceed state-of-the-art performance, surpassing the best benchmark by approximately 3% for all classes and around 6% for cracks specifically. Based on successful detection, we propose a method based on a monocular camera and the Hough Line Transform to automatically transform images into orthographic projection maps. By incorporating known real sizes of brick units, we accurately estimate crack dimensions, with the results differing by less than 10% from those obtained by laser scanning. Overall, we address important research gaps in automated masonry crack detection and size estimation.

Motivation & Objective

• To address the lack of publicly available datasets for masonry crack instance segmentation.
• To automate structural damage detection in masonry using deep learning, focusing on cracks and broken bricks.
• To overcome limitations of prompt dependency in Segment Anything Model (SAM) for real-world structural inspection.
• To enable accurate crack dimension estimation from monocular images using geometric transformation.
• To develop a scalable, low-cost solution for automated visual inspection of civil infrastructure.

Proposed method

• Proposed a new dataset, MCrack1300, with 1,300 annotated images (640×640 px) of bricks, broken bricks, and cracks.
• Fine-tuned the SAM encoder using Low-Rank Adaptation (LoRA) for efficient parameter adaptation.
• Introduced a prompt-free method by replacing the prompt encoder with alternative decoders connected directly to the SAM encoder.
• Developed a learnable self-generating prompter that autonomously generates effective prompts for segmentation.
• Redesigned the feature extractor to ensure seamless integration of both novel methods with the SAM encoder.
• Applied monocular camera imaging combined with Hough Line Transform to generate orthographic projection maps from real-world images.

Experimental results

Research questions

• RQ1 Can a fine-tuned, prompt-free adaptation of SAM achieve superior instance segmentation performance on masonry cracks compared to standard prompt-based inference?
• RQ2 Can a learnable self-generating prompter effectively replace human-provided prompts in SAM for automated structural damage detection?
• RQ3 To what extent can monocular camera-based orthographic projection enable accurate crack length estimation when combined with known brick dimensions?
• RQ4 How does the proposed method compare to state-of-the-art models in terms of mAP across all crack and masonry defect classes?
• RQ5 Can the system achieve sub-10% error in crack size estimation relative to laser scanning benchmarks?

Key findings

• The proposed prompt-free method achieved a 3% improvement in mean average precision (mAP) over the best benchmark model across all defect classes.
• For crack-specific detection, the method surpassed the state-of-the-art by approximately 6% in mAP.
• The learnable self-generating prompter demonstrated robust performance without requiring manual prompt engineering.
• Crack length estimation using the monocular camera and Hough Line Transform method showed less than 10% deviation from laser scanning measurements.
• The MCrack1300 dataset provides a new benchmark for masonry crack instance segmentation with 1,300 high-resolution, fully annotated images.
• The integration of the redesigned feature extractor enabled stable and high-performing inference across both proposed methods.