[Paper Review] Foundation Models for Structural Health Monitoring

Luca Benfenati, Daniele Jahier Pagliari+7 more|arXiv (Cornell University)|Apr 3, 2024

Structural Health Monitoring Techniques4 citations

TL;DR

This paper introduces the first foundation model for Structural Health Monitoring (SHM) using Transformer-based Masked Autoencoders trained via self-supervised pre-training on vibration data from accelerometers. The approach achieves state-of-the-art performance in anomaly detection (99.9% accuracy in 15 windows) and traffic load estimation (R² up to 0.97), with knowledge distillation enabling deployment of compact models (0.5 MB) on resource-constrained edge devices.

ABSTRACT

Structural Health Monitoring (SHM) is a critical task for ensuring the safety and reliability of civil infrastructures, typically realized on bridges and viaducts by means of vibration monitoring. In this paper, we propose for the first time the use of Transformer neural networks, with a Masked Auto-Encoder architecture, as Foundation Models for SHM. We demonstrate the ability of these models to learn generalizable representations from multiple large datasets through self-supervised pre-training, which, coupled with task-specific fine-tuning, allows them to outperform state-of-the-art traditional methods on diverse tasks, including Anomaly Detection (AD) and Traffic Load Estimation (TLE). We then extensively explore model size versus accuracy trade-offs and experiment with Knowledge Distillation (KD) to improve the performance of smaller Transformers, enabling their embedding directly into the SHM edge nodes. We showcase the effectiveness of our foundation models using data from three operational viaducts. For AD, we achieve a near-perfect 99.9% accuracy with a monitoring time span of just 15 windows. In contrast, a state-of-the-art method based on Principal Component Analysis (PCA) obtains its first good result (95.03% accuracy), only considering 120 windows. On two different TLE tasks, our models obtain state-of-the-art performance on multiple evaluation metrics (R$^2$ score, MAE% and MSE%). On the first benchmark, we achieve an R$^2$ score of 0.97 and 0.90 for light and heavy vehicle traffic, respectively, while the best previous approach (a Random Forest) stops at 0.91 and 0.84. On the second one, we achieve an R$^2$ score of 0.54 versus the 0.51 of the best competitor method, a Long-Short Term Memory network.

Motivation & Objective

• To address the challenge of limited labeled data in SHM by leveraging self-supervised pre-training on large-scale unlabeled vibration data.
• To develop a unified foundation model capable of generalizing across multiple SHM tasks, including anomaly detection and traffic load estimation.
• To optimize model efficiency through knowledge distillation, enabling deployment on low-resource edge nodes in SHM systems.
• To demonstrate the superiority of foundation models over task-specific models and traditional methods like PCA on real-world SHM datasets.
• To explore the trade-off between model size and performance, particularly for edge deployment in structural monitoring.

Proposed method

• A Transformer-based Masked Autoencoder architecture is used for self-supervised pre-training on three large-scale, fully labeled SHM datasets collected from operational viaducts.
• Pre-training is performed without labels, learning generalizable representations of accelerometer time-series data through masked sequence reconstruction.
• Task-specific fine-tuning is applied using only labeled data during the downstream phase, simulating real-world data scarcity.
• Knowledge distillation is applied to compress large teacher models into smaller student models, preserving performance while reducing model size.
• Model compression is evaluated across multiple configurations, with performance measured via MAE%, R², and MSE% on anomaly detection and traffic load estimation tasks.
• A two-stage training protocol is employed: pre-train on all datasets (Pretrain All) followed by fine-tuning per task, enabling transfer learning across diverse SHM scenarios.

Experimental results

Research questions

• RQ1 Can a foundation model based on Transformers and self-supervised learning outperform task-specific models in structural health monitoring?
• RQ2 How effective is self-supervised pre-training using masked autoencoders on vibration data for downstream SHM tasks?
• RQ3 To what extent can knowledge distillation reduce model size without sacrificing performance for edge deployment in SHM systems?
• RQ4 What is the impact of dataset size and diversity on the generalization capability of foundation models in SHM?
• RQ5 How does the performance of foundation models compare to state-of-the-art methods like PCA on anomaly detection and traffic load estimation?

Key findings

• The foundation model achieves 99.9% anomaly detection accuracy with only 15 monitoring windows, significantly outperforming a PCA-based method that requires 120 windows to reach 95.03% accuracy.
• On the first traffic load estimation benchmark, the model achieves an R² score of 0.97 for light vehicles and 0.85 for heavy vehicles, surpassing the previous best of 0.91 and 0.84, respectively.
• On the second traffic load estimation benchmark, the model achieves an R² score of 0.54, compared to 0.10 for the best existing method, demonstrating a substantial improvement.
• Knowledge distillation enables a 195× reduction in model size (from 98 MB to 0.5 MB) while reducing MAE% by 8.6% on the most challenging dataset (UC3), with minimal performance loss.
• The smallest student model (0.5 MB) achieves competitive performance on UC3, with a 5% MAE% reduction compared to the baseline at 3.9× smaller size, indicating strong potential for edge deployment.
• Pre-training on all three datasets (Pretrain All) yields the best results on the most complex downstream task (UC3), highlighting the benefit of diverse pre-training data for generalization.