[Paper Review] Safer Together: Machine Learning Models Trained on Shared Accident Datasets Predict Construction Injuries Better than Company-Specific Models

Antoine J. -P. Tixier, Matthew R. Hallowell|arXiv (Cornell University)|Jan 9, 2023

Occupational Health and Safety Research4 citations

TL;DR

This study demonstrates that machine learning models trained on a shared, aggregated dataset of 57,000 construction accident reports from nine companies outperform company-specific models in predicting injury severity, injury type, and accident type. By leveraging transfer learning across multiple organizations, generic models achieve average F1 score improvements of 4.4 (up to +15.3) and predict 2.26 more outcome categories on average, enabling broader, more actionable safety forecasts while reducing the need for individual company data and model training.

ABSTRACT

In this study, we capitalized on a collective dataset repository of 57k accidents from 9 companies belonging to 3 domains and tested whether models trained on multiple datasets (generic models) predicted safety outcomes better than the company-specific models. We experimented with full generic models (trained on all data), per-domain generic models (construction, electric T&D, oil & gas), and with ensembles of generic and specific models. Results are very positive, with generic models outperforming the company-specific models in most cases while also generating finer-grained, hence more useful, forecasts. Successful generic models remove the needs for training company-specific models, saving a lot of time and resources, and give small companies, whose accident datasets are too limited to train their own models, access to safety outcome predictions. It may still however be advantageous to train specific models to get an extra boost in performance through ensembling with the generic models. Overall, by learning lessons from a pool of datasets whose accumulated experience far exceeds that of any single company, and making these lessons easily accessible in the form of simple forecasts, generic models tackle the holy grail of safety cross-organizational learning and dissemination in the construction industry.

Motivation & Objective

• To evaluate whether generic machine learning models trained on pooled accident data from multiple companies outperform company-specific models in predicting construction safety outcomes.
• To assess the value of data sharing and collective intelligence in improving predictive accuracy for construction injury risk.
• To determine whether per-domain or full generic models are more effective, and whether ensembling generic and specific models enhances performance.
• To enable small companies with limited accident data to access high-quality safety predictions through generic models.
• To establish a scalable, privacy-preserving framework for cross-organizational safety forecasting using standardized attributes.

Proposed method

• The study used a standardized attribute framework to extract fundamental work conditions (e.g., objects on floor, tools, materials) from narrative accident reports using a validated NLP tool based on rules and lexicons.
• All datasets from nine companies across three domains (construction, electric T&D, oil & gas) were standardized into a common taxonomy to ensure data interoperability and privacy.
• Supervised learning models were trained on three types of data: company-specific datasets, full generic datasets (all companies), and per-domain generic datasets (construction, T&D, oil & gas).
• Model performance was evaluated using F1 score, precision, recall, and area under the ROC curve, with stratified cross-validation to ensure robustness.
• Ensemble models were created by combining predictions from generic and company-specific models using weighted averaging, with optimal weights determined on validation sets.
• The best-performing models were selected via hyperparameter tuning and model stacking, with Random Forest, XGBoost, and SVM tested across all configurations.

Figure 1: Company-level max gains, across all domains and outcomes. n=41, min=0.2, max=15.3, mean=4.4.

Experimental results

Research questions

• RQ1 Do generic models trained on shared accident data from multiple companies predict construction injury outcomes more accurately than models trained on individual company data?
• RQ2 Is the performance gain from generic models consistent across different safety outcomes (e.g., injury severity, injury type, accident type) and domains?
• RQ3 Does ensembling generic and company-specific models yield better performance than using either model type alone?
• RQ4 Are per-domain generic models (e.g., construction-only) as effective as full generic models trained on all domains?
• RQ5 What is the optimal balance between data quantity (generic) and data relevance (specific) in ensemble models?

Key findings

• Generic models trained on all 57,000 accident records outperformed company-specific models in 82% of the company-domain-outcome combinations, with an average F1 score improvement of 4.4 (up to +15.3).
• For injury type prediction, generic models achieved an average F1 gain of 6.17, with the highest improvement reaching 10.64 in the construction domain.
• Generic models predicted an average of 2.26 more categories per outcome than company-specific models, with gains up to 7 additional categories, enabling more actionable safety forecasts.
• Ensembling generic and specific models yielded the highest performance, particularly for injury type and severity, where the best results were achieved only when both model types were combined.
• Full generic models were not always superior to per-domain generic models, and in some cases, per-domain models performed better, indicating domain-specific patterns matter.
• XGBoost was the most practical choice for deployment due to its smaller model size compared to Random Forest, despite similar performance, while Random Forest achieved the highest individual scores in 6 out of 15 combinations.

Figure 2: Gains averaged over companies. Left: n=48, min=0.16, max=9.98, mean=2.22. Right: n=11, min=0.95, max=9.98, mean=3.37.